2024-09-18T11:33:31.597 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T11:33:31.611 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T11:33:31.690 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910981 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} email: skanta@redhat.com first_in_suite: false job_id: '7910981' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNfo/MHNdWKPVfFdQFiqWiwS/7ZKRIhzLElZ4yU03QpgvfyvEaXgOEVe4A8O+9UEbfIor5pFA1OQpEfFa5Z+w1A= tasks: - install: null - ceph: create_rbd_pool: false log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - ec_lost_unfound: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T11:33:31.690 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T11:33:31.691 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T11:33:31.691 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T11:33:31.693 INFO:teuthology.task.internal:Checking packages... 2024-09-18T11:33:31.714 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T11:33:31.714 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T11:33:31.714 INFO:teuthology.packaging:ref: None 2024-09-18T11:33:31.714 INFO:teuthology.packaging:tag: None 2024-09-18T11:33:31.714 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T11:33:31.714 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:33:31.715 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T11:33:31.864 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645.g7061ea6c 2024-09-18T11:33:31.866 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T11:33:31.871 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T11:33:31.871 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T11:33:31.890 INFO:teuthology.task.internal:Saving configuration 2024-09-18T11:33:31.902 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T11:33:31.907 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T11:33:31.930 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910981', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 11:28:43.127674', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNfo/MHNdWKPVfFdQFiqWiwS/7ZKRIhzLElZ4yU03QpgvfyvEaXgOEVe4A8O+9UEbfIor5pFA1OQpEfFa5Z+w1A='} 2024-09-18T11:33:31.930 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T11:33:31.936 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-18T11:33:31.936 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T11:33:31.971 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7e725a3ac0>, signals=[15]) 2024-09-18T11:33:31.971 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T11:33:31.977 INFO:teuthology.task.internal:Opening connections... 2024-09-18T11:33:31.977 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T11:33:31.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:33:32.056 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T11:33:32.062 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-09-18T11:33:32.135 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-09-18T11:33:32.135 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-09-18T11:33:32.193 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-09-18T11:33:32.193 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2024-09-18T11:33:32.193 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-09-18T11:33:32.193 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-09-18T11:33:32.193 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2024-09-18T11:33:32.193 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2024-09-18T11:33:32.194 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T11:33:32.194 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-09-18T11:33:32.194 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2024-09-18T11:33:32.194 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T11:33:32.194 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-09-18T11:33:32.194 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T11:33:32.195 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T11:33:32.195 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T11:33:32.196 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-09-18T11:33:32.219 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T11:33:32.227 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T11:33:32.237 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T11:33:32.237 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-09-18T11:33:32.254 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T11:33:32.259 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T11:33:32.259 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-09-18T11:33:32.355 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T11:33:32.385 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T11:33:32.398 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T11:33:32.398 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T11:33:32.399 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T11:33:32.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:33:32.416 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-09-18T11:33:32.474 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-503.el9.x86_64 2024-09-18T11:33:32.474 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.14.0-503.el9.x86_64 2024-09-18T11:33:32.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2024-09-18T11:33:35.005 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-18T11:33:40.522 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-18T11:33:46.206 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-18T11:33:47.673 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 19 kB 00:00 2024-09-18T11:33:49.002 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-18T11:33:54.460 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-09-18T11:33:56.356 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T11:33:56.356 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T11:33:56.461 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:33:56.466 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:33:56.467 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:33:56.467 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-09-18T11:33:56.467 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:33:56.467 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 79 M 2024-09-18T11:33:56.467 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 126 M 2024-09-18T11:33:56.467 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:33:56.609 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 299 kB/s | 36 kB 00:00 2024-09-18T11:34:02.733 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.5 MB/s | 16 MB 00:06 2024-09-18T11:34:10.158 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 2.1 MB/s | 29 MB 00:13 2024-09-18T11:34:13.040 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 2.1 MB/s | 35 MB 00:16 2024-09-18T11:34:13.041 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:34:13.041 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.8 MB/s | 79 MB 00:16 2024-09-18T11:34:13.727 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:34:14.090 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:34:14.090 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:34:15.035 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:34:15.036 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:34:17.463 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:34:18.094 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T11:34:18.184 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T11:34:20.938 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T11:34:21.072 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T11:34:25.927 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T11:34:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:34:30.897 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:35:09.128 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:35:09.168 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:35:09.517 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:35:09.517 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T11:35:09.517 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T11:35:09.517 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T11:35:10.338 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:35:10.338 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:35:10.338 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:35:10.339 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T11:35:10.339 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T11:35:10.339 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T11:35:10.339 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T11:35:10.339 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:35:10.339 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:35:10.697 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T11:35:10.774 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T11:35:10.774 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T11:35:10.775 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T11:35:10.775 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T11:35:10.775 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T11:35:10.775 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T11:35:10.775 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T11:35:10.775 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T11:35:10.834 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T11:35:10.834 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T11:35:10.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2024-09-18T11:35:10.869 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2024-09-18T11:35:10.869 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2024-09-18T11:35:10.869 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.06 2024-09-18T11:35:10.869 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 82.el9 2024-09-18T11:35:10.869 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2024-09-18T11:35:10.869 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T11:35:10.869 INFO:teuthology.orchestra.run.smithi071.stdout:Group : Unspecified 2024-09-18T11:35:10.869 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 8274670 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : builder@centos.org 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T11:35:10.870 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T11:35:10.871 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T11:35:10.871 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2024-09-18T11:35:10.871 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:35:10.871 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T11:35:10.872 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T11:35:10.872 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 5.14.0-508.el9.x86_64 2024-09-18T11:35:10.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T11:35:11.574 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-09-18T11:35:12.860 INFO:teuthology.orchestra.run.smithi071.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T11:35:12.888 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-09-18T11:35:12.895 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-09-18T11:35:12.910 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.jS9l6pP67j 2024-09-18T11:35:12.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jS9l6pP67j 2024-09-18T11:35:12.978 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.jS9l6pP67j 2024-09-18T11:35:13.115 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.jS9l6pP67j is 6KB 2024-09-18T11:35:13.164 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.jS9l6pP67j 2024-09-18T11:35:13.179 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T11:35:13.241 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T11:35:13.242 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T11:35:13.242 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T11:35:13.242 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T11:35:13.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T11:35:13.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-09-18T11:35:43.426 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T11:35:43.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-18T11:35:43.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:36:01.853 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-09-18T11:36:10.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-18T11:36:10.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:36:13.917 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-09-18T11:36:25.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-18T11:36:25.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:36:57.686 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-09-18T11:36:58.019 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-18T11:36:58.019 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-18T11:36:58.020 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-09-18T11:36:58.035 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-508.el9.x86_64 2024-09-18T11:36:58.035 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T11:36:58.035 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T11:36:58.036 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T11:36:59.037 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T11:36:59.037 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T11:36:59.105 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-09-18T11:36:59.133 DEBUG:teuthology.parallel:result is None 2024-09-18T11:36:59.134 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T11:36:59.140 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T11:36:59.140 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T11:36:59.159 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T11:36:59.166 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T11:36:59.174 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T11:36:59.174 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T11:36:59.247 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T11:36:59.253 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T11:36:59.254 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T11:36:59.327 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:59.338 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:59.340 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T11:36:59.347 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T11:36:59.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T11:36:59.374 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T11:36:59.383 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T11:36:59.383 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T11:36:59.431 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T11:36:59.516 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T11:36:59.565 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T11:36:59.647 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T11:36:59.689 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:36:59.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T11:36:59.754 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-09-18T11:36:59.835 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T11:37:00.221 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T11:37:00.231 INFO:teuthology.task.internal:Starting timer... 2024-09-18T11:37:00.232 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T11:37:00.243 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T11:37:00.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-09-18T11:37:00.301 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-09-18T11:37:00.302 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T11:37:00.303 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-09-18T11:37:00.372 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-09-18T11:37:00.372 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi071.front.sepia.ceph.com': 'permissive'} 2024-09-18T11:37:00.372 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-18T11:37:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:37:00.437 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T11:37:00.437 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-09-18T11:37:00.509 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T11:37:00.522 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T11:37:00.535 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T11:37:00.536 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi071.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T11:40:43.525 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi071.front.sepia.ceph.com')] 2024-09-18T11:40:43.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-18T11:40:43.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:40:43.603 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-09-18T11:40:43.684 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-18T11:40:43.685 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T11:40:43.699 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T11:40:43.700 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T11:40:43.700 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T11:40:43.753 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T11:40:43.767 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T11:40:43.791 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-09-18T11:40:43.802 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-09-18T11:40:43.815 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T11:40:43.829 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T11:40:43.891 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-09-18T11:40:43.895 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T11:40:43.895 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-09-18T11:40:43.895 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T11:40:43.895 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T11:40:43.895 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T11:40:43.895 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T11:40:43.895 INFO:teuthology.run_tasks:Running task install... 2024-09-18T11:40:43.905 DEBUG:teuthology.task.install:project ceph 2024-09-18T11:40:43.905 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T11:40:43.905 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T11:40:43.906 INFO:teuthology.task.install:Using flavor: default 2024-09-18T11:40:43.913 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T11:40:43.914 INFO:teuthology.task.install:extra packages: [] 2024-09-18T11:40:43.914 DEBUG:teuthology.orchestra.run.smithi071:> sudo dnf -y install 'dnf-command(copr)' 2024-09-18T11:40:44.507 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:45 ago on Wed 18 Sep 2024 11:39:59 AM UTC. 2024-09-18T11:40:44.592 INFO:teuthology.orchestra.run.smithi071.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-18T11:40:44.635 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:40:44.635 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-18T11:40:44.636 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:40:44.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo dnf -y copr enable ceph/el9 2024-09-18T11:40:44.879 INFO:teuthology.orchestra.run.smithi071.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-18T11:40:44.879 INFO:teuthology.orchestra.run.smithi071.stderr:of the main distribution, and quality may vary. 2024-09-18T11:40:44.880 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T11:40:44.880 INFO:teuthology.orchestra.run.smithi071.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-18T11:40:44.880 INFO:teuthology.orchestra.run.smithi071.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-18T11:40:44.880 INFO:teuthology.orchestra.run.smithi071.stderr:, 2024-09-18T11:40:44.880 INFO:teuthology.orchestra.run.smithi071.stderr:and packages are not held to any quality or security level. 2024-09-18T11:40:44.880 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T11:40:44.880 INFO:teuthology.orchestra.run.smithi071.stderr:Please do not file bug reports about these packages in Fedora 2024-09-18T11:40:44.880 INFO:teuthology.orchestra.run.smithi071.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-18T11:40:44.938 INFO:teuthology.orchestra.run.smithi071.stdout:Repository successfully enabled. 2024-09-18T11:40:44.973 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'tag': None, 'wait_for_package': False} 2024-09-18T11:40:44.974 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:40:45.118 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/ 2024-09-18T11:40:45.119 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1645.g7061ea6c 2024-09-18T11:40:45.290 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T11:40:45.290 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:40:45.291 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T11:40:45.317 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-18T11:40:45.318 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7061ea6ca50908d104e9f3979dc615e118d9e153/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T11:40:45.383 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T11:40:45.458 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T11:40:45.522 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2024-09-18T11:40:45.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2024-09-18T11:40:45.749 INFO:teuthology.orchestra.run.smithi071.stdout:45 files removed 2024-09-18T11:40:45.776 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2024-09-18T11:40:46.560 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 154 kB/s | 77 kB 00:00 2024-09-18T11:40:46.952 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-09-18T11:40:47.386 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 4.5 kB/s | 1.8 kB 00:00 2024-09-18T11:40:47.561 INFO:teuthology.orchestra.run.smithi071.stdout:Copr repo for el9 owned by ceph 54 kB/s | 7.6 kB 00:00 2024-09-18T11:40:51.203 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.3 MB 00:03 2024-09-18T11:40:52.945 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-18T11:40:57.221 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-18T11:40:58.838 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-18T11:41:00.084 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-18T11:41:05.514 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-18T11:41:07.856 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:41:07.881 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:07.881 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-18T11:41:07.881 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:07.881 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:41:07.881 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 14 M 2024-09-18T11:41:07.881 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-09-18T11:41:07.881 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 3.1 M 2024-09-18T11:41:07.881 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.9 M 2024-09-18T11:41:07.881 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-18T11:41:07.881 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 5.0 M 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 22 M 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 25 k 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 637 k 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 458 k 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 4.9 M 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T11:41:07.882 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 45 k 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 120 k 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 156 k 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 320 k 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 296 k 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 99 k 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout:Install 22 Packages 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2024-09-18T11:41:07.883 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:07.885 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 58 M 2024-09-18T11:41:07.885 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:41:08.759 INFO:teuthology.orchestra.run.smithi071.stdout:(1/24): ceph-base-17.2.7-1645.g7061ea6c.el9.x86 11 MB/s | 5.0 MB 00:00 2024-09-18T11:41:08.834 INFO:teuthology.orchestra.run.smithi071.stdout:(2/24): ceph-selinux-17.2.7-1645.g7061ea6c.el9. 337 kB/s | 25 kB 00:00 2024-09-18T11:41:08.934 INFO:teuthology.orchestra.run.smithi071.stdout:(3/24): libcephfs2-17.2.7-1645.g7061ea6c.el9.x8 6.2 MB/s | 637 kB 00:00 2024-09-18T11:41:09.018 INFO:teuthology.orchestra.run.smithi071.stdout:(4/24): libradosstriper1-17.2.7-1645.g7061ea6c. 5.4 MB/s | 458 kB 00:00 2024-09-18T11:41:09.368 INFO:teuthology.orchestra.run.smithi071.stdout:(5/24): librgw2-17.2.7-1645.g7061ea6c.el9.x86_6 14 MB/s | 4.9 MB 00:00 2024-09-18T11:41:09.419 INFO:teuthology.orchestra.run.smithi071.stdout:(6/24): python3-ceph-argparse-17.2.7-1645.g7061 906 kB/s | 45 kB 00:00 2024-09-18T11:41:09.652 INFO:teuthology.orchestra.run.smithi071.stdout:(7/24): ceph-common-17.2.7-1645.g7061ea6c.el9.x 17 MB/s | 22 MB 00:01 2024-09-18T11:41:09.669 INFO:teuthology.orchestra.run.smithi071.stdout:(8/24): python3-ceph-common-17.2.7-1645.g7061ea 480 kB/s | 120 kB 00:00 2024-09-18T11:41:09.811 INFO:teuthology.orchestra.run.smithi071.stdout:(9/24): ceph-radosgw-17.2.7-1645.g7061ea6c.el9. 9.5 MB/s | 14 MB 00:01 2024-09-18T11:41:09.836 INFO:teuthology.orchestra.run.smithi071.stdout:(10/24): python3-cephfs-17.2.7-1645.g7061ea6c.e 852 kB/s | 156 kB 00:00 2024-09-18T11:41:09.861 INFO:teuthology.orchestra.run.smithi071.stdout:(11/24): python3-rados-17.2.7-1645.g7061ea6c.el 1.6 MB/s | 320 kB 00:00 2024-09-18T11:41:09.886 INFO:teuthology.orchestra.run.smithi071.stdout:(12/24): python3-rgw-17.2.7-1645.g7061ea6c.el9. 1.9 MB/s | 99 kB 00:00 2024-09-18T11:41:09.912 INFO:teuthology.orchestra.run.smithi071.stdout:(13/24): python3-rbd-17.2.7-1645.g7061ea6c.el9. 2.9 MB/s | 296 kB 00:00 2024-09-18T11:41:09.970 INFO:teuthology.orchestra.run.smithi071.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-09-18T11:41:09.987 INFO:teuthology.orchestra.run.smithi071.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-09-18T11:41:10.029 INFO:teuthology.orchestra.run.smithi071.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-18T11:41:10.046 INFO:teuthology.orchestra.run.smithi071.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 721 kB/s | 42 kB 00:00 2024-09-18T11:41:10.080 INFO:teuthology.orchestra.run.smithi071.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-09-18T11:41:10.130 INFO:teuthology.orchestra.run.smithi071.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-09-18T11:41:10.155 INFO:teuthology.orchestra.run.smithi071.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-18T11:41:10.181 INFO:teuthology.orchestra.run.smithi071.stdout:(21/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-18T11:41:10.306 INFO:teuthology.orchestra.run.smithi071.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-09-18T11:41:10.356 INFO:teuthology.orchestra.run.smithi071.stdout:(23/24): librados2-17.2.7-1645.g7061ea6c.el9.x8 15 MB/s | 3.1 MB 00:00 2024-09-18T11:41:10.423 INFO:teuthology.orchestra.run.smithi071.stdout:(24/24): librbd1-17.2.7-1645.g7061ea6c.el9.x86_ 12 MB/s | 2.9 MB 00:00 2024-09-18T11:41:10.428 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:41:10.428 INFO:teuthology.orchestra.run.smithi071.stdout:Total 23 MB/s | 58 MB 00:02 2024-09-18T11:41:10.492 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:41:10.525 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:41:10.525 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:41:11.180 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:41:11.180 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:41:12.411 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:41:12.738 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-18T11:41:12.842 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T11:41:12.933 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T11:41:13.012 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/26 2024-09-18T11:41:13.051 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T11:41:13.140 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T11:41:13.185 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-18T11:41:13.214 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-18T11:41:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-18T11:41:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 8/26 2024-09-18T11:41:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 8/26 2024-09-18T11:41:13.795 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-18T11:41:13.838 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9. 10/26 2024-09-18T11:41:13.875 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 11/26 2024-09-18T11:41:13.942 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 12/26 2024-09-18T11:41:13.973 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 13/26 2024-09-18T11:41:14.039 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 13/26 2024-09-18T11:41:14.079 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-18T11:41:14.123 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-18T11:41:14.165 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-18T11:41:14.459 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-18T11:41:14.502 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 18/26 2024-09-18T11:41:14.592 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 18/26 2024-09-18T11:41:14.664 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 19/26 2024-09-18T11:41:14.696 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x8 20/26 2024-09-18T11:41:17.154 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T11:41:17.200 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T11:41:17.733 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T11:41:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 22/26 2024-09-18T11:41:17.869 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 22/26 2024-09-18T11:41:17.869 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T11:41:17.869 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T11:41:17.869 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:17.904 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T11:41:27.738 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T11:41:27.739 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2024-09-18T11:41:27.739 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2024-09-18T11:41:27.739 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2024-09-18T11:41:27.739 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2024-09-18T11:41:27.739 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2024-09-18T11:41:27.739 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2024-09-18T11:41:27.739 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2024-09-18T11:41:27.739 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:29.059 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 24/26 2024-09-18T11:41:29.134 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 24/26 2024-09-18T11:41:29.134 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:41:29.134 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T11:41:29.135 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T11:41:29.135 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T11:41:29.135 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:29.136 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-18T11:41:29.228 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-18T11:41:29.228 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T11:41:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T11:41:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/26 2024-09-18T11:41:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T11:41:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/26 2024-09-18T11:41:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T11:41:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 5/26 2024-09-18T11:41:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 6/26 2024-09-18T11:41:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 7/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9. 8/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x8 9/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 10/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 11/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 12/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 13/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-18T11:41:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-18T11:41:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-18T11:41:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-18T11:41:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T11:41:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-18T11:41:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 25/26 2024-09-18T11:41:31.457 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T11:41:31.457 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:31.457 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-09-18T11:41:31.457 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.457 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.457 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:41:31.457 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.457 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.457 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T11:41:31.458 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T11:41:31.459 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.459 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.459 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.459 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T11:41:31.459 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.459 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.459 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:31.459 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T11:41:31.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:31.459 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:41:31.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2024-09-18T11:41:32.398 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:27 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:41:32.535 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:41:32.535 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 48 M 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 163 k 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:32.536 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-09-18T11:41:32.537 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:32.537 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 48 M 2024-09-18T11:41:32.537 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 193 M 2024-09-18T11:41:32.537 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:41:32.768 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-18T11:41:32.826 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-18T11:41:32.852 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): libcephsqlite-17.2.7-1645.g7061ea6c.el9. 759 kB/s | 163 kB 00:00 2024-09-18T11:41:34.102 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): ceph-test-17.2.7-1645.g7061ea6c.el9.x86_ 33 MB/s | 48 MB 00:01 2024-09-18T11:41:34.104 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:41:34.104 INFO:teuthology.orchestra.run.smithi071.stdout:Total 31 MB/s | 48 MB 00:01 2024-09-18T11:41:34.131 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:41:34.150 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:41:34.150 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:41:34.515 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:41:34.516 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:41:35.036 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:41:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T11:41:35.167 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T11:41:35.186 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/4 2024-09-18T11:41:39.568 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/4 2024-09-18T11:41:39.642 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/4 2024-09-18T11:41:41.697 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/4 2024-09-18T11:41:41.697 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/4 2024-09-18T11:41:41.697 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/4 2024-09-18T11:41:41.697 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T11:41:42.016 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T11:41:42.016 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:42.016 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:41:42.016 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:42.016 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:42.016 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T11:41:42.017 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T11:41:42.017 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:42.017 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:41:42.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2024-09-18T11:41:42.898 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:37 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:41:43.030 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:41:43.032 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:43.032 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 6.5 k 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.1 M 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 1.4 M 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 240 k 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 3.9 M 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 15 M 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T11:41:43.033 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T11:41:43.034 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:43.035 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:41:43.036 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:43.036 INFO:teuthology.orchestra.run.smithi071.stdout:Install 34 Packages 2024-09-18T11:41:43.036 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:43.036 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 27 M 2024-09-18T11:41:43.036 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 96 M 2024-09-18T11:41:43.036 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:41:43.601 INFO:teuthology.orchestra.run.smithi071.stdout:(1/34): ceph-17.2.7-1645.g7061ea6c.el9.x86_64.r 51 kB/s | 6.5 kB 00:00 2024-09-18T11:41:43.793 INFO:teuthology.orchestra.run.smithi071.stdout:(2/34): ceph-mgr-17.2.7-1645.g7061ea6c.el9.x86_ 4.4 MB/s | 1.4 MB 00:00 2024-09-18T11:41:43.843 INFO:teuthology.orchestra.run.smithi071.stdout:(3/34): ceph-mds-17.2.7-1645.g7061ea6c.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-09-18T11:41:43.927 INFO:teuthology.orchestra.run.smithi071.stdout:(4/34): ceph-mon-17.2.7-1645.g7061ea6c.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-18T11:41:43.952 INFO:teuthology.orchestra.run.smithi071.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1645.g7061 2.2 MB/s | 240 kB 00:00 2024-09-18T11:41:44.077 INFO:teuthology.orchestra.run.smithi071.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-18T11:41:44.110 INFO:teuthology.orchestra.run.smithi071.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 3.2 MB/s | 106 kB 00:00 2024-09-18T11:41:44.153 INFO:teuthology.orchestra.run.smithi071.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 6.1 MB/s | 1.2 MB 00:00 2024-09-18T11:41:44.178 INFO:teuthology.orchestra.run.smithi071.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-18T11:41:44.203 INFO:teuthology.orchestra.run.smithi071.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-18T11:41:44.228 INFO:teuthology.orchestra.run.smithi071.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-18T11:41:44.387 INFO:teuthology.orchestra.run.smithi071.stdout:(12/34): ceph-osd-17.2.7-1645.g7061ea6c.el9.x86 26 MB/s | 15 MB 00:00 2024-09-18T11:41:44.471 INFO:teuthology.orchestra.run.smithi071.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 144 kB/s | 35 kB 00:00 2024-09-18T11:41:44.496 INFO:teuthology.orchestra.run.smithi071.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 588 kB/s | 172 kB 00:00 2024-09-18T11:41:44.546 INFO:teuthology.orchestra.run.smithi071.stdout:(15/34): python3-toml-0.10.2-6.el9.noarch.rpm 836 kB/s | 42 kB 00:00 2024-09-18T11:41:44.563 INFO:teuthology.orchestra.run.smithi071.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 439 kB/s | 77 kB 00:00 2024-09-18T11:41:44.588 INFO:teuthology.orchestra.run.smithi071.stdout:(17/34): python3-pytz-2021.1-5.el9.noarch.rpm 433 kB/s | 51 kB 00:00 2024-09-18T11:41:44.647 INFO:teuthology.orchestra.run.smithi071.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-18T11:41:44.672 INFO:teuthology.orchestra.run.smithi071.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2024-09-18T11:41:44.698 INFO:teuthology.orchestra.run.smithi071.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-18T11:41:44.723 INFO:teuthology.orchestra.run.smithi071.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-18T11:41:44.748 INFO:teuthology.orchestra.run.smithi071.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-18T11:41:44.773 INFO:teuthology.orchestra.run.smithi071.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-18T11:41:44.790 INFO:teuthology.orchestra.run.smithi071.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 468 kB/s | 20 kB 00:00 2024-09-18T11:41:44.815 INFO:teuthology.orchestra.run.smithi071.stdout:(25/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-18T11:41:44.841 INFO:teuthology.orchestra.run.smithi071.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 690 kB/s | 46 kB 00:00 2024-09-18T11:41:44.866 INFO:teuthology.orchestra.run.smithi071.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-18T11:41:44.883 INFO:teuthology.orchestra.run.smithi071.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2024-09-18T11:41:44.908 INFO:teuthology.orchestra.run.smithi071.stdout:(29/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-09-18T11:41:44.933 INFO:teuthology.orchestra.run.smithi071.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-09-18T11:41:44.959 INFO:teuthology.orchestra.run.smithi071.stdout:(31/34): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-09-18T11:41:44.976 INFO:teuthology.orchestra.run.smithi071.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-18T11:41:45.001 INFO:teuthology.orchestra.run.smithi071.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-18T11:41:45.034 INFO:teuthology.orchestra.run.smithi071.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-18T11:41:45.038 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:41:45.038 INFO:teuthology.orchestra.run.smithi071.stdout:Total 14 MB/s | 27 MB 00:02 2024-09-18T11:41:45.161 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:41:45.198 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:41:45.198 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:41:45.600 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:41:45.601 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:41:46.829 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:41:46.887 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-18T11:41:46.919 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-18T11:41:46.952 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-18T11:41:47.043 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-18T11:41:47.130 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-18T11:41:47.171 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-18T11:41:47.207 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-18T11:41:47.236 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-18T11:41:47.271 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-18T11:41:47.329 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-18T11:41:47.369 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-18T11:41:47.408 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-18T11:41:47.442 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-18T11:41:47.491 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-18T11:41:47.533 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-18T11:41:47.613 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-18T11:41:47.752 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-18T11:41:47.838 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-18T11:41:47.892 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-18T11:41:47.930 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-18T11:41:47.979 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-18T11:41:48.035 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-18T11:41:48.221 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-18T11:41:48.268 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-18T11:41:48.346 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-18T11:41:48.441 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-18T11:41:48.500 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-18T11:41:48.616 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-18T11:41:48.762 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9. 29/34 2024-09-18T11:41:48.793 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 30/34 2024-09-18T11:41:48.818 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 30/34 2024-09-18T11:41:48.818 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:41:48.818 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T11:41:48.818 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T11:41:48.818 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T11:41:48.819 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:50.153 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 31/34 2024-09-18T11:41:50.182 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 31/34 2024-09-18T11:41:50.182 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:41:50.182 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T11:41:50.182 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T11:41:50.182 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T11:41:50.182 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 32/34 2024-09-18T11:41:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 32/34 2024-09-18T11:41:50.586 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:41:50.586 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T11:41:50.586 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T11:41:50.587 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T11:41:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 33/34 2024-09-18T11:41:50.820 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 33/34 2024-09-18T11:41:50.820 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:41:50.820 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T11:41:50.820 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T11:41:50.820 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T11:41:50.820 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:50.885 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 34/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 34/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 5/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9. 6/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-18T11:41:52.606 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-18T11:41:52.607 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-18T11:41:52.608 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-18T11:41:52.608 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-18T11:41:52.608 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-18T11:41:52.608 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-18T11:41:52.609 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-18T11:41:52.609 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-18T11:41:52.609 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-18T11:41:52.609 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-18T11:41:52.609 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-18T11:41:53.556 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-18T11:41:53.556 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:53.556 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:41:53.556 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:53.556 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:53.556 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:53.556 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:41:53.556 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:53.556 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T11:41:53.557 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T11:41:53.558 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T11:41:53.559 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T11:41:53.559 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T11:41:53.559 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T11:41:53.559 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:53.559 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:41:53.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2024-09-18T11:41:54.544 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:49 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:41:54.631 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:41:54.672 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:41:54.673 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-18T11:41:54.673 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:41:54.750 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2024-09-18T11:41:55.324 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:50 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:41:55.455 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:41:55.455 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:55.455 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 83 k 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 83 k 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 375 k 2024-09-18T11:41:55.456 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:41:55.867 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-17.2.7-1645.g7061ea6c.el9.noarch.rpm 201 kB/s | 83 kB 00:00 2024-09-18T11:41:55.868 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:41:55.868 INFO:teuthology.orchestra.run.smithi071.stdout:Total 200 kB/s | 83 kB 00:00 2024-09-18T11:41:55.868 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:41:55.873 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:41:55.873 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:41:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:41:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:41:55.944 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:41:56.366 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:41:56.455 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:41:56.792 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:41:57.048 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:41:57.048 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:57.049 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:41:57.049 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:41:57.049 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:57.049 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:41:57.249 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T11:41:57.830 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:52 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:41:57.962 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:41:57.963 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:57.963 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-09-18T11:41:57.963 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:57.963 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:41:57.964 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 137 k 2024-09-18T11:41:57.964 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:57.964 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:41:57.964 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:41:57.964 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-18T11:41:57.964 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:57.964 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 137 k 2024-09-18T11:41:57.964 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 379 k 2024-09-18T11:41:57.964 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:41:58.165 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-17.2.7-1645.g7061ea 686 kB/s | 137 kB 00:00 2024-09-18T11:41:58.165 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:41:58.166 INFO:teuthology.orchestra.run.smithi071.stdout:Total 681 kB/s | 137 kB 00:00 2024-09-18T11:41:58.166 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:41:58.173 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:41:58.173 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:41:58.229 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:41:58.229 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:41:58.381 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:41:58.457 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T11:41:58.481 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T11:41:58.482 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:41:58.482 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T11:41:58.482 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:59.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T11:41:59.447 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:59.447 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:41:59.447 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:41:59.447 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:41:59.447 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:41:59.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2024-09-18T11:42:00.236 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:55 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:00.329 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:42:00.372 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:00.373 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-18T11:42:00.373 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:00.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T11:42:01.010 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:56 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:01.140 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 3.4 M 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 24 k 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 15 k 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T11:42:01.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T11:42:01.142 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-09-18T11:42:01.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-18T11:42:01.142 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:01.142 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:01.142 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:01.142 INFO:teuthology.orchestra.run.smithi071.stdout:Install 7 Packages 2024-09-18T11:42:01.142 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:01.143 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.7 M 2024-09-18T11:42:01.143 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 80 M 2024-09-18T11:42:01.143 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:01.350 INFO:teuthology.orchestra.run.smithi071.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1645.g706 168 kB/s | 24 kB 00:00 2024-09-18T11:42:01.376 INFO:teuthology.orchestra.run.smithi071.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1645.g7061 86 kB/s | 15 kB 00:00 2024-09-18T11:42:01.409 INFO:teuthology.orchestra.run.smithi071.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 154 kB/s | 9.0 kB 00:00 2024-09-18T11:42:01.435 INFO:teuthology.orchestra.run.smithi071.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-18T11:42:01.460 INFO:teuthology.orchestra.run.smithi071.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 488 kB/s | 41 kB 00:00 2024-09-18T11:42:01.485 INFO:teuthology.orchestra.run.smithi071.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-18T11:42:01.594 INFO:teuthology.orchestra.run.smithi071.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1645.g7061ea6c 8.6 MB/s | 3.4 MB 00:00 2024-09-18T11:42:01.597 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:01.597 INFO:teuthology.orchestra.run.smithi071.stdout:Total 8.0 MB/s | 3.7 MB 00:00 2024-09-18T11:42:01.597 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:01.605 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:01.605 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:01.706 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:01.707 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:02.016 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:02.070 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-18T11:42:02.103 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-18T11:42:02.158 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-18T11:42:02.191 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-18T11:42:02.219 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.n 5/7 2024-09-18T11:42:03.206 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9. 6/7 2024-09-18T11:42:03.250 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 7/7 2024-09-18T11:42:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 7/7 2024-09-18T11:42:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9. 1/7 2024-09-18T11:42:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 2/7 2024-09-18T11:42:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.n 3/7 2024-09-18T11:42:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-18T11:42:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-18T11:42:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:04.539 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:04.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T11:42:05.348 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:00 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:05.481 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:05.482 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2024-09-18T11:42:05.483 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-18T11:42:05.483 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2024-09-18T11:42:05.483 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:05.483 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 7.4 M 2024-09-18T11:42:05.483 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-18T11:42:05.483 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T11:42:05.483 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T11:42:05.484 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T11:42:05.484 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T11:42:05.484 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T11:42:05.484 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T11:42:05.484 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T11:42:05.484 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T11:42:05.484 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T11:42:05.484 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T11:42:05.484 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T11:42:05.484 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:05.485 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:05.485 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2024-09-18T11:42:05.485 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-09-18T11:42:05.485 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:05.485 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 38 M 2024-09-18T11:42:05.486 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 212 M 2024-09-18T11:42:05.486 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:05.978 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-18T11:42:06.011 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.9 MB/s | 801 kB 00:00 2024-09-18T11:42:06.362 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 13 MB/s | 7.4 MB 00:00 2024-09-18T11:42:07.045 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 27 kB/s | 18 kB 00:00 2024-09-18T11:42:07.179 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): flexiblas-3.0.4-8.el9.x86_64.rpm 27 kB/s | 33 kB 00:01 2024-09-18T11:42:08.662 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 1.1 MB/s | 3.0 MB 00:02 2024-09-18T11:42:08.704 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 23 kB/s | 38 kB 00:01 2024-09-18T11:42:09.446 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 2.2 MB/s | 4.9 MB 00:02 2024-09-18T11:42:09.496 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 294 kB/s | 245 kB 00:00 2024-09-18T11:42:09.954 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.1 MB/s | 5.1 MB 00:01 2024-09-18T11:42:10.304 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 292 kB/s | 250 kB 00:00 2024-09-18T11:42:11.113 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-18T11:42:11.114 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:11.115 INFO:teuthology.orchestra.run.smithi071.stdout:Total 6.8 MB/s | 38 MB 00:05 2024-09-18T11:42:11.468 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:11.494 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:11.494 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:11.997 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:11.998 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:12.749 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:12.815 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T11:42:12.852 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T11:42:12.953 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T11:42:12.996 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T11:42:13.179 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T11:42:13.219 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T11:42:13.295 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T11:42:13.960 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T11:42:14.066 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T11:42:15.310 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T11:42:16.337 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T11:42:16.368 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 12/12 2024-09-18T11:42:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 12/12 2024-09-18T11:42:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 1/12 2024-09-18T11:42:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T11:42:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T11:42:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T11:42:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T11:42:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T11:42:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T11:42:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T11:42:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T11:42:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T11:42:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T11:42:19.263 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T11:42:19.264 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T11:42:19.264 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T11:42:19.264 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T11:42:19.264 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T11:42:19.264 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:19.264 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:19.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2024-09-18T11:42:20.227 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:15 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:20.368 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:20.369 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:20.369 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-18T11:42:20.371 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 47 k 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T11:42:20.372 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T11:42:20.373 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T11:42:20.373 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T11:42:20.373 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:20.373 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:20.373 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:20.373 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2024-09-18T11:42:20.373 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:20.374 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.2 M 2024-09-18T11:42:20.374 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-09-18T11:42:20.374 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:20.875 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 160 kB/s | 19 kB 00:00 2024-09-18T11:42:20.900 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 176 kB/s | 26 kB 00:00 2024-09-18T11:42:20.925 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): ceph-mgr-rook-17.2.7-1645.g7061ea6c.el9 278 kB/s | 47 kB 00:00 2024-09-18T11:42:21.009 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-18T11:42:21.042 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-18T11:42:21.068 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-18T11:42:21.127 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 384 kB/s | 32 kB 00:00 2024-09-18T11:42:21.152 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 168 kB/s | 14 kB 00:00 2024-09-18T11:42:21.177 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-18T11:42:21.211 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-18T11:42:21.253 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 10 MB/s | 1.0 MB 00:00 2024-09-18T11:42:21.278 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 590 kB/s | 59 kB 00:00 2024-09-18T11:42:21.294 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-09-18T11:42:21.297 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:21.297 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-18T11:42:21.358 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:21.374 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:21.374 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:21.462 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:21.462 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:21.963 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:22.125 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T11:42:22.179 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T11:42:22.234 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T11:42:22.271 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T11:42:22.313 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T11:42:22.416 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T11:42:22.516 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T11:42:22.581 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T11:42:23.296 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T11:42:23.337 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T11:42:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T11:42:23.416 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T11:42:23.440 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 13/13 2024-09-18T11:42:24.245 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 13/13 2024-09-18T11:42:24.245 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 1/13 2024-09-18T11:42:24.245 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T11:42:24.245 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T11:42:24.245 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T11:42:24.245 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T11:42:24.245 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T11:42:24.245 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T11:42:24.245 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T11:42:24.245 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T11:42:24.246 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T11:42:24.246 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T11:42:24.246 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T11:42:24.592 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T11:42:24.592 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:24.592 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T11:42:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T11:42:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T11:42:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:24.594 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:24.876 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T11:42:25.457 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:20 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:25.591 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:25.592 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:25.592 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-18T11:42:25.593 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:25.593 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:25.593 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 124 k 2024-09-18T11:42:25.593 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-18T11:42:25.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T11:42:25.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T11:42:25.594 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T11:42:25.594 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T11:42:25.594 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T11:42:25.594 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:25.594 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:25.594 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:25.595 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-09-18T11:42:25.595 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:25.595 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 7.0 M 2024-09-18T11:42:25.595 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 33 M 2024-09-18T11:42:25.595 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:25.996 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1645.g7061ea6c.e 609 kB/s | 124 kB 00:00 2024-09-18T11:42:26.088 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.8 MB/s | 548 kB 00:00 2024-09-18T11:42:26.113 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-18T11:42:26.168 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 1.3 MB/s | 74 kB 00:00 2024-09-18T11:42:26.622 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 299 kB/s | 248 kB 00:00 2024-09-18T11:42:27.097 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-09-18T11:42:27.099 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:27.099 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.6 MB/s | 7.0 MB 00:01 2024-09-18T11:42:27.170 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:27.181 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:27.182 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:27.279 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:27.280 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:27.752 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T11:42:27.798 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T11:42:28.312 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T11:42:28.392 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T11:42:28.448 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T11:42:28.477 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 6/6 2024-09-18T11:42:29.214 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 6/6 2024-09-18T11:42:29.214 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/6 2024-09-18T11:42:29.214 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T11:42:29.214 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T11:42:29.214 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T11:42:29.214 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T11:42:29.499 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T11:42:29.499 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:29.499 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:29.499 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:42:29.499 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T11:42:29.499 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T11:42:29.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T11:42:29.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T11:42:29.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T11:42:29.500 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:29.500 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:29.755 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2024-09-18T11:42:30.334 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:25 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:30.466 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 813 k 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:30.467 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-09-18T11:42:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:30.468 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 893 k 2024-09-18T11:42:30.468 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.6 M 2024-09-18T11:42:30.468 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:30.873 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 627 kB/s | 80 kB 00:00 2024-09-18T11:42:31.014 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-17.2.7-1645.g7061ea6c.el9.x86_ 3.0 MB/s | 813 kB 00:00 2024-09-18T11:42:31.015 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:31.015 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.6 MB/s | 893 kB 00:00 2024-09-18T11:42:31.027 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:31.049 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:31.049 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:31.129 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:31.130 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:31.317 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:31.464 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T11:42:31.530 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/2 2024-09-18T11:42:32.211 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/2 2024-09-18T11:42:32.212 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/2 2024-09-18T11:42:32.485 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T11:42:32.485 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:32.485 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:32.485 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T11:42:32.485 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:32.485 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:32.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2024-09-18T11:42:33.329 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:28 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:33.465 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:33.466 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-18T11:42:33.466 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:33.466 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 270 k 2024-09-18T11:42:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:33.466 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:33.466 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:33.467 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-18T11:42:33.467 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:33.467 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 270 k 2024-09-18T11:42:33.467 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.3 M 2024-09-18T11:42:33.467 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:33.702 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-volume-17.2.7-1645.g7061ea6c.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-18T11:42:33.702 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:33.702 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-18T11:42:33.703 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:33.712 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:33.712 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:33.739 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:33.740 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:34.021 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:42:34.037 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:42:34.037 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:42:34.037 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T11:42:34.037 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:34.977 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:42:34.978 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:34.978 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:34.978 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:42:34.978 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:34.978 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:35.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2024-09-18T11:42:35.872 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:30 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:36.003 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:36.004 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:36.004 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-09-18T11:42:36.004 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:36.004 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:36.004 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 126 k 2024-09-18T11:42:36.004 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:36.004 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:36.004 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:36.004 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-18T11:42:36.004 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:36.005 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 126 k 2024-09-18T11:42:36.005 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 448 k 2024-09-18T11:42:36.005 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:36.206 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-17.2.7-1645.g7061ea6c.el9.x86_64 627 kB/s | 126 kB 00:00 2024-09-18T11:42:36.207 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:36.207 INFO:teuthology.orchestra.run.smithi071.stdout:Total 622 kB/s | 126 kB 00:00 2024-09-18T11:42:36.207 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:36.212 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:36.212 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:36.275 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:36.275 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:36.906 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:37.150 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:37.150 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:37.151 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:42:37.151 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:37.151 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:37.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2024-09-18T11:42:37.996 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:32 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:38.085 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:42:38.131 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:38.133 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-18T11:42:38.133 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:38.204 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2024-09-18T11:42:38.782 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:33 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:38.914 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:38.914 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:38.914 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 25 k 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 25 k 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 110 k 2024-09-18T11:42:38.915 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:39.067 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-17.2.7-1645.g7061ea6c.el9.x86_6 164 kB/s | 25 kB 00:00 2024-09-18T11:42:39.068 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:39.068 INFO:teuthology.orchestra.run.smithi071.stdout:Total 163 kB/s | 25 kB 00:00 2024-09-18T11:42:39.068 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:39.071 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:39.071 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:39.073 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:39.073 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:39.152 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:39.247 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:39.619 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:39.862 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:39.863 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:39.863 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:39.863 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:42:39.863 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:39.863 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:40.066 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2024-09-18T11:42:40.647 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:35 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:40.737 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:42:40.779 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:40.780 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-18T11:42:40.780 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:40.855 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2024-09-18T11:42:41.425 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:36 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:41.514 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:42:41.556 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:41.557 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-18T11:42:41.557 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:41.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2024-09-18T11:42:42.212 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:37 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:42.300 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:42:42.343 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:42.344 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-18T11:42:42.344 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:42.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2024-09-18T11:42:43.000 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:37 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:43.089 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:42:43.132 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:43.133 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-18T11:42:43.133 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:43.205 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2024-09-18T11:42:43.781 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:38 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:43.870 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:42:43.912 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:43.913 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-18T11:42:43.913 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:43.987 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2024-09-18T11:42:44.567 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:39 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:44.655 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:42:44.698 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:44.699 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-18T11:42:44.699 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:44.776 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2024-09-18T11:42:45.348 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:40 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:45.482 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:45.483 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:45.483 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-18T11:42:45.483 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:45.483 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:45.483 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 82 k 2024-09-18T11:42:45.483 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:45.483 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:45.483 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:45.483 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-18T11:42:45.484 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:45.484 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 82 k 2024-09-18T11:42:45.484 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 222 k 2024-09-18T11:42:45.484 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:45.680 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 418 kB/s | 82 kB 00:00 2024-09-18T11:42:45.680 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:45.681 INFO:teuthology.orchestra.run.smithi071.stdout:Total 415 kB/s | 82 kB 00:00 2024-09-18T11:42:45.681 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:45.687 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:45.687 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:45.750 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:45.751 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:45.908 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:46.016 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:46.389 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:46.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:46.629 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:46.629 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:46.629 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:42:46.629 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:46.629 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:46.895 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2024-09-18T11:42:47.482 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:47.620 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:47.621 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:47.621 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-09-18T11:42:47.621 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:47.621 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:47.621 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.9 M 2024-09-18T11:42:47.621 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:47.621 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:47.621 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:47.622 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-18T11:42:47.622 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:47.622 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.9 M 2024-09-18T11:42:47.622 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 12 M 2024-09-18T11:42:47.622 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:48.356 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 4.0 MB/s | 2.9 MB 00:00 2024-09-18T11:42:48.356 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:48.357 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.0 MB/s | 2.9 MB 00:00 2024-09-18T11:42:48.357 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:48.366 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:48.366 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:48.446 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:48.446 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:48.868 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:48.932 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:48.956 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:48.956 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:42:48.956 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T11:42:48.957 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T11:42:48.957 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T11:42:48.957 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:50.039 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:50.039 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:50.039 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:50.039 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:42:50.039 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:50.040 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:50.292 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2024-09-18T11:42:50.874 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:45 ago on Wed 18 Sep 2024 11:41:05 AM UTC. 2024-09-18T11:42:51.012 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-18T11:42:51.013 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:51.013 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-18T11:42:51.013 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:51.013 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-18T11:42:51.013 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 165 k 2024-09-18T11:42:51.014 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:51.014 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-18T11:42:51.014 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-18T11:42:51.014 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-18T11:42:51.014 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:51.014 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 165 k 2024-09-18T11:42:51.014 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 477 k 2024-09-18T11:42:51.014 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-18T11:42:51.467 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 364 kB/s | 165 kB 00:00 2024-09-18T11:42:51.468 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:42:51.468 INFO:teuthology.orchestra.run.smithi071.stdout:Total 363 kB/s | 165 kB 00:00 2024-09-18T11:42:51.468 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-18T11:42:51.475 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-18T11:42:51.475 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-18T11:42:51.538 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-18T11:42:51.539 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-18T11:42:51.699 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-18T11:42:51.788 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:52.268 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:52.541 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:42:52.541 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:52.541 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-18T11:42:52.541 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:42:52.541 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:52.541 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-18T11:42:52.783 DEBUG:teuthology.parallel:result is None 2024-09-18T11:42:52.784 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:42:52.923 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T11:42:52.944 INFO:teuthology.orchestra.run.smithi071.stdout:17.2.7-1645.g7061ea6c.el9 2024-09-18T11:42:52.944 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645.g7061ea6c.el9 2024-09-18T11:42:52.944 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645.g7061ea6c is installed. 2024-09-18T11:42:52.946 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T11:42:52.946 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:42:52.947 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T11:42:53.009 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T11:42:53.010 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:42:53.010 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T11:42:53.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T11:42:53.139 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T11:42:53.140 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:42:53.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T11:42:53.204 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T11:42:53.267 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T11:42:53.268 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:42:53.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T11:42:53.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T11:42:53.397 INFO:teuthology.run_tasks:Running task ceph... 2024-09-18T11:42:53.494 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-18T11:42:53.494 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2024-09-18T11:42:53.519 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-18T11:42:53.519 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T11:42:53.589 INFO:tasks.ceph:Creating extra log directories... 2024-09-18T11:42:53.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T11:42:53.663 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-18T11:42:53.664 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-18T11:42:53.664 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910981', 'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'description': 'rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7910981', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 4631, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'targets': {'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNfo/MHNdWKPVfFdQFiqWiwS/7ZKRIhzLElZ4yU03QpgvfyvEaXgOEVe4A8O+9UEbfIor5pFA1OQpEfFa5Z+w1A='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'create_rbd_pool': False, 'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'cluster': 'ceph'}}, {'ec_lost_unfound': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-18_09:13:00', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-18T11:42:53.664 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T11:42:53.710 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T11:42:53.775 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:42:53.775 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T11:42:53.832 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T11:42:53.833 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-09-18T11:42:53.888 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T11:42:53.888 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:42:53.888 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-18T11:42:53.888 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:42:53.888 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T11:42:53.888 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-18 11:42:49.207446684 +0000 2024-09-18T11:42:53.889 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-18 11:40:23.702574780 +0000 2024-09-18T11:42:53.889 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-18 11:40:23.702574780 +0000 2024-09-18T11:42:53.889 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-18 11:40:23.702574780 +0000 2024-09-18T11:42:53.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T11:42:53.952 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-18T11:42:53.952 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-18T11:42:53.952 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000203025 s, 2.5 MB/s 2024-09-18T11:42:53.953 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T11:42:54.010 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-09-18T11:42:54.064 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T11:42:54.064 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:42:54.065 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-18T11:42:54.065 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:42:54.065 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T11:42:54.065 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-18 11:42:49.208446666 +0000 2024-09-18T11:42:54.065 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-18 11:40:24.039568798 +0000 2024-09-18T11:42:54.065 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-18 11:40:24.039568798 +0000 2024-09-18T11:42:54.065 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-18 11:40:24.039568798 +0000 2024-09-18T11:42:54.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T11:42:54.127 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-18T11:42:54.128 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-18T11:42:54.128 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000198667 s, 2.6 MB/s 2024-09-18T11:42:54.129 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T11:42:54.185 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-09-18T11:42:54.241 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T11:42:54.241 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:42:54.241 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-18T11:42:54.242 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:42:54.242 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T11:42:54.242 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-18 11:42:49.208446666 +0000 2024-09-18T11:42:54.242 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-18 11:40:24.374562851 +0000 2024-09-18T11:42:54.242 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-18 11:40:24.374562851 +0000 2024-09-18T11:42:54.242 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-18 11:40:24.374562851 +0000 2024-09-18T11:42:54.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T11:42:54.304 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-18T11:42:54.305 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-18T11:42:54.305 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000187856 s, 2.7 MB/s 2024-09-18T11:42:54.306 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T11:42:54.361 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-09-18T11:42:54.417 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T11:42:54.417 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:42:54.417 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-18T11:42:54.417 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:42:54.417 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T11:42:54.417 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-18 11:42:49.208446666 +0000 2024-09-18T11:42:54.417 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-18 11:40:24.681557401 +0000 2024-09-18T11:42:54.417 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-18 11:40:24.681557401 +0000 2024-09-18T11:42:54.417 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-18 11:40:24.681557401 +0000 2024-09-18T11:42:54.418 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T11:42:54.480 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-18T11:42:54.481 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-18T11:42:54.481 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000172379 s, 3.0 MB/s 2024-09-18T11:42:54.482 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T11:42:54.539 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-18T11:42:54.539 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-18T11:42:54.539 INFO:tasks.ceph:Generating config... 2024-09-18T11:42:54.541 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-18T11:42:54.541 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-18T11:42:54.541 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-18T11:42:54.541 INFO:tasks.ceph:[global] ms type = async 2024-09-18T11:42:54.542 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-18T11:42:54.542 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-18T11:42:54.542 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-18T11:42:54.542 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-18T11:42:54.542 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-18T11:42:54.542 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-18T11:42:54.542 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-18T11:42:54.542 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-18T11:42:54.542 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-18T11:42:54.542 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-18T11:42:54.543 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-18T11:42:54.544 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-18T11:42:54.544 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-18T11:42:54.544 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-18T11:42:54.544 INFO:tasks.ceph:Setting up mon.a... 2024-09-18T11:42:54.544 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-18T11:42:54.620 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.keyring 2024-09-18T11:42:54.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-18T11:42:54.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T11:42:54.767 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.71'), ('mon.b', '[v2:172.21.15.71:3301,v1:172.21.15.71:6790]'), ('mon.c', '[v2:172.21.15.71:3302,v1:172.21.15.71:6791]')] 2024-09-18T11:42:54.767 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.71,[v2:172.21.15.71:3301,v1:172.21.15.71:6790],[v2:172.21.15.71:3302,v1:172.21.15.71:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-18T11:42:54.768 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:42:54.768 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T11:42:54.824 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.71 --addv b '[v2:172.21.15.71:3301,v1:172.21.15.71:6790]' --addv c '[v2:172.21.15.71:3302,v1:172.21.15.71:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: generated fsid c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:setting min_mon_release = octopus 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 0 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:fsid c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:last_changed 2024-09-18T11:42:54.904143+0000 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:created 2024-09-18T11:42:54.904143+0000 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:min_mon_release 15 (octopus) 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:election_strategy: 1 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:0: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.a 2024-09-18T11:42:54.905 INFO:teuthology.orchestra.run.smithi071.stdout:1: [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] mon.b 2024-09-18T11:42:54.906 INFO:teuthology.orchestra.run.smithi071.stdout:2: [v2:172.21.15.71:3302/0,v1:172.21.15.71:6791/0] mon.c 2024-09-18T11:42:54.906 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-18T11:42:54.907 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T11:42:54.963 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab... 2024-09-18T11:42:54.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T11:42:55.046 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2024-09-18T11:42:55.046 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2024-09-18T11:42:55.046 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2024-09-18T11:42:55.047 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2024-09-18T11:42:55.048 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2024-09-18T11:42:55.049 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = "172.21.15.71,[v2:172.21.15.71:3301,v1:172.21.15.71:6790],[v2:172.21.15.71:3302,v1:172.21.15.71:6791]" 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: mon client directed command retry = 5 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: mon election default strategy = 1 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject socket failures = 5000 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: ms type = async 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2024-09-18T11:42:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression algorithm = zlib 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression mode = aggressive 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore zero block detection = True 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 20 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 20 2024-09-18T11:42:55.052 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 10 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify cached snaps = True 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify missing on start = True 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock override recovery settings = True 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue = debug_random 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue cut off = debug_random 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.053 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-09-18T11:42:55.054 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.055 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub interval = 300 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2024-09-18T11:42:55.056 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2024-09-18T11:42:55.057 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T11:42:55.057 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T11:42:55.057 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2024-09-18T11:42:55.057 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.b] 2024-09-18T11:42:55.057 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.c] 2024-09-18T11:42:55.058 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-18T11:42:55.058 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-18T11:42:55.149 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-18T11:42:55.150 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:42:55.150 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-18T11:42:55.165 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:42:55.165 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-18T11:42:55.220 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T11:42:55.221 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:42:55.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T11:42:55.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T11:42:55.294 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:42:55.295 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:42:55.350 INFO:tasks.ceph:Setting up mon nodes... 2024-09-18T11:42:55.351 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-18T11:42:55.351 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T11:42:55.437 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T11:42:55.439 INFO:tasks.ceph:Setting up mds nodes... 2024-09-18T11:42:55.439 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-18T11:42:55.439 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-18T11:42:55.439 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-18T11:42:55.439 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-18T11:42:55.464 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-18T11:42:55.464 INFO:tasks.ceph:role: osd.0 2024-09-18T11:42:55.464 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T11:42:55.464 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T11:42:55.536 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:42:55.536 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:42:55.536 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:42:55.537 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T11:42:55.537 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:42:55.537 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:42:55.537 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:42:55.537 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T11:42:55.537 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:42:55.537 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:42:56.046 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-09-18T11:42:56.048 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-09-18T11:42:56.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-18T11:42:56.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-18T11:42:56.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-18T11:42:56.291 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-18T11:42:56.291 INFO:tasks.ceph:role: osd.1 2024-09-18T11:42:56.291 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T11:42:56.292 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T11:42:56.364 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:42:56.364 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:42:56.364 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:42:56.364 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T11:42:56.364 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:42:56.364 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:42:56.365 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:42:56.365 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T11:42:56.365 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:42:56.365 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:42:56.814 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-09-18T11:42:56.816 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-09-18T11:42:56.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-18T11:42:56.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-18T11:42:56.941 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-18T11:42:57.006 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-18T11:42:57.006 INFO:tasks.ceph:role: osd.2 2024-09-18T11:42:57.006 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T11:42:57.006 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T11:42:57.077 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:42:57.077 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:42:57.078 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:42:57.078 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T11:42:57.078 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:42:57.078 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:42:57.078 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:42:57.078 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T11:42:57.078 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:42:57.078 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:42:57.565 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-09-18T11:42:57.567 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-09-18T11:42:57.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-18T11:42:57.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-18T11:42:57.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-18T11:42:57.800 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-18T11:42:57.800 INFO:tasks.ceph:role: osd.3 2024-09-18T11:42:57.800 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T11:42:57.800 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-18T11:42:57.873 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:42:57.873 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:42:57.873 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:42:57.873 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T11:42:57.873 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:42:57.873 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:42:57.873 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:42:57.873 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T11:42:57.873 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:42:57.873 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:42:58.409 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-09-18T11:42:58.411 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-09-18T11:42:58.411 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-18T11:42:58.556 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-18T11:42:58.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:42:58.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:42:58.689+0000 7f3fe351e440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-18T11:42:58.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:42:58.689+0000 7f3fe351e440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-18T11:42:58.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:42:58.689+0000 7f3fe351e440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T11:42:58.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:42:58.689+0000 7f3fe351e440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T11:42:58.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:42:58.689+0000 7f3fe351e440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T11:42:58.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:42:58.690+0000 7f3fe351e440 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-18T11:42:59.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:42:59.215+0000 7f3fe351e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:42:59.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:42:59.985+0000 7f3fe351e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:00.749 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:00.747+0000 7f3fe351e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:01.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-18T11:43:01.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:01.658 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:01.656+0000 7fe63f00e440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-18T11:43:01.658 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:01.656+0000 7fe63f00e440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-18T11:43:01.658 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:01.656+0000 7fe63f00e440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T11:43:01.658 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:01.656+0000 7fe63f00e440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T11:43:01.658 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:01.656+0000 7fe63f00e440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T11:43:01.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:01.657+0000 7fe63f00e440 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-18T11:43:02.182 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:02.180+0000 7fe63f00e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:02.947 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:02.946+0000 7fe63f00e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:03.710 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:03.708+0000 7fe63f00e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:04.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T11:43:04.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:04.617+0000 7fefc5425440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-18T11:43:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:04.617+0000 7fefc5425440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-18T11:43:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:04.618+0000 7fefc5425440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T11:43:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:04.618+0000 7fefc5425440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T11:43:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:04.618+0000 7fefc5425440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T11:43:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:04.618+0000 7fefc5425440 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-18T11:43:05.147 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:05.145+0000 7fefc5425440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:05.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:05.911+0000 7fefc5425440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:06.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:06.672+0000 7fefc5425440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:07.465 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-18T11:43:07.531 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:07.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:07.612+0000 7f8d5251f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-18T11:43:07.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:07.613+0000 7f8d5251f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-18T11:43:07.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:07.613+0000 7f8d5251f440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T11:43:07.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:07.613+0000 7f8d5251f440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T11:43:07.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:07.613+0000 7f8d5251f440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T11:43:07.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:07.613+0000 7f8d5251f440 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-18T11:43:08.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:08.138+0000 7f8d5251f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:08.905 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:08.904+0000 7f8d5251f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:09.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:43:09.666+0000 7f8d5251f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:10.465 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-18T11:43:10.534 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-18T11:43:10.535 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:43:10.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-18T11:43:10.604 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:43:10.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-18T11:43:10.673 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:43:10.674 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-18T11:43:10.741 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:43:10.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-18T11:43:10.809 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:43:10.809 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-18T11:43:10.877 INFO:tasks.ceph:Adding keys to all mons... 2024-09-18T11:43:10.878 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T11:43:10.943 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2024-09-18T11:43:10.943 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQC/vOpmSXgFGhAAHBmWiWoez3sBQc6iriDpeQ== 2024-09-18T11:43:10.943 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-09-18T11:43:10.943 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDCvOpmqRYrKRAAuQcvXOUxNBu/sfhk1Nk15A== 2024-09-18T11:43:10.943 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-09-18T11:43:10.943 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDFvOpm+ywyJxAApe86vZjdFW7BiObFMD2pJA== 2024-09-18T11:43:10.943 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-09-18T11:43:10.943 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDIvOpm2hviJBAAr3JxQkKeM3aEfIPsNQwE8g== 2024-09-18T11:43:10.943 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2024-09-18T11:43:10.943 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDLvOpm54WYJBAAL2TASGkbZXDsQTQDpQl42w== 2024-09-18T11:43:10.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T11:43:11.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:43:11.081 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:43:11.126 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:43:11.172 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:43:11.218 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-18T11:43:11.218 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-18T11:43:11.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T11:43:11.333 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-18T11:43:11.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-18T11:43:11.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T11:43:11.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-18T11:43:11.536 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-18T11:43:11.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T11:43:11.692 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-18T11:43:11.716 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:11.772 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-18T11:43:11.772 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T11:43:11.772 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T11:43:11.814 INFO:tasks.ceph.mon.a:Started 2024-09-18T11:43:11.814 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T11:43:11.814 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T11:43:11.816 INFO:tasks.ceph.mon.b:Started 2024-09-18T11:43:11.816 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T11:43:11.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T11:43:11.818 INFO:tasks.ceph.mon.c:Started 2024-09-18T11:43:11.818 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph config set mgr mgr_pool false --force 2024-09-18T11:43:22.017 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-18T11:43:22.018 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-18T11:43:22.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-18T11:43:22.020 INFO:tasks.ceph.mgr.x:Started 2024-09-18T11:43:22.020 INFO:tasks.ceph:Setting crush tunables to default 2024-09-18T11:43:22.021 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd crush tunables default 2024-09-18T11:43:22.210 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:22.208+0000 7fae5c721ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-18T11:43:22.215 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2024-09-18T11:43:22.226 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-18T11:43:22.226 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:43:22.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-18T11:43:22.250 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:43:22.250 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-18T11:43:22.252 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:22.250+0000 7fae5c721ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-18T11:43:22.275 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:43:22.276 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-18T11:43:22.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:22.302+0000 7fae5c721ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-18T11:43:22.328 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-18T11:43:22.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-18T11:43:22.393 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 182a21f6-a1df-455b-ac1c-b1a90ea63d37 0 2024-09-18T11:43:22.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:22.440+0000 7fae5c721ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-18T11:43:22.554 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:22.553+0000 7fae5c721ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-18T11:43:22.609 INFO:teuthology.orchestra.run.smithi071.stdout:0 2024-09-18T11:43:22.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 9a5cd690-48d3-4692-890f-ac01f7d42f93 1 2024-09-18T11:43:22.834 INFO:teuthology.orchestra.run.smithi071.stdout:1 2024-09-18T11:43:22.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:22.835+0000 7fae5c721ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-18T11:43:22.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new b66ec834-18a9-428d-9af7-657a268cb3b9 2 2024-09-18T11:43:22.899 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:22.897+0000 7fae5c721ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-18T11:43:23.019 INFO:teuthology.orchestra.run.smithi071.stdout:2 2024-09-18T11:43:23.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8 3 2024-09-18T11:43:23.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:23.038+0000 7fae5c721ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-18T11:43:23.207 INFO:teuthology.orchestra.run.smithi071.stdout:3 2024-09-18T11:43:23.217 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-18T11:43:23.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-18T11:43:23.260 INFO:tasks.ceph.osd.0:Started 2024-09-18T11:43:23.260 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-18T11:43:23.260 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-18T11:43:23.262 INFO:tasks.ceph.osd.1:Started 2024-09-18T11:43:23.262 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T11:43:23.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T11:43:23.264 INFO:tasks.ceph.osd.2:Started 2024-09-18T11:43:23.264 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-18T11:43:23.264 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-18T11:43:23.266 INFO:tasks.ceph.osd.3:Started 2024-09-18T11:43:23.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:43:23.453 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:23.453 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":6,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:43:23.196826+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:43:23.464 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-18T11:43:23.464 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-18T11:43:23.866 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T11:43:23.872 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-18T11:43:23.870+0000 7fc51e324440 -1 Falling back to public interface 2024-09-18T11:43:23.880 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:23.878+0000 7f409103b440 -1 Falling back to public interface 2024-09-18T11:43:23.880 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-09-18T11:43:23.878+0000 7fdee3409440 -1 Falling back to public interface 2024-09-18T11:43:23.881 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-09-18T11:43:23.878+0000 7f428b32f440 -1 Falling back to public interface 2024-09-18T11:43:23.881 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:23.879+0000 7f408f9fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2024-09-18T11:43:23.881 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:23.880+0000 7f408f9fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2024-09-18T11:43:23.883 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-09-18T11:43:23.881+0000 7fdee13fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6826/0: (98) Address already in use 2024-09-18T11:43:24.017 INFO:teuthology.misc.health.smithi071.stdout: 2024-09-18T11:43:24.017 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":6,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:43:23.196826+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:43:24.027 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-18T11:43:24.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.047+0000 7fae5c721ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-18T11:43:24.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.097+0000 7fae5c721ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-18T11:43:24.457 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.455+0000 7fae5c721ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-18T11:43:24.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.535+0000 7fae5c721ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-18T11:43:24.623 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.622+0000 7fae5c721ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-18T11:43:24.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.665+0000 7fae5c721ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-18T11:43:24.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.760+0000 7fae5c721ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-18T11:43:24.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.810+0000 7fae5c721ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-18T11:43:24.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.852+0000 7fae5c721ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-18T11:43:24.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.945+0000 7fae5c721ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-18T11:43:24.997 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:24.995+0000 7fae5c721ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-18T11:43:25.046 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:25.044+0000 7fae5c721ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-18T11:43:25.110 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:25.109+0000 7fae5c721ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-18T11:43:25.152 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:25.150+0000 7fae5c721ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-18T11:43:25.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:25.192+0000 7fae5c721ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-18T11:43:25.800 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:25.798+0000 7fae5c721ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-18T11:43:26.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:26.815+0000 7fae41635640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-18T11:43:26.818 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-18T11:43:26.816+0000 7fae41635640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-18T11:43:27.403 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-09-18T11:43:27.402+0000 7fdee3409440 -1 osd.2 0 log_to_monitors true 2024-09-18T11:43:27.407 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-09-18T11:43:27.405+0000 7f428b32f440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:43:27.407 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-18T11:43:27.406+0000 7fc51e324440 -1 osd.3 0 log_to_monitors true 2024-09-18T11:43:27.409 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:27.408+0000 7f409103b440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:43:28.825 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:28.823+0000 7f408c1f6640 -1 osd.0 0 waiting for initial osdmap 2024-09-18T11:43:28.826 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-18T11:43:28.824+0000 7fc518df6640 -1 osd.3 0 waiting for initial osdmap 2024-09-18T11:43:28.826 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-09-18T11:43:28.825+0000 7fdeddbf6640 -1 osd.2 0 waiting for initial osdmap 2024-09-18T11:43:28.829 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-09-18T11:43:28.827+0000 7f4285df6640 -1 osd.1 0 waiting for initial osdmap 2024-09-18T11:43:28.830 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:28.828+0000 7f408908c640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:43:28.830 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-18T11:43:28.828+0000 7fc515c8c640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:43:28.830 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-09-18T11:43:28.828+0000 7fdedabf0640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:43:28.832 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-09-18T11:43:28.831+0000 7f4282ddf640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:43:30.430 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T11:43:30.692 INFO:teuthology.misc.health.smithi071.stdout: 2024-09-18T11:43:30.692 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":9,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:43:29.822230+0000","last_up_change":"2024-09-18T11:43:29.822230+0000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":33273},{"type":"v1","addr":"172.21.15.71:6813","nonce":33273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6817","nonce":33273},{"type":"v1","addr":"172.21.15.71:6820","nonce":33273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6829","nonce":33273},{"type":"v1","addr":"172.21.15.71:6831","nonce":33273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":33273},{"type":"v1","addr":"172.21.15.71:6826","nonce":33273}]},"public_addr":"172.21.15.71:6813/33273","cluster_addr":"172.21.15.71:6820/33273","heartbeat_back_addr":"172.21.15.71:6831/33273","heartbeat_front_addr":"172.21.15.71:6826/33273","state":["exists","up"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":33274},{"type":"v1","addr":"172.21.15.71:6812","nonce":33274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":33274},{"type":"v1","addr":"172.21.15.71:6816","nonce":33274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6825","nonce":33274},{"type":"v1","addr":"172.21.15.71:6828","nonce":33274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":33274},{"type":"v1","addr":"172.21.15.71:6822","nonce":33274}]},"public_addr":"172.21.15.71:6812/33274","cluster_addr":"172.21.15.71:6816/33274","heartbeat_back_addr":"172.21.15.71:6828/33274","heartbeat_front_addr":"172.21.15.71:6822/33274","state":["exists","up"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":33275},{"type":"v1","addr":"172.21.15.71:6811","nonce":33275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":33275},{"type":"v1","addr":"172.21.15.71:6818","nonce":33275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6827","nonce":33275},{"type":"v1","addr":"172.21.15.71:6830","nonce":33275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":33275},{"type":"v1","addr":"172.21.15.71:6823","nonce":33275}]},"public_addr":"172.21.15.71:6811/33275","cluster_addr":"172.21.15.71:6818/33275","heartbeat_back_addr":"172.21.15.71:6830/33275","heartbeat_front_addr":"172.21.15.71:6823/33275","state":["exists","up"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":33272},{"type":"v1","addr":"172.21.15.71:6801","nonce":33272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":33272},{"type":"v1","addr":"172.21.15.71:6803","nonce":33272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":33272},{"type":"v1","addr":"172.21.15.71:6807","nonce":33272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":33272},{"type":"v1","addr":"172.21.15.71:6805","nonce":33272}]},"public_addr":"172.21.15.71:6801/33272","cluster_addr":"172.21.15.71:6803/33272","heartbeat_back_addr":"172.21.15.71:6807/33272","heartbeat_front_addr":"172.21.15.71:6805/33272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:43:30.703 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-18T11:43:30.703 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-18T11:43:30.703 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-18T11:43:30.703 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-18T11:43:30.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-18T11:43:30.704 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-18T11:43:31.009 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:31.020 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":4,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":32897},{"type":"v1","addr":"172.21.15.71:6833","nonce":32897}]},"active_addr":"172.21.15.71:6833/32897","active_change":"2024-09-18T11:43:25.801173+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":1332276138}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":165853243}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":2505087722}]}]} 2024-09-18T11:43:31.022 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-18T11:43:31.022 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-18T11:43:31.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:43:31.282 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:31.283 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":10,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:43:30.829642+0000","last_up_change":"2024-09-18T11:43:29.822230+0000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":33273},{"type":"v1","addr":"172.21.15.71:6813","nonce":33273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6817","nonce":33273},{"type":"v1","addr":"172.21.15.71:6820","nonce":33273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6829","nonce":33273},{"type":"v1","addr":"172.21.15.71:6831","nonce":33273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":33273},{"type":"v1","addr":"172.21.15.71:6826","nonce":33273}]},"public_addr":"172.21.15.71:6813/33273","cluster_addr":"172.21.15.71:6820/33273","heartbeat_back_addr":"172.21.15.71:6831/33273","heartbeat_front_addr":"172.21.15.71:6826/33273","state":["exists","up"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":33274},{"type":"v1","addr":"172.21.15.71:6812","nonce":33274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":33274},{"type":"v1","addr":"172.21.15.71:6816","nonce":33274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6825","nonce":33274},{"type":"v1","addr":"172.21.15.71:6828","nonce":33274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":33274},{"type":"v1","addr":"172.21.15.71:6822","nonce":33274}]},"public_addr":"172.21.15.71:6812/33274","cluster_addr":"172.21.15.71:6816/33274","heartbeat_back_addr":"172.21.15.71:6828/33274","heartbeat_front_addr":"172.21.15.71:6822/33274","state":["exists","up"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":33275},{"type":"v1","addr":"172.21.15.71:6811","nonce":33275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":33275},{"type":"v1","addr":"172.21.15.71:6818","nonce":33275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6827","nonce":33275},{"type":"v1","addr":"172.21.15.71:6830","nonce":33275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":33275},{"type":"v1","addr":"172.21.15.71:6823","nonce":33275}]},"public_addr":"172.21.15.71:6811/33275","cluster_addr":"172.21.15.71:6818/33275","heartbeat_back_addr":"172.21.15.71:6830/33275","heartbeat_front_addr":"172.21.15.71:6823/33275","state":["exists","up"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":33272},{"type":"v1","addr":"172.21.15.71:6801","nonce":33272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":33272},{"type":"v1","addr":"172.21.15.71:6803","nonce":33272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":33272},{"type":"v1","addr":"172.21.15.71:6807","nonce":33272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":33272},{"type":"v1","addr":"172.21.15.71:6805","nonce":33272}]},"public_addr":"172.21.15.71:6801/33272","cluster_addr":"172.21.15.71:6803/33272","heartbeat_back_addr":"172.21.15.71:6807/33272","heartbeat_front_addr":"172.21.15.71:6805/33272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.447050+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.374310+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.355034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.394307+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:43:31.294 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-18T11:43:31.295 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:43:31.554 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:31.554 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":10,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:43:30.829642+0000","last_up_change":"2024-09-18T11:43:29.822230+0000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":33273},{"type":"v1","addr":"172.21.15.71:6813","nonce":33273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6817","nonce":33273},{"type":"v1","addr":"172.21.15.71:6820","nonce":33273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6829","nonce":33273},{"type":"v1","addr":"172.21.15.71:6831","nonce":33273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":33273},{"type":"v1","addr":"172.21.15.71:6826","nonce":33273}]},"public_addr":"172.21.15.71:6813/33273","cluster_addr":"172.21.15.71:6820/33273","heartbeat_back_addr":"172.21.15.71:6831/33273","heartbeat_front_addr":"172.21.15.71:6826/33273","state":["exists","up"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":33274},{"type":"v1","addr":"172.21.15.71:6812","nonce":33274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":33274},{"type":"v1","addr":"172.21.15.71:6816","nonce":33274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6825","nonce":33274},{"type":"v1","addr":"172.21.15.71:6828","nonce":33274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":33274},{"type":"v1","addr":"172.21.15.71:6822","nonce":33274}]},"public_addr":"172.21.15.71:6812/33274","cluster_addr":"172.21.15.71:6816/33274","heartbeat_back_addr":"172.21.15.71:6828/33274","heartbeat_front_addr":"172.21.15.71:6822/33274","state":["exists","up"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":33275},{"type":"v1","addr":"172.21.15.71:6811","nonce":33275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":33275},{"type":"v1","addr":"172.21.15.71:6818","nonce":33275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6827","nonce":33275},{"type":"v1","addr":"172.21.15.71:6830","nonce":33275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":33275},{"type":"v1","addr":"172.21.15.71:6823","nonce":33275}]},"public_addr":"172.21.15.71:6811/33275","cluster_addr":"172.21.15.71:6818/33275","heartbeat_back_addr":"172.21.15.71:6830/33275","heartbeat_front_addr":"172.21.15.71:6823/33275","state":["exists","up"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":33272},{"type":"v1","addr":"172.21.15.71:6801","nonce":33272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":33272},{"type":"v1","addr":"172.21.15.71:6803","nonce":33272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":33272},{"type":"v1","addr":"172.21.15.71:6807","nonce":33272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":33272},{"type":"v1","addr":"172.21.15.71:6805","nonce":33272}]},"public_addr":"172.21.15.71:6801/33272","cluster_addr":"172.21.15.71:6803/33272","heartbeat_back_addr":"172.21.15.71:6807/33272","heartbeat_front_addr":"172.21.15.71:6805/33272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.447050+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.374310+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.355034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.394307+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:43:31.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:43:31.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T11:43:31.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T11:43:31.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T11:43:31.703 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2024-09-18T11:43:31.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:43:31.705 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2024-09-18T11:43:31.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:43:31.707 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2024-09-18T11:43:31.707 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T11:43:31.709 INFO:teuthology.orchestra.run.smithi071.stdout:38654705667 2024-09-18T11:43:31.709 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:43:31.995 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2024-09-18T11:43:31.995 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2024-09-18T11:43:32.009 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.2 2024-09-18T11:43:32.010 DEBUG:teuthology.parallel:result is None 2024-09-18T11:43:32.010 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.1 2024-09-18T11:43:32.010 DEBUG:teuthology.parallel:result is None 2024-09-18T11:43:32.031 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2024-09-18T11:43:32.043 INFO:teuthology.orchestra.run.smithi071.stdout:38654705667 2024-09-18T11:43:32.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.3 2024-09-18T11:43:32.044 DEBUG:teuthology.parallel:result is None 2024-09-18T11:43:32.055 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.0 2024-09-18T11:43:32.055 DEBUG:teuthology.parallel:result is None 2024-09-18T11:43:32.055 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T11:43:32.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:43:32.350 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:32.351 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:43:32.365 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-09-18T11:43:31.814077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":1189768,"kb_used_data":344,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297592,"statfs":{"total":386547056640,"available":385328734208,"internally_reserved":0,"allocated":352256,"data_stored":77639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-09-18T11:43:32.366 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:43:32.620 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:32.621 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:43:32.632 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-09-18T11:43:31.814077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":1189768,"kb_used_data":344,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297592,"statfs":{"total":386547056640,"available":385328734208,"internally_reserved":0,"allocated":352256,"data_stored":77639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-09-18T11:43:32.633 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T11:43:32.633 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-18T11:43:32.633 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-18T11:43:32.633 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-18T11:43:32.923 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:32.923 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T11:43:32.935 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-18T11:43:32.935 INFO:teuthology.run_tasks:Running task ec_lost_unfound... 2024-09-18T11:43:32.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:43:33.227 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:33.239 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":10,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:43:30.829642+0000","last_up_change":"2024-09-18T11:43:29.822230+0000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":33273},{"type":"v1","addr":"172.21.15.71:6813","nonce":33273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6817","nonce":33273},{"type":"v1","addr":"172.21.15.71:6820","nonce":33273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6829","nonce":33273},{"type":"v1","addr":"172.21.15.71:6831","nonce":33273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":33273},{"type":"v1","addr":"172.21.15.71:6826","nonce":33273}]},"public_addr":"172.21.15.71:6813/33273","cluster_addr":"172.21.15.71:6820/33273","heartbeat_back_addr":"172.21.15.71:6831/33273","heartbeat_front_addr":"172.21.15.71:6826/33273","state":["exists","up"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":33274},{"type":"v1","addr":"172.21.15.71:6812","nonce":33274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":33274},{"type":"v1","addr":"172.21.15.71:6816","nonce":33274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6825","nonce":33274},{"type":"v1","addr":"172.21.15.71:6828","nonce":33274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":33274},{"type":"v1","addr":"172.21.15.71:6822","nonce":33274}]},"public_addr":"172.21.15.71:6812/33274","cluster_addr":"172.21.15.71:6816/33274","heartbeat_back_addr":"172.21.15.71:6828/33274","heartbeat_front_addr":"172.21.15.71:6822/33274","state":["exists","up"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":33275},{"type":"v1","addr":"172.21.15.71:6811","nonce":33275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":33275},{"type":"v1","addr":"172.21.15.71:6818","nonce":33275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6827","nonce":33275},{"type":"v1","addr":"172.21.15.71:6830","nonce":33275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":33275},{"type":"v1","addr":"172.21.15.71:6823","nonce":33275}]},"public_addr":"172.21.15.71:6811/33275","cluster_addr":"172.21.15.71:6818/33275","heartbeat_back_addr":"172.21.15.71:6830/33275","heartbeat_front_addr":"172.21.15.71:6823/33275","state":["exists","up"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":33272},{"type":"v1","addr":"172.21.15.71:6801","nonce":33272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":33272},{"type":"v1","addr":"172.21.15.71:6803","nonce":33272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":33272},{"type":"v1","addr":"172.21.15.71:6807","nonce":33272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":33272},{"type":"v1","addr":"172.21.15.71:6805","nonce":33272}]},"public_addr":"172.21.15.71:6801/33272","cluster_addr":"172.21.15.71:6803/33272","heartbeat_back_addr":"172.21.15.71:6807/33272","heartbeat_front_addr":"172.21.15.71:6805/33272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.447050+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.374310+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.355034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.394307+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:43:33.241 INFO:tasks.ec_lost_unfound.ceph_manager:[] 2024-09-18T11:43:33.241 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-18T11:43:33.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:43:33.495 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:33.495 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:43:33.506 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-09-18T11:43:31.814077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":1189768,"kb_used_data":344,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297592,"statfs":{"total":386547056640,"available":385328734208,"internally_reserved":0,"allocated":352256,"data_stored":77639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-09-18T11:43:33.507 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:43:33.761 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:33.761 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:43:33.772 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-09-18T11:43:31.814077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":1189768,"kb_used_data":344,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297592,"statfs":{"total":386547056640,"available":385328734208,"internally_reserved":0,"allocated":352256,"data_stored":77639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-09-18T11:43:33.773 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-18T11:43:33.773 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lost_unfound k=2 m=2 crush-failure-domain=osd 2024-09-18T11:43:34.880 INFO:tasks.ec_lost_unfound.ceph_manager:creating pool_name unique_pool_0 2024-09-18T11:43:34.881 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lost_unfound 2024-09-18T11:43:36.875 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'unique_pool_0' created 2024-09-18T11:43:36.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2024-09-18T11:43:37.877 INFO:teuthology.orchestra.run.smithi071.stderr:set pool 1 min_size to 2 2024-09-18T11:43:37.895 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-18T11:43:38.878 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-18T11:43:39.911 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2024-09-18T11:43:39.912 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2024-09-18T11:43:39.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:43:39.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T11:43:40.098 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2024-09-18T11:43:40.099 INFO:teuthology.orchestra.run.smithi071.stdout:38654705668 2024-09-18T11:43:40.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:43:40.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:43:40.368 INFO:teuthology.orchestra.run.smithi071.stdout:38654705667 2024-09-18T11:43:40.368 INFO:teuthology.orchestra.run.smithi071.stdout:38654705668 2024-09-18T11:43:40.381 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705668 got 38654705667 for osd.1 2024-09-18T11:43:40.382 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705669 got 38654705668 for osd.0 2024-09-18T11:43:41.384 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:43:41.385 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:43:41.661 INFO:teuthology.orchestra.run.smithi071.stdout:38654705668 2024-09-18T11:43:41.665 INFO:teuthology.orchestra.run.smithi071.stdout:38654705667 2024-09-18T11:43:41.674 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705669 got 38654705668 for osd.0 2024-09-18T11:43:41.678 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705668 got 38654705667 for osd.1 2024-09-18T11:43:42.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:43:42.679 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:43:42.981 INFO:teuthology.orchestra.run.smithi071.stdout:38654705670 2024-09-18T11:43:42.994 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705669 got 38654705670 for osd.0 2024-09-18T11:43:42.994 DEBUG:teuthology.parallel:result is None 2024-09-18T11:43:43.015 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2024-09-18T11:43:43.027 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705668 got 38654705669 for osd.1 2024-09-18T11:43:43.027 DEBUG:teuthology.parallel:result is None 2024-09-18T11:43:43.027 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-18T11:43:43.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:43:43.323 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:43.324 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:43:43.335 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-18T11:43:41.815616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189984,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297376,"statfs":{"total":386547056640,"available":385328513024,"internally_reserved":0,"allocated":573440,"data_stored":187864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898327+0000","last_change":"2024-09-18T11:43:38.892890+0000","last_active":"2024-09-18T11:43:38.898327+0000","last_peered":"2024-09-18T11:43:38.898327+0000","last_clean":"2024-09-18T11:43:38.898327+0000","last_became_active":"2024-09-18T11:43:38.892493+0000","last_became_peered":"2024-09-18T11:43:38.892493+0000","last_unstale":"2024-09-18T11:43:38.898327+0000","last_undegraded":"2024-09-18T11:43:38.898327+0000","last_fullsized":"2024-09-18T11:43:38.898327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:32.748874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.897625+0000","last_change":"2024-09-18T11:43:38.889740+0000","last_active":"2024-09-18T11:43:38.897625+0000","last_peered":"2024-09-18T11:43:38.897625+0000","last_clean":"2024-09-18T11:43:38.897625+0000","last_became_active":"2024-09-18T11:43:38.889346+0000","last_became_peered":"2024-09-18T11:43:38.889346+0000","last_unstale":"2024-09-18T11:43:38.897625+0000","last_undegraded":"2024-09-18T11:43:38.897625+0000","last_fullsized":"2024-09-18T11:43:38.897625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:43:51.002757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.895531+0000","last_change":"2024-09-18T11:43:38.889956+0000","last_active":"2024-09-18T11:43:38.895531+0000","last_peered":"2024-09-18T11:43:38.895531+0000","last_clean":"2024-09-18T11:43:38.895531+0000","last_became_active":"2024-09-18T11:43:38.889587+0000","last_became_peered":"2024-09-18T11:43:38.889587+0000","last_unstale":"2024-09-18T11:43:38.895531+0000","last_undegraded":"2024-09-18T11:43:38.895531+0000","last_fullsized":"2024-09-18T11:43:38.895531+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:30:52.309731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898323+0000","last_change":"2024-09-18T11:43:38.891760+0000","last_active":"2024-09-18T11:43:38.898323+0000","last_peered":"2024-09-18T11:43:38.898323+0000","last_clean":"2024-09-18T11:43:38.898323+0000","last_became_active":"2024-09-18T11:43:38.890024+0000","last_became_peered":"2024-09-18T11:43:38.890024+0000","last_unstale":"2024-09-18T11:43:38.898323+0000","last_undegraded":"2024-09-18T11:43:38.898323+0000","last_fullsized":"2024-09-18T11:43:38.898323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:10:41.978045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898158+0000","last_change":"2024-09-18T11:43:38.893954+0000","last_active":"2024-09-18T11:43:38.898158+0000","last_peered":"2024-09-18T11:43:38.898158+0000","last_clean":"2024-09-18T11:43:38.898158+0000","last_became_active":"2024-09-18T11:43:38.893572+0000","last_became_peered":"2024-09-18T11:43:38.893572+0000","last_unstale":"2024-09-18T11:43:38.898158+0000","last_undegraded":"2024-09-18T11:43:38.898158+0000","last_fullsized":"2024-09-18T11:43:38.898158+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:36:38.481036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898327+0000","last_change":"2024-09-18T11:43:38.893185+0000","last_active":"2024-09-18T11:43:38.898327+0000","last_peered":"2024-09-18T11:43:38.898327+0000","last_clean":"2024-09-18T11:43:38.898327+0000","last_became_active":"2024-09-18T11:43:38.892767+0000","last_became_peered":"2024-09-18T11:43:38.892767+0000","last_unstale":"2024-09-18T11:43:38.898327+0000","last_undegraded":"2024-09-18T11:43:38.898327+0000","last_fullsized":"2024-09-18T11:43:38.898327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:40:31.694447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.893020+0000","last_change":"2024-09-18T11:43:38.891420+0000","last_active":"2024-09-18T11:43:38.893020+0000","last_peered":"2024-09-18T11:43:38.893020+0000","last_clean":"2024-09-18T11:43:38.893020+0000","last_became_active":"2024-09-18T11:43:38.891032+0000","last_became_peered":"2024-09-18T11:43:38.891032+0000","last_unstale":"2024-09-18T11:43:38.893020+0000","last_undegraded":"2024-09-18T11:43:38.893020+0000","last_fullsized":"2024-09-18T11:43:38.893020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:55.895683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898207+0000","last_change":"2024-09-18T11:43:38.890848+0000","last_active":"2024-09-18T11:43:38.898207+0000","last_peered":"2024-09-18T11:43:38.898207+0000","last_clean":"2024-09-18T11:43:38.898207+0000","last_became_active":"2024-09-18T11:43:38.890379+0000","last_became_peered":"2024-09-18T11:43:38.890379+0000","last_unstale":"2024-09-18T11:43:38.898207+0000","last_undegraded":"2024-09-18T11:43:38.898207+0000","last_fullsized":"2024-09-18T11:43:38.898207+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:12:27.009802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"16'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T11:43:39.961455+0000","last_change":"2024-09-18T11:43:38.888825+0000","last_active":"2024-09-18T11:43:39.961455+0000","last_peered":"2024-09-18T11:43:39.961455+0000","last_clean":"2024-09-18T11:43:39.961455+0000","last_became_active":"2024-09-18T11:43:38.888434+0000","last_became_peered":"2024-09-18T11:43:38.888434+0000","last_unstale":"2024-09-18T11:43:39.961455+0000","last_undegraded":"2024-09-18T11:43:39.961455+0000","last_fullsized":"2024-09-18T11:43:39.961455+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:27:25.919184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.895512+0000","last_change":"2024-09-18T11:43:38.889914+0000","last_active":"2024-09-18T11:43:38.895512+0000","last_peered":"2024-09-18T11:43:38.895512+0000","last_clean":"2024-09-18T11:43:38.895512+0000","last_became_active":"2024-09-18T11:43:38.889538+0000","last_became_peered":"2024-09-18T11:43:38.889538+0000","last_unstale":"2024-09-18T11:43:38.895512+0000","last_undegraded":"2024-09-18T11:43:38.895512+0000","last_fullsized":"2024-09-18T11:43:38.895512+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:19.859564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.899354+0000","last_change":"2024-09-18T11:43:38.897687+0000","last_active":"2024-09-18T11:43:38.899354+0000","last_peered":"2024-09-18T11:43:38.899354+0000","last_clean":"2024-09-18T11:43:38.899354+0000","last_became_active":"2024-09-18T11:43:38.897340+0000","last_became_peered":"2024-09-18T11:43:38.897340+0000","last_unstale":"2024-09-18T11:43:38.899354+0000","last_undegraded":"2024-09-18T11:43:38.899354+0000","last_fullsized":"2024-09-18T11:43:38.899354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:23:18.548387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.897742+0000","last_change":"2024-09-18T11:43:38.889930+0000","last_active":"2024-09-18T11:43:38.897742+0000","last_peered":"2024-09-18T11:43:38.897742+0000","last_clean":"2024-09-18T11:43:38.897742+0000","last_became_active":"2024-09-18T11:43:38.889487+0000","last_became_peered":"2024-09-18T11:43:38.889487+0000","last_unstale":"2024-09-18T11:43:38.897742+0000","last_undegraded":"2024-09-18T11:43:38.897742+0000","last_fullsized":"2024-09-18T11:43:38.897742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:17:28.677932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.897832+0000","last_change":"2024-09-18T11:43:38.889608+0000","last_active":"2024-09-18T11:43:38.897832+0000","last_peered":"2024-09-18T11:43:38.897832+0000","last_clean":"2024-09-18T11:43:38.897832+0000","last_became_active":"2024-09-18T11:43:38.888122+0000","last_became_peered":"2024-09-18T11:43:38.888122+0000","last_unstale":"2024-09-18T11:43:38.897832+0000","last_undegraded":"2024-09-18T11:43:38.897832+0000","last_fullsized":"2024-09-18T11:43:38.897832+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:34:09.972037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898137+0000","last_change":"2024-09-18T11:43:38.890561+0000","last_active":"2024-09-18T11:43:38.898137+0000","last_peered":"2024-09-18T11:43:38.898137+0000","last_clean":"2024-09-18T11:43:38.898137+0000","last_became_active":"2024-09-18T11:43:38.890201+0000","last_became_peered":"2024-09-18T11:43:38.890201+0000","last_unstale":"2024-09-18T11:43:38.898137+0000","last_undegraded":"2024-09-18T11:43:38.898137+0000","last_fullsized":"2024-09-18T11:43:38.898137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:09:08.786872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898365+0000","last_change":"2024-09-18T11:43:38.897004+0000","last_active":"2024-09-18T11:43:38.898365+0000","last_peered":"2024-09-18T11:43:38.898365+0000","last_clean":"2024-09-18T11:43:38.898365+0000","last_became_active":"2024-09-18T11:43:38.892354+0000","last_became_peered":"2024-09-18T11:43:38.892354+0000","last_unstale":"2024-09-18T11:43:38.898365+0000","last_undegraded":"2024-09-18T11:43:38.898365+0000","last_fullsized":"2024-09-18T11:43:38.898365+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:19:33.351846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898227+0000","last_change":"2024-09-18T11:43:38.891122+0000","last_active":"2024-09-18T11:43:38.898227+0000","last_peered":"2024-09-18T11:43:38.898227+0000","last_clean":"2024-09-18T11:43:38.898227+0000","last_became_active":"2024-09-18T11:43:38.889521+0000","last_became_peered":"2024-09-18T11:43:38.889521+0000","last_unstale":"2024-09-18T11:43:38.898227+0000","last_undegraded":"2024-09-18T11:43:38.898227+0000","last_fullsized":"2024-09-18T11:43:38.898227+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:36:11.632639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:43:43.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:43:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:43.591 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:43:43.603 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-18T11:43:41.815616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189984,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297376,"statfs":{"total":386547056640,"available":385328513024,"internally_reserved":0,"allocated":573440,"data_stored":187864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898327+0000","last_change":"2024-09-18T11:43:38.892890+0000","last_active":"2024-09-18T11:43:38.898327+0000","last_peered":"2024-09-18T11:43:38.898327+0000","last_clean":"2024-09-18T11:43:38.898327+0000","last_became_active":"2024-09-18T11:43:38.892493+0000","last_became_peered":"2024-09-18T11:43:38.892493+0000","last_unstale":"2024-09-18T11:43:38.898327+0000","last_undegraded":"2024-09-18T11:43:38.898327+0000","last_fullsized":"2024-09-18T11:43:38.898327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:32.748874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.897625+0000","last_change":"2024-09-18T11:43:38.889740+0000","last_active":"2024-09-18T11:43:38.897625+0000","last_peered":"2024-09-18T11:43:38.897625+0000","last_clean":"2024-09-18T11:43:38.897625+0000","last_became_active":"2024-09-18T11:43:38.889346+0000","last_became_peered":"2024-09-18T11:43:38.889346+0000","last_unstale":"2024-09-18T11:43:38.897625+0000","last_undegraded":"2024-09-18T11:43:38.897625+0000","last_fullsized":"2024-09-18T11:43:38.897625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:43:51.002757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.895531+0000","last_change":"2024-09-18T11:43:38.889956+0000","last_active":"2024-09-18T11:43:38.895531+0000","last_peered":"2024-09-18T11:43:38.895531+0000","last_clean":"2024-09-18T11:43:38.895531+0000","last_became_active":"2024-09-18T11:43:38.889587+0000","last_became_peered":"2024-09-18T11:43:38.889587+0000","last_unstale":"2024-09-18T11:43:38.895531+0000","last_undegraded":"2024-09-18T11:43:38.895531+0000","last_fullsized":"2024-09-18T11:43:38.895531+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:30:52.309731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898323+0000","last_change":"2024-09-18T11:43:38.891760+0000","last_active":"2024-09-18T11:43:38.898323+0000","last_peered":"2024-09-18T11:43:38.898323+0000","last_clean":"2024-09-18T11:43:38.898323+0000","last_became_active":"2024-09-18T11:43:38.890024+0000","last_became_peered":"2024-09-18T11:43:38.890024+0000","last_unstale":"2024-09-18T11:43:38.898323+0000","last_undegraded":"2024-09-18T11:43:38.898323+0000","last_fullsized":"2024-09-18T11:43:38.898323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:10:41.978045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898158+0000","last_change":"2024-09-18T11:43:38.893954+0000","last_active":"2024-09-18T11:43:38.898158+0000","last_peered":"2024-09-18T11:43:38.898158+0000","last_clean":"2024-09-18T11:43:38.898158+0000","last_became_active":"2024-09-18T11:43:38.893572+0000","last_became_peered":"2024-09-18T11:43:38.893572+0000","last_unstale":"2024-09-18T11:43:38.898158+0000","last_undegraded":"2024-09-18T11:43:38.898158+0000","last_fullsized":"2024-09-18T11:43:38.898158+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:36:38.481036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898327+0000","last_change":"2024-09-18T11:43:38.893185+0000","last_active":"2024-09-18T11:43:38.898327+0000","last_peered":"2024-09-18T11:43:38.898327+0000","last_clean":"2024-09-18T11:43:38.898327+0000","last_became_active":"2024-09-18T11:43:38.892767+0000","last_became_peered":"2024-09-18T11:43:38.892767+0000","last_unstale":"2024-09-18T11:43:38.898327+0000","last_undegraded":"2024-09-18T11:43:38.898327+0000","last_fullsized":"2024-09-18T11:43:38.898327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:40:31.694447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.893020+0000","last_change":"2024-09-18T11:43:38.891420+0000","last_active":"2024-09-18T11:43:38.893020+0000","last_peered":"2024-09-18T11:43:38.893020+0000","last_clean":"2024-09-18T11:43:38.893020+0000","last_became_active":"2024-09-18T11:43:38.891032+0000","last_became_peered":"2024-09-18T11:43:38.891032+0000","last_unstale":"2024-09-18T11:43:38.893020+0000","last_undegraded":"2024-09-18T11:43:38.893020+0000","last_fullsized":"2024-09-18T11:43:38.893020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:55.895683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898207+0000","last_change":"2024-09-18T11:43:38.890848+0000","last_active":"2024-09-18T11:43:38.898207+0000","last_peered":"2024-09-18T11:43:38.898207+0000","last_clean":"2024-09-18T11:43:38.898207+0000","last_became_active":"2024-09-18T11:43:38.890379+0000","last_became_peered":"2024-09-18T11:43:38.890379+0000","last_unstale":"2024-09-18T11:43:38.898207+0000","last_undegraded":"2024-09-18T11:43:38.898207+0000","last_fullsized":"2024-09-18T11:43:38.898207+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:12:27.009802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"16'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T11:43:39.961455+0000","last_change":"2024-09-18T11:43:38.888825+0000","last_active":"2024-09-18T11:43:39.961455+0000","last_peered":"2024-09-18T11:43:39.961455+0000","last_clean":"2024-09-18T11:43:39.961455+0000","last_became_active":"2024-09-18T11:43:38.888434+0000","last_became_peered":"2024-09-18T11:43:38.888434+0000","last_unstale":"2024-09-18T11:43:39.961455+0000","last_undegraded":"2024-09-18T11:43:39.961455+0000","last_fullsized":"2024-09-18T11:43:39.961455+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:27:25.919184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.895512+0000","last_change":"2024-09-18T11:43:38.889914+0000","last_active":"2024-09-18T11:43:38.895512+0000","last_peered":"2024-09-18T11:43:38.895512+0000","last_clean":"2024-09-18T11:43:38.895512+0000","last_became_active":"2024-09-18T11:43:38.889538+0000","last_became_peered":"2024-09-18T11:43:38.889538+0000","last_unstale":"2024-09-18T11:43:38.895512+0000","last_undegraded":"2024-09-18T11:43:38.895512+0000","last_fullsized":"2024-09-18T11:43:38.895512+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:19.859564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.899354+0000","last_change":"2024-09-18T11:43:38.897687+0000","last_active":"2024-09-18T11:43:38.899354+0000","last_peered":"2024-09-18T11:43:38.899354+0000","last_clean":"2024-09-18T11:43:38.899354+0000","last_became_active":"2024-09-18T11:43:38.897340+0000","last_became_peered":"2024-09-18T11:43:38.897340+0000","last_unstale":"2024-09-18T11:43:38.899354+0000","last_undegraded":"2024-09-18T11:43:38.899354+0000","last_fullsized":"2024-09-18T11:43:38.899354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:23:18.548387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.897742+0000","last_change":"2024-09-18T11:43:38.889930+0000","last_active":"2024-09-18T11:43:38.897742+0000","last_peered":"2024-09-18T11:43:38.897742+0000","last_clean":"2024-09-18T11:43:38.897742+0000","last_became_active":"2024-09-18T11:43:38.889487+0000","last_became_peered":"2024-09-18T11:43:38.889487+0000","last_unstale":"2024-09-18T11:43:38.897742+0000","last_undegraded":"2024-09-18T11:43:38.897742+0000","last_fullsized":"2024-09-18T11:43:38.897742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:17:28.677932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.897832+0000","last_change":"2024-09-18T11:43:38.889608+0000","last_active":"2024-09-18T11:43:38.897832+0000","last_peered":"2024-09-18T11:43:38.897832+0000","last_clean":"2024-09-18T11:43:38.897832+0000","last_became_active":"2024-09-18T11:43:38.888122+0000","last_became_peered":"2024-09-18T11:43:38.888122+0000","last_unstale":"2024-09-18T11:43:38.897832+0000","last_undegraded":"2024-09-18T11:43:38.897832+0000","last_fullsized":"2024-09-18T11:43:38.897832+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:34:09.972037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898137+0000","last_change":"2024-09-18T11:43:38.890561+0000","last_active":"2024-09-18T11:43:38.898137+0000","last_peered":"2024-09-18T11:43:38.898137+0000","last_clean":"2024-09-18T11:43:38.898137+0000","last_became_active":"2024-09-18T11:43:38.890201+0000","last_became_peered":"2024-09-18T11:43:38.890201+0000","last_unstale":"2024-09-18T11:43:38.898137+0000","last_undegraded":"2024-09-18T11:43:38.898137+0000","last_fullsized":"2024-09-18T11:43:38.898137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:09:08.786872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898365+0000","last_change":"2024-09-18T11:43:38.897004+0000","last_active":"2024-09-18T11:43:38.898365+0000","last_peered":"2024-09-18T11:43:38.898365+0000","last_clean":"2024-09-18T11:43:38.898365+0000","last_became_active":"2024-09-18T11:43:38.892354+0000","last_became_peered":"2024-09-18T11:43:38.892354+0000","last_unstale":"2024-09-18T11:43:38.898365+0000","last_undegraded":"2024-09-18T11:43:38.898365+0000","last_fullsized":"2024-09-18T11:43:38.898365+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:19:33.351846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T11:43:38.898227+0000","last_change":"2024-09-18T11:43:38.891122+0000","last_active":"2024-09-18T11:43:38.898227+0000","last_peered":"2024-09-18T11:43:38.898227+0000","last_clean":"2024-09-18T11:43:38.898227+0000","last_became_active":"2024-09-18T11:43:38.889521+0000","last_became_peered":"2024-09-18T11:43:38.889521+0000","last_unstale":"2024-09-18T11:43:38.898227+0000","last_undegraded":"2024-09-18T11:43:38.898227+0000","last_fullsized":"2024-09-18T11:43:38.898227+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:36:11.632639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:43:43.604 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-18T11:43:43.604 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-18T11:43:43.604 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-18T11:43:43.650 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-18T11:43:43.650 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-18T11:43:43.695 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_1 2024-09-18T11:43:43.695 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_1 2024-09-18T11:43:43.739 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-18T11:43:43.739 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-18T11:43:43.785 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-18T11:43:43.785 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-18T11:43:43.830 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_2 2024-09-18T11:43:43.830 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_2 2024-09-18T11:43:43.874 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-18T11:43:43.875 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-18T11:43:43.920 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-18T11:43:43.920 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-18T11:43:43.968 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_3 2024-09-18T11:43:43.968 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_3 2024-09-18T11:43:44.012 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-18T11:43:44.012 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-18T11:43:44.098 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-18T11:43:44.098 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-18T11:43:44.143 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_4 2024-09-18T11:43:44.143 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_4 2024-09-18T11:43:44.188 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-18T11:43:44.188 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-18T11:43:44.233 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-18T11:43:44.233 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-18T11:43:44.279 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_5 2024-09-18T11:43:44.279 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_5 2024-09-18T11:43:44.323 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-18T11:43:44.323 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-18T11:43:44.368 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-18T11:43:44.368 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-18T11:43:44.414 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_6 2024-09-18T11:43:44.414 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_6 2024-09-18T11:43:44.458 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-18T11:43:44.458 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-18T11:43:44.504 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-18T11:43:44.504 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-18T11:43:44.552 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_7 2024-09-18T11:43:44.553 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_7 2024-09-18T11:43:44.597 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-18T11:43:44.597 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-18T11:43:44.643 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-18T11:43:44.643 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-18T11:43:44.689 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_8 2024-09-18T11:43:44.690 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_8 2024-09-18T11:43:44.733 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-18T11:43:44.734 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-18T11:43:44.820 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-18T11:43:44.820 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-18T11:43:44.866 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_9 2024-09-18T11:43:44.866 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_9 2024-09-18T11:43:44.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-09-18T11:43:45.067 INFO:teuthology.orchestra.run.smithi071.stdout:{} 2024-09-18T11:43:45.067 INFO:teuthology.orchestra.run.smithi071.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-09-18T11:43:45.077 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-18T11:43:45.077 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:43:45.116 INFO:tasks.ceph.osd.0:Stopped 2024-09-18T11:43:45.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-09-18T11:43:45.426 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-09-18T11:43:45.424+0000 7f05047f1640 -1 mon.a@0(leader).osd e16 definitely_dead 0 2024-09-18T11:43:45.968 INFO:teuthology.orchestra.run.smithi071.stderr:marked down osd.0. 2024-09-18T11:43:45.986 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-18T11:43:45.986 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:43:46.022 INFO:tasks.ceph.osd.3:Stopped 2024-09-18T11:43:46.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-09-18T11:43:46.329 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-09-18T11:43:46.328+0000 7f05047f1640 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-18T11:43:46.976 INFO:teuthology.orchestra.run.smithi071.stderr:marked down osd.3. 2024-09-18T11:43:46.990 INFO:tasks.util.rados:rados -p unique_pool_0 put new_1 /etc/group 2024-09-18T11:43:46.990 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_1 /etc/group 2024-09-18T11:43:47.994 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-18T11:43:47.994 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-18T11:43:48.045 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-18T11:43:48.045 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-18T11:43:48.092 INFO:tasks.util.rados:rados -p unique_pool_0 put new_2 /etc/group 2024-09-18T11:43:48.093 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_2 /etc/group 2024-09-18T11:43:48.138 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-18T11:43:48.139 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-18T11:43:48.184 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-18T11:43:48.184 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-18T11:43:48.229 INFO:tasks.util.rados:rados -p unique_pool_0 put new_3 /etc/group 2024-09-18T11:43:48.229 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_3 /etc/group 2024-09-18T11:43:48.274 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-18T11:43:48.274 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-18T11:43:48.320 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-18T11:43:48.320 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-18T11:43:48.365 INFO:tasks.util.rados:rados -p unique_pool_0 put new_4 /etc/group 2024-09-18T11:43:48.365 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_4 /etc/group 2024-09-18T11:43:48.413 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-18T11:43:48.413 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-18T11:43:48.458 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-18T11:43:48.458 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-18T11:43:48.504 INFO:tasks.util.rados:rados -p unique_pool_0 put new_5 /etc/group 2024-09-18T11:43:48.504 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_5 /etc/group 2024-09-18T11:43:48.549 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-18T11:43:48.549 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-18T11:43:48.594 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-18T11:43:48.595 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-18T11:43:48.640 INFO:tasks.util.rados:rados -p unique_pool_0 put new_6 /etc/group 2024-09-18T11:43:48.640 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_6 /etc/group 2024-09-18T11:43:48.685 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-18T11:43:48.686 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-18T11:43:48.731 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-18T11:43:48.731 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-18T11:43:48.776 INFO:tasks.util.rados:rados -p unique_pool_0 put new_7 /etc/group 2024-09-18T11:43:48.777 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_7 /etc/group 2024-09-18T11:43:48.822 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-18T11:43:48.823 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-18T11:43:48.868 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-18T11:43:48.868 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-18T11:43:48.914 INFO:tasks.util.rados:rados -p unique_pool_0 put new_8 /etc/group 2024-09-18T11:43:48.914 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_8 /etc/group 2024-09-18T11:43:48.962 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-18T11:43:48.962 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-18T11:43:49.008 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-18T11:43:49.008 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-18T11:43:49.053 INFO:tasks.util.rados:rados -p unique_pool_0 put new_9 /etc/group 2024-09-18T11:43:49.053 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_9 /etc/group 2024-09-18T11:43:49.098 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-18T11:43:49.098 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-18T11:43:49.144 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-18T11:43:49.144 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-18T11:43:49.189 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-18T11:43:49.189 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:43:49.219 INFO:tasks.ceph.osd.1:Stopped 2024-09-18T11:43:49.219 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-18T11:43:49.528 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-09-18T11:43:49.527+0000 7f05047f1640 -1 mon.a@0(leader).osd e19 definitely_dead 0 2024-09-18T11:43:49.982 INFO:teuthology.orchestra.run.smithi071.stderr:marked down osd.1. 2024-09-18T11:43:49.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-09-18T11:43:50.985 INFO:teuthology.orchestra.run.smithi071.stderr:marked osd lost in epoch 20 2024-09-18T11:43:51.000 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-18T11:43:51.000 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-18T11:43:51.003 INFO:tasks.ceph.osd.0:Started 2024-09-18T11:43:51.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T11:43:51.099 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2024-09-18T11:43:51.099 INFO:teuthology.orchestra.run.smithi071.stderr:0 2024-09-18T11:43:51.099 INFO:teuthology.orchestra.run.smithi071.stderr:1 2024-09-18T11:43:51.099 INFO:teuthology.orchestra.run.smithi071.stderr:2 2024-09-18T11:43:51.099 INFO:teuthology.orchestra.run.smithi071.stderr:abort 2024-09-18T11:43:51.099 INFO:teuthology.orchestra.run.smithi071.stderr:assert 2024-09-18T11:43:51.099 INFO:teuthology.orchestra.run.smithi071.stderr:config diff 2024-09-18T11:43:51.099 INFO:teuthology.orchestra.run.smithi071.stderr:config diff get 2024-09-18T11:43:51.100 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2024-09-18T11:43:51.100 INFO:teuthology.orchestra.run.smithi071.stderr:config help [] 2024-09-18T11:43:51.100 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2024-09-18T11:43:51.100 INFO:teuthology.orchestra.run.smithi071.stderr:admin_socket: invalid command 2024-09-18T11:43:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T11:43:51.101 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-18T11:43:51.587 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:51.585+0000 7fca2648b440 -1 Falling back to public interface 2024-09-18T11:43:53.991 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:53.989+0000 7fca2648b440 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:43:54.748 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:54.746+0000 7fca2648b440 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:43:55.033 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:55.032+0000 7fca2648b440 -1 osd.0 16 log_to_monitors true 2024-09-18T11:43:56.021 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-09-18T11:43:56.019+0000 7fca1c488640 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:43:56.102 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T11:43:56.209 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:43:56.210 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-18T11:43:56.210 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-18T11:43:56.210 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:43:56.218 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.0 to be up 2024-09-18T11:43:56.218 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:43:56.478 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:56.479 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":22,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:43:55.997659+0000","last_up_change":"2024-09-18T11:43:49.974762+0000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-18T11:43:35.889679+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":33273},{"type":"v1","addr":"172.21.15.71:6813","nonce":33273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6817","nonce":33273},{"type":"v1","addr":"172.21.15.71:6820","nonce":33273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6829","nonce":33273},{"type":"v1","addr":"172.21.15.71:6831","nonce":33273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":33273},{"type":"v1","addr":"172.21.15.71:6826","nonce":33273}]},"public_addr":"172.21.15.71:6813/33273","cluster_addr":"172.21.15.71:6820/33273","heartbeat_back_addr":"172.21.15.71:6831/33273","heartbeat_front_addr":"172.21.15.71:6826/33273","state":["exists"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":33274},{"type":"v1","addr":"172.21.15.71:6812","nonce":33274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":33274},{"type":"v1","addr":"172.21.15.71:6816","nonce":33274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6825","nonce":33274},{"type":"v1","addr":"172.21.15.71:6828","nonce":33274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":33274},{"type":"v1","addr":"172.21.15.71:6822","nonce":33274}]},"public_addr":"172.21.15.71:6812/33274","cluster_addr":"172.21.15.71:6816/33274","heartbeat_back_addr":"172.21.15.71:6828/33274","heartbeat_front_addr":"172.21.15.71:6822/33274","state":["exists"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":33275},{"type":"v1","addr":"172.21.15.71:6811","nonce":33275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":33275},{"type":"v1","addr":"172.21.15.71:6818","nonce":33275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6827","nonce":33275},{"type":"v1","addr":"172.21.15.71:6830","nonce":33275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":33275},{"type":"v1","addr":"172.21.15.71:6823","nonce":33275}]},"public_addr":"172.21.15.71:6811/33275","cluster_addr":"172.21.15.71:6818/33275","heartbeat_back_addr":"172.21.15.71:6830/33275","heartbeat_front_addr":"172.21.15.71:6823/33275","state":["exists","up"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":33272},{"type":"v1","addr":"172.21.15.71:6801","nonce":33272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":33272},{"type":"v1","addr":"172.21.15.71:6803","nonce":33272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":33272},{"type":"v1","addr":"172.21.15.71:6807","nonce":33272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":33272},{"type":"v1","addr":"172.21.15.71:6805","nonce":33272}]},"public_addr":"172.21.15.71:6801/33272","cluster_addr":"172.21.15.71:6803/33272","heartbeat_back_addr":"172.21.15.71:6807/33272","heartbeat_front_addr":"172.21.15.71:6805/33272","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.447050+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.374310+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.355034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.394307+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:43:59.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:43:59.762 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:43:59.763 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":24,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:43:58.011412+0000","last_up_change":"2024-09-18T11:43:57.001339+0000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-18T11:43:35.889679+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":23,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":40343},{"type":"v1","addr":"172.21.15.71:6801","nonce":40343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":40343},{"type":"v1","addr":"172.21.15.71:6803","nonce":40343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":40343},{"type":"v1","addr":"172.21.15.71:6807","nonce":40343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":40343},{"type":"v1","addr":"172.21.15.71:6805","nonce":40343}]},"public_addr":"172.21.15.71:6801/40343","cluster_addr":"172.21.15.71:6803/40343","heartbeat_back_addr":"172.21.15.71:6807/40343","heartbeat_front_addr":"172.21.15.71:6805/40343","state":["exists","up"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":33274},{"type":"v1","addr":"172.21.15.71:6812","nonce":33274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":33274},{"type":"v1","addr":"172.21.15.71:6816","nonce":33274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6825","nonce":33274},{"type":"v1","addr":"172.21.15.71:6828","nonce":33274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":33274},{"type":"v1","addr":"172.21.15.71:6822","nonce":33274}]},"public_addr":"172.21.15.71:6812/33274","cluster_addr":"172.21.15.71:6816/33274","heartbeat_back_addr":"172.21.15.71:6828/33274","heartbeat_front_addr":"172.21.15.71:6822/33274","state":["exists"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":33275},{"type":"v1","addr":"172.21.15.71:6811","nonce":33275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":33275},{"type":"v1","addr":"172.21.15.71:6818","nonce":33275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6827","nonce":33275},{"type":"v1","addr":"172.21.15.71:6830","nonce":33275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":33275},{"type":"v1","addr":"172.21.15.71:6823","nonce":33275}]},"public_addr":"172.21.15.71:6811/33275","cluster_addr":"172.21.15.71:6818/33275","heartbeat_back_addr":"172.21.15.71:6830/33275","heartbeat_front_addr":"172.21.15.71:6823/33275","state":["exists","up"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":33272},{"type":"v1","addr":"172.21.15.71:6801","nonce":33272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":33272},{"type":"v1","addr":"172.21.15.71:6803","nonce":33272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":33272},{"type":"v1","addr":"172.21.15.71:6807","nonce":33272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":33272},{"type":"v1","addr":"172.21.15.71:6805","nonce":33272}]},"public_addr":"172.21.15.71:6801/33272","cluster_addr":"172.21.15.71:6803/33272","heartbeat_back_addr":"172.21.15.71:6807/33272","heartbeat_front_addr":"172.21.15.71:6805/33272","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.447050+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.374310+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.355034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.394307+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:43:59.775 INFO:tasks.ec_lost_unfound.ceph_manager:osd.0 is up 2024-09-18T11:43:59.775 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-18T11:43:59.775 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-18T11:43:59.777 INFO:tasks.ceph.osd.3:Started 2024-09-18T11:43:59.778 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T11:43:59.870 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:0 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:1 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:2 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:abort 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:assert 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:config diff 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:config diff get 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:config help [] 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2024-09-18T11:43:59.871 INFO:teuthology.orchestra.run.smithi071.stderr:admin_socket: invalid command 2024-09-18T11:43:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T11:43:59.873 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-18T11:44:00.362 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-18T11:44:00.360+0000 7f4fff31f440 -1 Falling back to public interface 2024-09-18T11:44:02.023 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-09-18T11:44:02.022+0000 7f0506ff6640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-09-18T11:44:02.771 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-18T11:44:02.770+0000 7f4fff31f440 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:44:03.533 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-18T11:44:03.532+0000 7f4fff31f440 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:44:03.820 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-18T11:44:03.819+0000 7f4fff31f440 -1 osd.3 17 log_to_monitors true 2024-09-18T11:44:04.097 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-18T11:44:04.097+0000 7f4ff55ed640 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:44:04.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T11:44:04.985 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:04.985 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-18T11:44:04.985 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-18T11:44:04.985 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:04.994 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.3 to be up 2024-09-18T11:44:04.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:44:05.265 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:05.265 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":26,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:44:05.079869+0000","last_up_change":"2024-09-18T11:44:05.079869+0000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-18T11:43:35.889679+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":23,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":40343},{"type":"v1","addr":"172.21.15.71:6801","nonce":40343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":40343},{"type":"v1","addr":"172.21.15.71:6803","nonce":40343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":40343},{"type":"v1","addr":"172.21.15.71:6807","nonce":40343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":40343},{"type":"v1","addr":"172.21.15.71:6805","nonce":40343}]},"public_addr":"172.21.15.71:6801/40343","cluster_addr":"172.21.15.71:6803/40343","heartbeat_back_addr":"172.21.15.71:6807/40343","heartbeat_front_addr":"172.21.15.71:6805/40343","state":["exists","up"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":33274},{"type":"v1","addr":"172.21.15.71:6812","nonce":33274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":33274},{"type":"v1","addr":"172.21.15.71:6816","nonce":33274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6825","nonce":33274},{"type":"v1","addr":"172.21.15.71:6828","nonce":33274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":33274},{"type":"v1","addr":"172.21.15.71:6822","nonce":33274}]},"public_addr":"172.21.15.71:6812/33274","cluster_addr":"172.21.15.71:6816/33274","heartbeat_back_addr":"172.21.15.71:6828/33274","heartbeat_front_addr":"172.21.15.71:6822/33274","state":["exists"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":33275},{"type":"v1","addr":"172.21.15.71:6811","nonce":33275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":33275},{"type":"v1","addr":"172.21.15.71:6818","nonce":33275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6827","nonce":33275},{"type":"v1","addr":"172.21.15.71:6830","nonce":33275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":33275},{"type":"v1","addr":"172.21.15.71:6823","nonce":33275}]},"public_addr":"172.21.15.71:6811/33275","cluster_addr":"172.21.15.71:6818/33275","heartbeat_back_addr":"172.21.15.71:6830/33275","heartbeat_front_addr":"172.21.15.71:6823/33275","state":["exists","up"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":41313},{"type":"v1","addr":"172.21.15.71:6810","nonce":41313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":41313},{"type":"v1","addr":"172.21.15.71:6813","nonce":41313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6817","nonce":41313},{"type":"v1","addr":"172.21.15.71:6819","nonce":41313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":41313},{"type":"v1","addr":"172.21.15.71:6816","nonce":41313}]},"public_addr":"172.21.15.71:6810/41313","cluster_addr":"172.21.15.71:6813/41313","heartbeat_back_addr":"172.21.15.71:6819/41313","heartbeat_front_addr":"172.21.15.71:6816/41313","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.447050+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.374310+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.355034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.394307+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:44:05.277 INFO:tasks.ec_lost_unfound.ceph_manager:osd.3 is up 2024-09-18T11:44:05.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:44:05.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T11:44:05.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T11:44:05.409 INFO:teuthology.orchestra.run.smithi071.stdout:111669149698 2024-09-18T11:44:05.409 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T11:44:05.411 INFO:teuthology.orchestra.run.smithi071.stdout:98784247811 2024-09-18T11:44:05.411 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:44:05.412 INFO:teuthology.orchestra.run.smithi071.stdout:38654705673 2024-09-18T11:44:05.412 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:44:05.723 INFO:teuthology.orchestra.run.smithi071.stdout:38654705672 2024-09-18T11:44:05.727 INFO:teuthology.orchestra.run.smithi071.stdout:98784247810 2024-09-18T11:44:05.729 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2024-09-18T11:44:05.737 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705673 got 38654705672 for osd.2 2024-09-18T11:44:05.744 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247811 got 98784247810 for osd.0 2024-09-18T11:44:05.746 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149698 got 38654705669 for osd.3 2024-09-18T11:44:06.737 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:44:06.744 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:44:06.746 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T11:44:07.058 INFO:teuthology.orchestra.run.smithi071.stdout:98784247811 2024-09-18T11:44:07.058 INFO:teuthology.orchestra.run.smithi071.stdout:38654705673 2024-09-18T11:44:07.072 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705673 got 38654705673 for osd.2 2024-09-18T11:44:07.072 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:07.072 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247811 got 98784247811 for osd.0 2024-09-18T11:44:07.072 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:07.089 INFO:teuthology.orchestra.run.smithi071.stdout:111669149698 2024-09-18T11:44:07.101 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149698 got 111669149698 for osd.3 2024-09-18T11:44:07.101 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:07.102 INFO:tasks.ec_lost_unfound.ceph_manager:waiting till active 2024-09-18T11:44:07.102 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:07.398 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:07.399 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:07.410 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-09-18T11:44:06.084886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":11,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1191396,"kb_used_data":1008,"kb_used_omap":12,"kb_used_meta":1190323,"kb_avail":376295964,"statfs":{"total":386547056640,"available":385327067136,"internally_reserved":0,"allocated":1032192,"data_stored":522953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12494,"internal_metadata":1218891570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":11,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.266024"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:05.092157+0000","last_change":"2024-09-18T11:44:05.088251+0000","last_active":"2024-09-18T11:43:45.967580+0000","last_peered":"2024-09-18T11:43:39.961502+0000","last_clean":"2024-09-18T11:43:39.961502+0000","last_became_active":"2024-09-18T11:43:38.892493+0000","last_became_peered":"2024-09-18T11:43:38.892493+0000","last_unstale":"2024-09-18T11:44:05.092157+0000","last_undegraded":"2024-09-18T11:44:05.092157+0000","last_fullsized":"2024-09-18T11:44:05.092157+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:24:10.779699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"19'4","reported_seq":35,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.004133+0000","last_change":"2024-09-18T11:44:05.101920+0000","last_active":"2024-09-18T11:43:49.984168+0000","last_peered":"2024-09-18T11:43:48.773708+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:43:47.987849+0000","last_became_peered":"2024-09-18T11:43:47.987849+0000","last_unstale":"2024-09-18T11:43:56.004133+0000","last_undegraded":"2024-09-18T11:43:56.004133+0000","last_fullsized":"2024-09-18T11:43:56.004133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:36.983411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"19'3","reported_seq":53,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:05.097282+0000","last_change":"2024-09-18T11:44:05.097282+0000","last_active":"2024-09-18T11:44:05.097023+0000","last_peered":"2024-09-18T11:44:04.084021+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:43:58.034001+0000","last_became_peered":"2024-09-18T11:43:58.034001+0000","last_unstale":"2024-09-18T11:44:05.097282+0000","last_undegraded":"2024-09-18T11:44:05.097282+0000","last_fullsized":"2024-09-18T11:44:05.097282+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:20:02.888206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4","reported_seq":35,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.003843+0000","last_change":"2024-09-18T11:44:05.101933+0000","last_active":"2024-09-18T11:43:49.982593+0000","last_peered":"2024-09-18T11:43:48.591809+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:43:47.988446+0000","last_became_peered":"2024-09-18T11:43:47.988446+0000","last_unstale":"2024-09-18T11:43:56.003843+0000","last_undegraded":"2024-09-18T11:43:56.003843+0000","last_fullsized":"2024-09-18T11:43:56.003843+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:38:18.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3","reported_seq":24,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.003841+0000","last_change":"2024-09-18T11:44:05.099406+0000","last_active":"2024-09-18T11:43:49.983183+0000","last_peered":"2024-09-18T11:43:48.362307+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:43:47.988364+0000","last_became_peered":"2024-09-18T11:43:47.988364+0000","last_unstale":"2024-09-18T11:43:56.003841+0000","last_undegraded":"2024-09-18T11:43:56.003841+0000","last_fullsized":"2024-09-18T11:43:56.003841+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:30.650564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:05.092382+0000","last_change":"2024-09-18T11:44:05.089462+0000","last_active":"2024-09-18T11:43:45.967036+0000","last_peered":"2024-09-18T11:43:39.961379+0000","last_clean":"2024-09-18T11:43:39.961379+0000","last_became_active":"2024-09-18T11:43:38.892767+0000","last_became_peered":"2024-09-18T11:43:38.892767+0000","last_unstale":"2024-09-18T11:44:05.092382+0000","last_undegraded":"2024-09-18T11:44:05.092382+0000","last_fullsized":"2024-09-18T11:44:05.092382+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:12:42.292117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"19'6","reported_seq":37,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.003955+0000","last_change":"2024-09-18T11:44:05.101839+0000","last_active":"2024-09-18T11:43:49.982898+0000","last_peered":"2024-09-18T11:43:49.141049+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:43:47.987727+0000","last_became_peered":"2024-09-18T11:43:47.987727+0000","last_unstale":"2024-09-18T11:43:56.003955+0000","last_undegraded":"2024-09-18T11:43:56.003955+0000","last_fullsized":"2024-09-18T11:43:56.003955+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:02:30.717874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'5","reported_seq":36,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.004062+0000","last_change":"2024-09-18T11:44:05.101922+0000","last_active":"2024-09-18T11:43:49.982935+0000","last_peered":"2024-09-18T11:43:49.050393+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:43:47.988693+0000","last_became_peered":"2024-09-18T11:43:47.988693+0000","last_unstale":"2024-09-18T11:43:56.004062+0000","last_undegraded":"2024-09-18T11:43:56.004062+0000","last_fullsized":"2024-09-18T11:43:56.004062+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:37:37.674520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'9","reported_seq":29,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.004175+0000","last_change":"2024-09-18T11:44:05.099500+0000","last_active":"2024-09-18T11:43:49.983701+0000","last_peered":"2024-09-18T11:43:48.911041+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:43:47.984651+0000","last_became_peered":"2024-09-18T11:43:47.984651+0000","last_unstale":"2024-09-18T11:43:56.004175+0000","last_undegraded":"2024-09-18T11:43:56.004175+0000","last_fullsized":"2024-09-18T11:43:56.004175+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:59:35.894764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"19'2","reported_seq":51,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:05.094642+0000","last_change":"2024-09-18T11:44:05.094642+0000","last_active":"2024-09-18T11:44:05.094370+0000","last_peered":"2024-09-18T11:44:04.084411+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:43:58.035347+0000","last_became_peered":"2024-09-18T11:43:58.035347+0000","last_unstale":"2024-09-18T11:44:05.094642+0000","last_undegraded":"2024-09-18T11:44:05.094642+0000","last_fullsized":"2024-09-18T11:44:05.094642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:32:38.210808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'5","reported_seq":25,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.004021+0000","last_change":"2024-09-18T11:44:05.099289+0000","last_active":"2024-09-18T11:43:49.983839+0000","last_peered":"2024-09-18T11:43:49.004673+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:43:47.987790+0000","last_became_peered":"2024-09-18T11:43:47.987790+0000","last_unstale":"2024-09-18T11:43:56.004021+0000","last_undegraded":"2024-09-18T11:43:56.004021+0000","last_fullsized":"2024-09-18T11:43:56.004021+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:09:02.838475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"19'1","reported_seq":10,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:04.084181+0000","last_change":"2024-09-18T11:44:05.099907+0000","last_active":"2024-09-18T11:44:05.099907+0000","last_peered":"2024-09-18T11:44:04.084181+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:43:58.035351+0000","last_became_peered":"2024-09-18T11:43:58.035351+0000","last_unstale":"2024-09-18T11:44:04.084181+0000","last_undegraded":"2024-09-18T11:43:58.027237+0000","last_fullsized":"2024-09-18T11:43:58.027036+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:23:31.350777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"19'1","reported_seq":50,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:05.095461+0000","last_change":"2024-09-18T11:44:05.095461+0000","last_active":"2024-09-18T11:44:05.094965+0000","last_peered":"2024-09-18T11:44:04.083806+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:43:58.033860+0000","last_became_peered":"2024-09-18T11:43:58.033860+0000","last_unstale":"2024-09-18T11:44:05.095461+0000","last_undegraded":"2024-09-18T11:44:05.095461+0000","last_fullsized":"2024-09-18T11:44:05.095461+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:07:49.011604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"19'4","reported_seq":22,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:04.084401+0000","last_change":"2024-09-18T11:44:05.099657+0000","last_active":"2024-09-18T11:44:05.099657+0000","last_peered":"2024-09-18T11:44:04.084401+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:43:58.030881+0000","last_became_peered":"2024-09-18T11:43:58.030881+0000","last_unstale":"2024-09-18T11:44:04.084401+0000","last_undegraded":"2024-09-18T11:43:58.026144+0000","last_fullsized":"2024-09-18T11:43:58.025925+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:58:09.393977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"19'3","reported_seq":53,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:05.095030+0000","last_change":"2024-09-18T11:44:05.095030+0000","last_active":"2024-09-18T11:44:05.094732+0000","last_peered":"2024-09-18T11:44:04.083892+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:43:58.030896+0000","last_became_peered":"2024-09-18T11:43:58.030896+0000","last_unstale":"2024-09-18T11:44:05.095030+0000","last_undegraded":"2024-09-18T11:44:05.095030+0000","last_fullsized":"2024-09-18T11:44:05.095030+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:43:46.116054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"19'5","reported_seq":36,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.003980+0000","last_change":"2024-09-18T11:44:05.102047+0000","last_active":"2024-09-18T11:43:49.982635+0000","last_peered":"2024-09-18T11:43:48.316977+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:43:47.988780+0000","last_became_peered":"2024-09-18T11:43:47.988780+0000","last_unstale":"2024-09-18T11:43:56.003980+0000","last_undegraded":"2024-09-18T11:43:56.003980+0000","last_fullsized":"2024-09-18T11:43:56.003980+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:16:19.829390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":11,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298116,"kb_used_data":248,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073724,"statfs":{"total":96636764160,"available":96331493376,"internally_reserved":0,"allocated":253952,"data_stored":118979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":335872,"data_stored":196936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23699999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":180224,"data_stored":83830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298064,"kb_used_data":256,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073776,"statfs":{"total":96636764160,"available":96331546624,"internally_reserved":0,"allocated":262144,"data_stored":123208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:07.411 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:44:07.724 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:07.724 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":45,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726659845,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":16}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":1219977216,"bytes_avail":385327079424,"bytes_total":386547056640,"inactive_pgs_ratio":1},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T11:43:21.914010+0000","services":{}},"progress_events":{}} 2024-09-18T11:44:07.737 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': 'c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab', 'health': {'status': 'HEALTH_WARN', 'checks': {'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 45, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726659845, 'num_in_osds': 4, 'osd_in_since': 1726659803, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'peering', 'count': 16}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 189812, 'bytes_used': 1219977216, 'bytes_avail': 385327079424, 'bytes_total': 386547056640, 'inactive_pgs_ratio': 1}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-18T11:43:21.914010+0000', 'services': {}}, 'progress_events': {}} 2024-09-18T11:44:09.872 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-09-18T11:44:09.871+0000 7f0506ff6640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 5 pgs recovery_unfound (PG_DAMAGED) 2024-09-18T11:44:10.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:11.008 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:11.008 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:11.019 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-18T11:44:09.820904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":11,"num_objects_missing":11,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":11,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1191420,"kb_used_data":1024,"kb_used_omap":12,"kb_used_meta":1190323,"kb_avail":376295940,"statfs":{"total":386547056640,"available":385327042560,"internally_reserved":0,"allocated":1048576,"data_stored":530721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12494,"internal_metadata":1218891570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-7,"num_objects_missing":-7,"num_objects_degraded":-49,"num_objects_misplaced":0,"num_objects_unfound":-16,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001147"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":29,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-18T11:44:06.095337+0000","last_change":"2024-09-18T11:44:06.095253+0000","last_active":"2024-09-18T11:44:06.095337+0000","last_peered":"2024-09-18T11:44:06.095337+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095253+0000","last_became_peered":"2024-09-18T11:44:06.095253+0000","last_unstale":"2024-09-18T11:44:06.095337+0000","last_undegraded":"2024-09-18T11:44:06.095337+0000","last_fullsized":"2024-09-18T11:44:06.087120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:29.394125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"19'4","reported_seq":35,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.004133+0000","last_change":"2024-09-18T11:44:05.101920+0000","last_active":"2024-09-18T11:43:49.984168+0000","last_peered":"2024-09-18T11:43:48.773708+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:43:47.987849+0000","last_became_peered":"2024-09-18T11:43:47.987849+0000","last_unstale":"2024-09-18T11:43:56.004133+0000","last_undegraded":"2024-09-18T11:43:56.004133+0000","last_fullsized":"2024-09-18T11:43:56.004133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:36.983411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"19'3","reported_seq":53,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:05.097282+0000","last_change":"2024-09-18T11:44:05.097282+0000","last_active":"2024-09-18T11:44:05.097023+0000","last_peered":"2024-09-18T11:44:04.084021+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:43:58.034001+0000","last_became_peered":"2024-09-18T11:43:58.034001+0000","last_unstale":"2024-09-18T11:44:05.097282+0000","last_undegraded":"2024-09-18T11:44:05.097282+0000","last_fullsized":"2024-09-18T11:44:05.097282+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:20:02.888206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4","reported_seq":35,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.003843+0000","last_change":"2024-09-18T11:44:05.101933+0000","last_active":"2024-09-18T11:43:49.982593+0000","last_peered":"2024-09-18T11:43:48.591809+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:43:47.988446+0000","last_became_peered":"2024-09-18T11:43:47.988446+0000","last_unstale":"2024-09-18T11:43:56.003843+0000","last_undegraded":"2024-09-18T11:43:56.003843+0000","last_fullsized":"2024-09-18T11:43:56.003843+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:38:18.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3","reported_seq":33,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.102381+0000","last_change":"2024-09-18T11:44:06.102623+0000","last_active":"2024-09-18T11:44:06.102381+0000","last_peered":"2024-09-18T11:44:06.102381+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:06.095158+0000","last_became_peered":"2024-09-18T11:44:06.095158+0000","last_unstale":"2024-09-18T11:44:06.102381+0000","last_undegraded":"2024-09-18T11:44:06.087666+0000","last_fullsized":"2024-09-18T11:44:06.086930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:30.650564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":29,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-18T11:44:06.095850+0000","last_change":"2024-09-18T11:44:06.095774+0000","last_active":"2024-09-18T11:44:06.095850+0000","last_peered":"2024-09-18T11:44:06.095850+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:06.095850+0000","last_undegraded":"2024-09-18T11:44:06.095850+0000","last_fullsized":"2024-09-18T11:44:06.088316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:21:46.952968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"19'6","reported_seq":37,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.003955+0000","last_change":"2024-09-18T11:44:05.101839+0000","last_active":"2024-09-18T11:43:49.982898+0000","last_peered":"2024-09-18T11:43:49.141049+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:43:47.987727+0000","last_became_peered":"2024-09-18T11:43:47.987727+0000","last_unstale":"2024-09-18T11:43:56.003955+0000","last_undegraded":"2024-09-18T11:43:56.003955+0000","last_fullsized":"2024-09-18T11:43:56.003955+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:02:30.717874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'5","reported_seq":36,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.004062+0000","last_change":"2024-09-18T11:44:05.101922+0000","last_active":"2024-09-18T11:43:49.982935+0000","last_peered":"2024-09-18T11:43:49.050393+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:43:47.988693+0000","last_became_peered":"2024-09-18T11:43:47.988693+0000","last_unstale":"2024-09-18T11:43:56.004062+0000","last_undegraded":"2024-09-18T11:43:56.004062+0000","last_fullsized":"2024-09-18T11:43:56.004062+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:37:37.674520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'9","reported_seq":38,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.107458+0000","last_change":"2024-09-18T11:44:06.107730+0000","last_active":"2024-09-18T11:44:06.107458+0000","last_peered":"2024-09-18T11:44:06.107458+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:06.095233+0000","last_became_peered":"2024-09-18T11:44:06.095233+0000","last_unstale":"2024-09-18T11:44:06.107458+0000","last_undegraded":"2024-09-18T11:44:06.087588+0000","last_fullsized":"2024-09-18T11:44:06.087287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:59:35.894764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"19'2","reported_seq":51,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:05.094642+0000","last_change":"2024-09-18T11:44:05.094642+0000","last_active":"2024-09-18T11:44:05.094370+0000","last_peered":"2024-09-18T11:44:04.084411+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:43:58.035347+0000","last_became_peered":"2024-09-18T11:43:58.035347+0000","last_unstale":"2024-09-18T11:44:05.094642+0000","last_undegraded":"2024-09-18T11:44:05.094642+0000","last_fullsized":"2024-09-18T11:44:05.094642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:32:38.210808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'5","reported_seq":34,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.110731+0000","last_change":"2024-09-18T11:44:06.110973+0000","last_active":"2024-09-18T11:44:06.110731+0000","last_peered":"2024-09-18T11:44:06.110731+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:06.099667+0000","last_became_peered":"2024-09-18T11:44:06.099667+0000","last_unstale":"2024-09-18T11:44:06.110731+0000","last_undegraded":"2024-09-18T11:44:06.087962+0000","last_fullsized":"2024-09-18T11:44:06.087548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:09:02.838475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"19'1","reported_seq":19,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.109306+0000","last_change":"2024-09-18T11:44:06.109597+0000","last_active":"2024-09-18T11:44:06.109306+0000","last_peered":"2024-09-18T11:44:06.109306+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:06.109306+0000","last_undegraded":"2024-09-18T11:44:06.087360+0000","last_fullsized":"2024-09-18T11:44:06.087125+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:23:31.350777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"19'1","reported_seq":50,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:05.095461+0000","last_change":"2024-09-18T11:44:05.095461+0000","last_active":"2024-09-18T11:44:05.094965+0000","last_peered":"2024-09-18T11:44:04.083806+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:43:58.033860+0000","last_became_peered":"2024-09-18T11:43:58.033860+0000","last_unstale":"2024-09-18T11:44:05.095461+0000","last_undegraded":"2024-09-18T11:44:05.095461+0000","last_fullsized":"2024-09-18T11:44:05.095461+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:07:49.011604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"19'4","reported_seq":31,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.105563+0000","last_change":"2024-09-18T11:44:06.105818+0000","last_active":"2024-09-18T11:44:06.105563+0000","last_peered":"2024-09-18T11:44:06.105563+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:06.095532+0000","last_became_peered":"2024-09-18T11:44:06.095532+0000","last_unstale":"2024-09-18T11:44:06.105563+0000","last_undegraded":"2024-09-18T11:44:06.087192+0000","last_fullsized":"2024-09-18T11:44:06.086893+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:58:09.393977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"19'3","reported_seq":53,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:44:05.095030+0000","last_change":"2024-09-18T11:44:05.095030+0000","last_active":"2024-09-18T11:44:05.094732+0000","last_peered":"2024-09-18T11:44:04.083892+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:43:58.030896+0000","last_became_peered":"2024-09-18T11:43:58.030896+0000","last_unstale":"2024-09-18T11:44:05.095030+0000","last_undegraded":"2024-09-18T11:44:05.095030+0000","last_fullsized":"2024-09-18T11:44:05.095030+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:43:46.116054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"19'5","reported_seq":36,"reported_epoch":26,"state":"peering","last_fresh":"2024-09-18T11:43:56.003980+0000","last_change":"2024-09-18T11:44:05.102047+0000","last_active":"2024-09-18T11:43:49.982635+0000","last_peered":"2024-09-18T11:43:48.316977+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:43:47.988780+0000","last_became_peered":"2024-09-18T11:43:47.988780+0000","last_unstale":"2024-09-18T11:43:56.003980+0000","last_undegraded":"2024-09-18T11:43:56.003980+0000","last_fullsized":"2024-09-18T11:43:56.003980+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:16:19.829390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":11,"num_objects_missing":11,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":11,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298140,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073700,"statfs":{"total":96636764160,"available":96331468800,"internally_reserved":0,"allocated":270336,"data_stored":126747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]}]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":335872,"data_stored":196936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23699999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":180224,"data_stored":83830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298064,"kb_used_data":256,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073776,"statfs":{"total":96636764160,"available":96331546624,"internally_reserved":0,"allocated":262144,"data_stored":123208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:11.020 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:44:11.332 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:11.332 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"11/28 objects unfound (39.286%)","count":11},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 5 pgs recovery_unfound","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34/112 objects degraded (30.357%), 5 pgs degraded","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":49,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726659845,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":9},{"state_name":"active+recovery_unfound+undersized+degraded","count":5},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":1220014080,"bytes_avail":385327042560,"bytes_total":386547056640,"inactive_pgs_ratio":0.5625,"degraded_objects":34,"degraded_total":112,"degraded_ratio":0.30357142857142855,"unfound_objects":11,"unfound_total":28,"unfound_ratio":0.39285714285714285},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T11:43:21.914010+0000","services":{}},"progress_events":{}} 2024-09-18T11:44:11.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': 'c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '11/28 objects unfound (39.286%)', 'count': 11}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 5 pgs recovery_unfound', 'count': 5}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 34/112 objects degraded (30.357%), 5 pgs degraded', 'count': 5}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 49, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726659845, 'num_in_osds': 4, 'osd_in_since': 1726659803, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'peering', 'count': 9}, {'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 5}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 189812, 'bytes_used': 1220014080, 'bytes_avail': 385327042560, 'bytes_total': 386547056640, 'inactive_pgs_ratio': 0.5625, 'degraded_objects': 34, 'degraded_total': 112, 'degraded_ratio': 0.30357142857142855, 'unfound_objects': 11, 'unfound_total': 28, 'unfound_ratio': 0.39285714285714285}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-18T11:43:21.914010+0000', 'services': {}}, 'progress_events': {}} 2024-09-18T11:44:14.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:14.616 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:14.616 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-18T11:44:13.821686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1191444,"kb_used_data":1040,"kb_used_omap":12,"kb_used_meta":1190323,"kb_avail":376295916,"statfs":{"total":386547056640,"available":385327017984,"internally_reserved":0,"allocated":1064960,"data_stored":537799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12494,"internal_metadata":1218891570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.738650"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":29,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-18T11:44:06.095337+0000","last_change":"2024-09-18T11:44:06.095253+0000","last_active":"2024-09-18T11:44:06.095337+0000","last_peered":"2024-09-18T11:44:06.095337+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095253+0000","last_became_peered":"2024-09-18T11:44:06.095253+0000","last_unstale":"2024-09-18T11:44:06.095337+0000","last_undegraded":"2024-09-18T11:44:06.095337+0000","last_fullsized":"2024-09-18T11:44:06.087120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:29.394125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"19'4","reported_seq":44,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.103330+0000","last_change":"2024-09-18T11:44:06.103665+0000","last_active":"2024-09-18T11:44:06.103330+0000","last_peered":"2024-09-18T11:44:06.103330+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:06.096040+0000","last_became_peered":"2024-09-18T11:44:06.096040+0000","last_unstale":"2024-09-18T11:44:06.103330+0000","last_undegraded":"2024-09-18T11:44:06.086523+0000","last_fullsized":"2024-09-18T11:44:06.086250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:36.983411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"19'3","reported_seq":62,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.110051+0000","last_change":"2024-09-18T11:44:06.110338+0000","last_active":"2024-09-18T11:44:06.110051+0000","last_peered":"2024-09-18T11:44:06.110051+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:06.100615+0000","last_became_peered":"2024-09-18T11:44:06.100615+0000","last_unstale":"2024-09-18T11:44:06.110051+0000","last_undegraded":"2024-09-18T11:44:06.093664+0000","last_fullsized":"2024-09-18T11:44:06.093409+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:20:02.888206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4","reported_seq":44,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.108204+0000","last_change":"2024-09-18T11:44:06.108449+0000","last_active":"2024-09-18T11:44:06.108204+0000","last_peered":"2024-09-18T11:44:06.108204+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:06.099305+0000","last_became_peered":"2024-09-18T11:44:06.099305+0000","last_unstale":"2024-09-18T11:44:06.108204+0000","last_undegraded":"2024-09-18T11:44:06.085610+0000","last_fullsized":"2024-09-18T11:44:06.085286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:38:18.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3","reported_seq":33,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.102381+0000","last_change":"2024-09-18T11:44:06.102623+0000","last_active":"2024-09-18T11:44:06.102381+0000","last_peered":"2024-09-18T11:44:06.102381+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:06.095158+0000","last_became_peered":"2024-09-18T11:44:06.095158+0000","last_unstale":"2024-09-18T11:44:06.102381+0000","last_undegraded":"2024-09-18T11:44:06.087666+0000","last_fullsized":"2024-09-18T11:44:06.086930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:30.650564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":29,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-18T11:44:06.095850+0000","last_change":"2024-09-18T11:44:06.095774+0000","last_active":"2024-09-18T11:44:06.095850+0000","last_peered":"2024-09-18T11:44:06.095850+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:06.095850+0000","last_undegraded":"2024-09-18T11:44:06.095850+0000","last_fullsized":"2024-09-18T11:44:06.088316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:21:46.952968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"19'6","reported_seq":46,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.104731+0000","last_change":"2024-09-18T11:44:06.105109+0000","last_active":"2024-09-18T11:44:06.104731+0000","last_peered":"2024-09-18T11:44:06.104731+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:06.096306+0000","last_became_peered":"2024-09-18T11:44:06.096306+0000","last_unstale":"2024-09-18T11:44:06.104731+0000","last_undegraded":"2024-09-18T11:44:06.085856+0000","last_fullsized":"2024-09-18T11:44:06.085557+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:02:30.717874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'5","reported_seq":45,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.101030+0000","last_change":"2024-09-18T11:44:06.101426+0000","last_active":"2024-09-18T11:44:06.101030+0000","last_peered":"2024-09-18T11:44:06.101030+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:06.095136+0000","last_became_peered":"2024-09-18T11:44:06.095136+0000","last_unstale":"2024-09-18T11:44:06.101030+0000","last_undegraded":"2024-09-18T11:44:06.086038+0000","last_fullsized":"2024-09-18T11:44:06.085713+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:37:37.674520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'9","reported_seq":38,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.107458+0000","last_change":"2024-09-18T11:44:06.107730+0000","last_active":"2024-09-18T11:44:06.107458+0000","last_peered":"2024-09-18T11:44:06.107458+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:06.095233+0000","last_became_peered":"2024-09-18T11:44:06.095233+0000","last_unstale":"2024-09-18T11:44:06.107458+0000","last_undegraded":"2024-09-18T11:44:06.087588+0000","last_fullsized":"2024-09-18T11:44:06.087287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:59:35.894764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"19'2","reported_seq":60,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.108637+0000","last_change":"2024-09-18T11:44:06.108847+0000","last_active":"2024-09-18T11:44:06.108637+0000","last_peered":"2024-09-18T11:44:06.108637+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:06.100048+0000","last_became_peered":"2024-09-18T11:44:06.100048+0000","last_unstale":"2024-09-18T11:44:06.108637+0000","last_undegraded":"2024-09-18T11:44:06.093124+0000","last_fullsized":"2024-09-18T11:44:06.092822+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:32:38.210808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'5","reported_seq":34,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.110731+0000","last_change":"2024-09-18T11:44:06.110973+0000","last_active":"2024-09-18T11:44:06.110731+0000","last_peered":"2024-09-18T11:44:06.110731+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:06.099667+0000","last_became_peered":"2024-09-18T11:44:06.099667+0000","last_unstale":"2024-09-18T11:44:06.110731+0000","last_undegraded":"2024-09-18T11:44:06.087962+0000","last_fullsized":"2024-09-18T11:44:06.087548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:09:02.838475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"19'1","reported_seq":19,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.109306+0000","last_change":"2024-09-18T11:44:06.109597+0000","last_active":"2024-09-18T11:44:06.109306+0000","last_peered":"2024-09-18T11:44:06.109306+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:06.109306+0000","last_undegraded":"2024-09-18T11:44:06.087360+0000","last_fullsized":"2024-09-18T11:44:06.087125+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:23:31.350777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"19'1","reported_seq":59,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.104069+0000","last_change":"2024-09-18T11:44:06.104303+0000","last_active":"2024-09-18T11:44:06.104069+0000","last_peered":"2024-09-18T11:44:06.104069+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:06.095142+0000","last_became_peered":"2024-09-18T11:44:06.095142+0000","last_unstale":"2024-09-18T11:44:06.104069+0000","last_undegraded":"2024-09-18T11:44:06.092217+0000","last_fullsized":"2024-09-18T11:44:06.091922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:07:49.011604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"19'4","reported_seq":31,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.105563+0000","last_change":"2024-09-18T11:44:06.105818+0000","last_active":"2024-09-18T11:44:06.105563+0000","last_peered":"2024-09-18T11:44:06.105563+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:06.095532+0000","last_became_peered":"2024-09-18T11:44:06.095532+0000","last_unstale":"2024-09-18T11:44:06.105563+0000","last_undegraded":"2024-09-18T11:44:06.087192+0000","last_fullsized":"2024-09-18T11:44:06.086893+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:58:09.393977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"19'3","reported_seq":62,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.106291+0000","last_change":"2024-09-18T11:44:06.106603+0000","last_active":"2024-09-18T11:44:06.106291+0000","last_peered":"2024-09-18T11:44:06.106291+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:06.100417+0000","last_became_peered":"2024-09-18T11:44:06.100417+0000","last_unstale":"2024-09-18T11:44:06.106291+0000","last_undegraded":"2024-09-18T11:44:06.093073+0000","last_fullsized":"2024-09-18T11:44:06.092766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:43:46.116054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"19'5","reported_seq":45,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:06.106874+0000","last_change":"2024-09-18T11:44:06.107245+0000","last_active":"2024-09-18T11:44:06.106874+0000","last_peered":"2024-09-18T11:44:06.106874+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:06.096352+0000","last_became_peered":"2024-09-18T11:44:06.096352+0000","last_unstale":"2024-09-18T11:44:06.106874+0000","last_undegraded":"2024-09-18T11:44:06.086622+0000","last_fullsized":"2024-09-18T11:44:06.086322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:16:19.829390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298140,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073700,"statfs":{"total":96636764160,"available":96331468800,"internally_reserved":0,"allocated":270336,"data_stored":126747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]}]},{"osd":2,"up_from":9,"seq":38654705675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297700,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074140,"statfs":{"total":96636764160,"available":96331919360,"internally_reserved":0,"allocated":344064,"data_stored":200475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":180224,"data_stored":83830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298076,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073764,"statfs":{"total":96636764160,"available":96331534336,"internally_reserved":0,"allocated":270336,"data_stored":126747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:14.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:44:14.949 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:14.949 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":53,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726659845,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":1220038656,"bytes_avail":385327017984,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T11:43:21.914010+0000","services":{}},"progress_events":{}} 2024-09-18T11:44:14.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': 'c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 53, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726659845, 'num_in_osds': 4, 'osd_in_since': 1726659803, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 189812, 'bytes_used': 1220038656, 'bytes_avail': 385327017984, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-18T11:43:21.914010+0000', 'services': {}}, 'progress_events': {}} 2024-09-18T11:44:14.961 INFO:tasks.ec_lost_unfound.ceph_manager:active! 2024-09-18T11:44:14.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:44:14.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T11:44:14.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T11:44:15.091 INFO:teuthology.orchestra.run.smithi071.stdout:98784247814 2024-09-18T11:44:15.091 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:44:15.108 INFO:teuthology.orchestra.run.smithi071.stdout:38654705676 2024-09-18T11:44:15.108 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:44:15.109 INFO:teuthology.orchestra.run.smithi071.stdout:111669149701 2024-09-18T11:44:15.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T11:44:15.398 INFO:teuthology.orchestra.run.smithi071.stdout:98784247813 2024-09-18T11:44:15.415 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247813 for osd.0 2024-09-18T11:44:15.429 INFO:teuthology.orchestra.run.smithi071.stdout:38654705675 2024-09-18T11:44:15.442 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705676 got 38654705675 for osd.2 2024-09-18T11:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout:111669149699 2024-09-18T11:44:15.461 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149701 got 111669149699 for osd.3 2024-09-18T11:44:16.416 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:44:16.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:44:16.462 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T11:44:16.738 INFO:teuthology.orchestra.run.smithi071.stdout:111669149701 2024-09-18T11:44:16.749 INFO:teuthology.orchestra.run.smithi071.stdout:98784247814 2024-09-18T11:44:16.754 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149701 got 111669149701 for osd.3 2024-09-18T11:44:16.755 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:16.763 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247814 for osd.0 2024-09-18T11:44:16.763 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:16.777 INFO:teuthology.orchestra.run.smithi071.stdout:38654705676 2024-09-18T11:44:16.790 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705676 got 38654705676 for osd.2 2024-09-18T11:44:16.790 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:16.790 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:44:16.896 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-09-18T11:44:16.894+0000 7f0506ff6640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-09-18T11:44:17.146 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:17.146 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":55,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726659845,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":1220038656,"bytes_avail":385327017984,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T11:43:21.914010+0000","services":{}},"progress_events":{}} 2024-09-18T11:44:17.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': 'c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 55, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726659845, 'num_in_osds': 4, 'osd_in_since': 1726659803, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 189812, 'bytes_used': 1220038656, 'bytes_avail': 385327017984, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-18T11:43:21.914010+0000', 'services': {}}, 'progress_events': {}} 2024-09-18T11:44:17.158 INFO:tasks.ec_lost_unfound:there are 27 unfound objects 2024-09-18T11:44:17.158 DEBUG:teuthology.orchestra.run.smithi071:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unique_pool_0 -t 20 bench 240 write' 2024-09-18T11:44:17.204 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stderr:INFO: op_size has been rounded to 8192 2024-09-18T11:44:17.205 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:hints = 1 2024-09-18T11:44:17.205 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Maintaining 20 concurrent writes of 8192 bytes to objects of size 8192 for up to 240 seconds or 0 objects 2024-09-18T11:44:17.205 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Object prefix: benchmark_data_smithi071_43251 2024-09-18T11:44:17.205 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:44:17.205 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 0 0 0 0 0 0 - 0 2024-09-18T11:44:18.205 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 1 20 1790 1770 13.8273 13.8281 0.010362 0.0112266 2024-09-18T11:44:19.205 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 2 20 3519 3499 13.667 13.5078 0.0154371 0.0113808 2024-09-18T11:44:20.205 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 3 20 5187 5167 13.4548 13.0312 0.013279 0.0115894 2024-09-18T11:44:21.205 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 4 19 6793 6774 13.2295 12.5547 0.0140638 0.0117819 2024-09-18T11:44:22.205 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 5 20 8363 8343 13.035 12.2578 0.00894371 0.0119688 2024-09-18T11:44:23.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 6 20 9870 9850 12.8235 11.7734 0.00968854 0.0121676 2024-09-18T11:44:24.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 7 20 11353 11333 12.6465 11.5859 0.0108267 0.0123397 2024-09-18T11:44:25.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 8 20 12799 12779 12.4777 11.2969 0.0135118 0.0125078 2024-09-18T11:44:26.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 9 20 14182 14162 12.2917 10.8047 0.0107069 0.0126946 2024-09-18T11:44:27.161 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:27.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 10 20 15545 15525 12.1273 10.6484 0.0197491 0.0128689 2024-09-18T11:44:27.659 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:27.660 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:27.681 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-18T11:44:25.824870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":69813620,"num_objects":8527,"num_object_clones":0,"num_object_copies":34108,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":8581,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":8527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":68314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8554,"ondisk_log_size":8554,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1305688,"kb_used_data":90360,"kb_used_omap":12,"kb_used_meta":1215283,"kb_avail":376181672,"statfs":{"total":386547056640,"available":385210032128,"internally_reserved":0,"allocated":92528640,"data_stored":92001479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12494,"internal_metadata":1244450610},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,15,8,8,15],"upper_bound":32},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":69623808,"num_objects":8499,"num_object_clones":0,"num_object_copies":33996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8499,"num_write_kb":67992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003184"},"pg_stats":[{"pgid":"1.f","version":"27'706","reported_seq":735,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:24.091237+0000","last_change":"2024-09-18T11:44:17.230219+0000","last_active":"2024-09-18T11:44:24.091237+0000","last_peered":"2024-09-18T11:44:24.091237+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095253+0000","last_became_peered":"2024-09-18T11:44:06.095253+0000","last_unstale":"2024-09-18T11:44:24.091237+0000","last_undegraded":"2024-09-18T11:44:17.229747+0000","last_fullsized":"2024-09-18T11:44:06.087120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:29.394125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5783552,"num_objects":706,"num_object_clones":0,"num_object_copies":2824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":706,"num_write_kb":5648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'376","reported_seq":416,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:20.981700+0000","last_change":"2024-09-18T11:44:06.103665+0000","last_active":"2024-09-18T11:44:20.981700+0000","last_peered":"2024-09-18T11:44:20.981700+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:06.096040+0000","last_became_peered":"2024-09-18T11:44:06.096040+0000","last_unstale":"2024-09-18T11:44:20.981700+0000","last_undegraded":"2024-09-18T11:44:06.086523+0000","last_fullsized":"2024-09-18T11:44:06.086250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:36.983411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3060982,"num_objects":374,"num_object_clones":0,"num_object_copies":1496,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":378,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":376,"num_write_kb":3004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":372},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'387","reported_seq":446,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:20.825778+0000","last_change":"2024-09-18T11:44:06.110338+0000","last_active":"2024-09-18T11:44:20.825778+0000","last_peered":"2024-09-18T11:44:20.825778+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:06.100615+0000","last_became_peered":"2024-09-18T11:44:06.100615+0000","last_unstale":"2024-09-18T11:44:20.825778+0000","last_undegraded":"2024-09-18T11:44:06.093664+0000","last_fullsized":"2024-09-18T11:44:06.093409+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:20:02.888206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3152507,"num_objects":385,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":387,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":387,"num_write_kb":3086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":384},{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'385","reported_seq":425,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:20.990381+0000","last_change":"2024-09-18T11:44:06.108449+0000","last_active":"2024-09-18T11:44:20.990381+0000","last_peered":"2024-09-18T11:44:20.990381+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:06.099305+0000","last_became_peered":"2024-09-18T11:44:06.099305+0000","last_unstale":"2024-09-18T11:44:20.990381+0000","last_undegraded":"2024-09-18T11:44:06.085610+0000","last_fullsized":"2024-09-18T11:44:06.085286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:38:18.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3134710,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":387,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":381},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'687","reported_seq":717,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:24.061406+0000","last_change":"2024-09-18T11:44:06.102623+0000","last_active":"2024-09-18T11:44:24.061406+0000","last_peered":"2024-09-18T11:44:24.061406+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:06.095158+0000","last_became_peered":"2024-09-18T11:44:06.095158+0000","last_unstale":"2024-09-18T11:44:24.061406+0000","last_undegraded":"2024-09-18T11:44:06.087666+0000","last_fullsized":"2024-09-18T11:44:06.086930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:30.650564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5616886,"num_objects":686,"num_object_clones":0,"num_object_copies":2744,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":690,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":687,"num_write_kb":5493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":684},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'714","reported_seq":743,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:24.087774+0000","last_change":"2024-09-18T11:44:17.229813+0000","last_active":"2024-09-18T11:44:24.087774+0000","last_peered":"2024-09-18T11:44:24.087774+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:24.087774+0000","last_undegraded":"2024-09-18T11:44:17.216477+0000","last_fullsized":"2024-09-18T11:44:06.088316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:21:46.952968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849088,"num_objects":714,"num_object_clones":0,"num_object_copies":2856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":714,"num_write_kb":5712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'420","reported_seq":460,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:20.967948+0000","last_change":"2024-09-18T11:44:06.105109+0000","last_active":"2024-09-18T11:44:20.967948+0000","last_peered":"2024-09-18T11:44:20.967948+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:06.096306+0000","last_became_peered":"2024-09-18T11:44:06.096306+0000","last_unstale":"2024-09-18T11:44:20.967948+0000","last_undegraded":"2024-09-18T11:44:06.085856+0000","last_fullsized":"2024-09-18T11:44:06.085557+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:02:30.717874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3405046,"num_objects":416,"num_object_clones":0,"num_object_copies":1664,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":420,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":3340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":414},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'385","reported_seq":425,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:20.969942+0000","last_change":"2024-09-18T11:44:06.101426+0000","last_active":"2024-09-18T11:44:20.969942+0000","last_peered":"2024-09-18T11:44:20.969942+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:06.095136+0000","last_became_peered":"2024-09-18T11:44:06.095136+0000","last_unstale":"2024-09-18T11:44:20.969942+0000","last_undegraded":"2024-09-18T11:44:06.086038+0000","last_fullsized":"2024-09-18T11:44:06.085713+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:37:37.674520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3126518,"num_objects":382,"num_object_clones":0,"num_object_copies":1528,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":386,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":3068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":380},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'743","reported_seq":772,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:24.083577+0000","last_change":"2024-09-18T11:44:06.107730+0000","last_active":"2024-09-18T11:44:24.083577+0000","last_peered":"2024-09-18T11:44:24.083577+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:06.095233+0000","last_became_peered":"2024-09-18T11:44:06.095233+0000","last_unstale":"2024-09-18T11:44:24.083577+0000","last_undegraded":"2024-09-18T11:44:06.087588+0000","last_fullsized":"2024-09-18T11:44:06.087287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:59:35.894764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6040044,"num_objects":738,"num_object_clones":0,"num_object_copies":2952,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":744,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":743,"num_write_kb":5921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":735},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'393","reported_seq":451,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:20.802831+0000","last_change":"2024-09-18T11:44:06.108847+0000","last_active":"2024-09-18T11:44:20.802831+0000","last_peered":"2024-09-18T11:44:20.802831+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:06.100048+0000","last_became_peered":"2024-09-18T11:44:06.100048+0000","last_unstale":"2024-09-18T11:44:20.802831+0000","last_undegraded":"2024-09-18T11:44:06.093124+0000","last_fullsized":"2024-09-18T11:44:06.092822+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:32:38.210808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3216630,"num_objects":393,"num_object_clones":0,"num_object_copies":1572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":397,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":393,"num_write_kb":3142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":391},{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'716","reported_seq":745,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:24.091943+0000","last_change":"2024-09-18T11:44:06.110973+0000","last_active":"2024-09-18T11:44:24.091943+0000","last_peered":"2024-09-18T11:44:24.091943+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:06.099667+0000","last_became_peered":"2024-09-18T11:44:06.099667+0000","last_unstale":"2024-09-18T11:44:24.091943+0000","last_undegraded":"2024-09-18T11:44:06.087962+0000","last_fullsized":"2024-09-18T11:44:06.087548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:09:02.838475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5844849,"num_objects":714,"num_object_clones":0,"num_object_copies":2856,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":720,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":716,"num_write_kb":5716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":711},{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'702","reported_seq":720,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:24.083556+0000","last_change":"2024-09-18T11:44:06.109597+0000","last_active":"2024-09-18T11:44:24.083556+0000","last_peered":"2024-09-18T11:44:24.083556+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:24.083556+0000","last_undegraded":"2024-09-18T11:44:06.087360+0000","last_fullsized":"2024-09-18T11:44:06.087125+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:23:31.350777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5749371,"num_objects":702,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":704,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":702,"num_write_kb":5615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":701},{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'376","reported_seq":434,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:20.790356+0000","last_change":"2024-09-18T11:44:06.104303+0000","last_active":"2024-09-18T11:44:20.790356+0000","last_peered":"2024-09-18T11:44:20.790356+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:06.095142+0000","last_became_peered":"2024-09-18T11:44:06.095142+0000","last_unstale":"2024-09-18T11:44:20.790356+0000","last_undegraded":"2024-09-18T11:44:06.092217+0000","last_fullsized":"2024-09-18T11:44:06.091922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:07:49.011604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3078779,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":378,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":376,"num_write_kb":3007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":375},{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'747","reported_seq":773,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:24.076883+0000","last_change":"2024-09-18T11:44:06.105818+0000","last_active":"2024-09-18T11:44:24.076883+0000","last_peered":"2024-09-18T11:44:24.076883+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:06.095532+0000","last_became_peered":"2024-09-18T11:44:06.095532+0000","last_unstale":"2024-09-18T11:44:24.076883+0000","last_undegraded":"2024-09-18T11:44:06.087192+0000","last_fullsized":"2024-09-18T11:44:06.086893+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":747,"log_dups_size":0,"ondisk_log_size":747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:58:09.393977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6100214,"num_objects":745,"num_object_clones":0,"num_object_copies":2980,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":749,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":747,"num_write_kb":5972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":743},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'420","reported_seq":479,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:20.814085+0000","last_change":"2024-09-18T11:44:06.106603+0000","last_active":"2024-09-18T11:44:20.814085+0000","last_peered":"2024-09-18T11:44:20.814085+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:06.100417+0000","last_became_peered":"2024-09-18T11:44:06.100417+0000","last_unstale":"2024-09-18T11:44:20.814085+0000","last_undegraded":"2024-09-18T11:44:06.093073+0000","last_fullsized":"2024-09-18T11:44:06.092766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:43:46.116054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3429622,"num_objects":419,"num_object_clones":0,"num_object_copies":1676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":423,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":3357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":417},{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'397","reported_seq":436,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-18T11:44:20.990616+0000","last_change":"2024-09-18T11:44:06.107245+0000","last_active":"2024-09-18T11:44:20.990616+0000","last_peered":"2024-09-18T11:44:20.990616+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:06.096352+0000","last_became_peered":"2024-09-18T11:44:06.096352+0000","last_unstale":"2024-09-18T11:44:20.990616+0000","last_undegraded":"2024-09-18T11:44:06.086622+0000","last_fullsized":"2024-09-18T11:44:06.086322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:16:19.829390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3224822,"num_objects":394,"num_object_clones":0,"num_object_copies":1576,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":398,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":397,"num_write_kb":3164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":392},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":69813620,"num_objects":8527,"num_object_clones":0,"num_object_copies":34108,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":8581,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":8527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":68314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98004992,"data_stored":98004992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8554,"ondisk_log_size":8554,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353640,"kb_used_data":43296,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94018200,"statfs":{"total":96636764160,"available":96274636800,"internally_reserved":0,"allocated":44335104,"data_stored":44191515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,2,5,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0310000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322660,"kb_used_data":21148,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94049180,"statfs":{"total":96636764160,"available":96306360320,"internally_reserved":0,"allocated":21655552,"data_stored":21511963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,3,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":180224,"data_stored":83830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331860,"kb_used_data":25740,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":94039980,"statfs":{"total":96636764160,"available":96296939520,"internally_reserved":0,"allocated":26357760,"data_stored":26214171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,3,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26578944,"data_stored":26578944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25489408,"data_stored":25489408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45895680,"data_stored":45895680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:27.683 INFO:tasks.ec_lost_unfound:no unfound in 1.f 2024-09-18T11:44:27.683 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.e state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:27.683 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.e list_unfound '{}' 2024-09-18T11:44:27.928 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:27.928 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:27.928 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:27.928 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:27.928 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:27.928 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:27.928 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 2, 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 2, 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existing_4", 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 1224592654, 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'3", 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "16'1", 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0", 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: "2(3)" 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:27.930 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existing_6", 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 3101443614, 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'4", 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "16'2", 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0", 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:27.931 INFO:teuthology.orchestra.run.smithi071.stdout: "2(3)" 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(0)", 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:27.932 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:27.933 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:27.951 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_4', 'key': '', 'snapid': -2, 'hash': 1224592654, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0', 'locations': ['2(3)']}, {'oid': {'oid': 'existing_6', 'key': '', 'snapid': -2, 'hash': 3101443614, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-18T11:44:27.951 INFO:tasks.ec_lost_unfound:reverting unfound in 1.e 2024-09-18T11:44:27.951 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.e mark_unfound_lost delete 2024-09-18T11:44:28.192 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:44:28.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 11 20 16699 16679 11.8444 9.01562 0.0162034 0.0131768 2024-09-18T11:44:28.211 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.d state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:28.211 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.d list_unfound '{}' 2024-09-18T11:44:28.436 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:28.436 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 1, 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 1, 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:28.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existed_7", 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 2358579165, 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'3", 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: "2(0)" 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(1)", 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:28.439 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:28.440 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:28.453 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'existed_7', 'key': '', 'snapid': -2, 'hash': 2358579165, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-18T11:44:28.454 INFO:tasks.ec_lost_unfound:reverting unfound in 1.d 2024-09-18T11:44:28.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.d mark_unfound_lost delete 2024-09-18T11:44:28.703 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-18T11:44:28.727 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.0 state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:28.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.0 list_unfound '{}' 2024-09-18T11:44:28.935 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:28.935 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:28.935 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:28.935 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:28.935 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:28.935 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:28.935 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:28.935 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:28.935 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 2, 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 2, 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "new_5", 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 4248945696, 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:28.936 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'3", 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existed_5", 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:28.937 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 4212789968, 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'4", 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:28.938 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:28.939 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:28.939 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:28.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:28.939 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(0)", 2024-09-18T11:44:28.939 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:28.939 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:28.939 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:28.939 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:28.939 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:28.952 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_5', 'key': '', 'snapid': -2, 'hash': 4248945696, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_5', 'key': '', 'snapid': -2, 'hash': 4212789968, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-18T11:44:28.952 INFO:tasks.ec_lost_unfound:reverting unfound in 1.0 2024-09-18T11:44:28.952 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.0 mark_unfound_lost delete 2024-09-18T11:44:29.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 12 20 17757 17737 11.5461 8.26562 0.0205975 0.0135178 2024-09-18T11:44:29.224 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:44:29.248 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.1 state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:29.249 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.1 list_unfound '{}' 2024-09-18T11:44:29.462 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:29.462 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:29.462 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:29.462 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:29.462 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 2, 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 2, 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existing_3", 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 459768081, 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'3", 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "16'1", 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0", 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: "2(3)" 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:29.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "new_3", 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 480221105, 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'2", 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:29.465 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:29.466 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:29.466 INFO:teuthology.orchestra.run.smithi071.stdout: "2(3)" 2024-09-18T11:44:29.466 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:29.466 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:29.466 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:29.466 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:29.466 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:29.466 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:29.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:29.466 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(1)", 2024-09-18T11:44:29.467 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:29.467 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:29.467 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:29.467 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:29.467 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:29.483 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_3', 'key': '', 'snapid': -2, 'hash': 459768081, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0', 'locations': ['2(3)']}, {'oid': {'oid': 'new_3', 'key': '', 'snapid': -2, 'hash': 480221105, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-18T11:44:29.483 INFO:tasks.ec_lost_unfound:reverting unfound in 1.1 2024-09-18T11:44:29.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.1 mark_unfound_lost delete 2024-09-18T11:44:29.711 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:44:29.734 INFO:tasks.ec_lost_unfound:no unfound in 1.c 2024-09-18T11:44:29.734 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.2 state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:29.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.2 list_unfound '{}' 2024-09-18T11:44:29.952 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:29.952 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:29.952 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:29.952 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 2, 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 2, 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existed_6", 2024-09-18T11:44:29.953 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:29.954 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:29.954 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 1708673474, 2024-09-18T11:44:29.954 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:29.954 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:29.954 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:29.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:29.954 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'5", 2024-09-18T11:44:29.954 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:29.954 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "2(3)" 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existed_9", 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 831402034, 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:29.955 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'6", 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: "2(3)" 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:29.956 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:29.957 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:29.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:29.957 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(2)", 2024-09-18T11:44:29.957 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:29.957 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:29.957 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:29.957 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:29.957 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:29.972 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_6', 'key': '', 'snapid': -2, 'hash': 1708673474, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}, {'oid': {'oid': 'existed_9', 'key': '', 'snapid': -2, 'hash': 831402034, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'6", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-18T11:44:29.973 INFO:tasks.ec_lost_unfound:reverting unfound in 1.2 2024-09-18T11:44:29.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.2 mark_unfound_lost delete 2024-09-18T11:44:30.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 13 20 18871 18851 11.3274 8.70312 0.025711 0.0137817 2024-09-18T11:44:30.211 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:44:30.236 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.3 state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:30.236 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.3 list_unfound '{}' 2024-09-18T11:44:30.448 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 2, 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 2, 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existing_8", 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 1901787523, 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'5", 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "16'3", 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0", 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existed_1", 2024-09-18T11:44:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 3122379363, 2024-09-18T11:44:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'4", 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(0)", 2024-09-18T11:44:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:30.453 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:30.477 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_8', 'key': '', 'snapid': -2, 'hash': 1901787523, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "16'3", 'flags': 'none', 'clean_regions': 'clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_1', 'key': '', 'snapid': -2, 'hash': 3122379363, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-18T11:44:30.477 INFO:tasks.ec_lost_unfound:reverting unfound in 1.3 2024-09-18T11:44:30.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.3 mark_unfound_lost delete 2024-09-18T11:44:30.687 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:44:30.708 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.4 state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:30.708 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.4 list_unfound '{}' 2024-09-18T11:44:30.932 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:30.932 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 3, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 3, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existed_4", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 873891716, 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'8", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existing_7", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 3996155460, 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'9", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "16'6", 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0", 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existed_2", 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 1767667508, 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'7", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(3)", 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:30.951 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_4', 'key': '', 'snapid': -2, 'hash': 873891716, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'8", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_7', 'key': '', 'snapid': -2, 'hash': 3996155460, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'9", 'have': "16'6", 'flags': 'none', 'clean_regions': 'clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_2', 'key': '', 'snapid': -2, 'hash': 1767667508, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'7", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-18T11:44:30.951 INFO:tasks.ec_lost_unfound:reverting unfound in 1.4 2024-09-18T11:44:30.951 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.4 mark_unfound_lost delete 2024-09-18T11:44:31.189 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-18T11:44:31.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 14 20 19929 19909 11.1086 8.26562 0.0190696 0.0140507 2024-09-18T11:44:31.214 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.5 state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:31.215 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.5 list_unfound '{}' 2024-09-18T11:44:31.432 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:31.432 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:31.432 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:31.432 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:31.432 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:31.432 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:31.432 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:31.432 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:31.432 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 2, 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 2, 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "new_1", 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 26281365, 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:31.433 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'1", 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: "2(0)" 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "new_2", 2024-09-18T11:44:31.434 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 3196340981, 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'2", 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout: "2(0)" 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(3)", 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:31.453 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_1', 'key': '', 'snapid': -2, 'hash': 26281365, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(0)']}, {'oid': {'oid': 'new_2', 'key': '', 'snapid': -2, 'hash': 3196340981, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-18T11:44:31.453 INFO:tasks.ec_lost_unfound:reverting unfound in 1.5 2024-09-18T11:44:31.453 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.5 mark_unfound_lost delete 2024-09-18T11:44:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:44:31.717 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.6 state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:31.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.6 list_unfound '{}' 2024-09-18T11:44:31.931 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:31.931 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:31.931 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 3, 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 3, 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:31.932 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existed_8", 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 1861424966, 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'5", 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:31.933 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "2(3)" 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "new_6", 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 3910131350, 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'4", 2024-09-18T11:44:31.934 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "2(3)" 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "new_4", 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 2787839990, 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:31.935 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'3", 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: "2(3)" 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:31.936 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:31.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:31.937 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(2)", 2024-09-18T11:44:31.937 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:31.937 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:31.937 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:31.937 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:31.937 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:31.949 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_8', 'key': '', 'snapid': -2, 'hash': 1861424966, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}, {'oid': {'oid': 'new_6', 'key': '', 'snapid': -2, 'hash': 3910131350, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}, {'oid': {'oid': 'new_4', 'key': '', 'snapid': -2, 'hash': 2787839990, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-18T11:44:31.949 INFO:tasks.ec_lost_unfound:reverting unfound in 1.6 2024-09-18T11:44:31.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.6 mark_unfound_lost delete 2024-09-18T11:44:32.011 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-09-18T11:44:32.009+0000 7f0506ff6640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 9 pgs recovery_unfound (PG_DAMAGED) 2024-09-18T11:44:32.196 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-18T11:44:32.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 15 20 21009 20989 10.9305 8.4375 0.0117714 0.0142823 2024-09-18T11:44:32.222 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.7 state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:32.222 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.7 list_unfound '{}' 2024-09-18T11:44:32.463 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:32.463 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:32.463 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:32.463 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 1, 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 1, 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "new_7", 2024-09-18T11:44:32.464 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 3761017527, 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'1", 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:32.465 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(0)", 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:32.466 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:32.482 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_7', 'key': '', 'snapid': -2, 'hash': 3761017527, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-18T11:44:32.483 INFO:tasks.ec_lost_unfound:reverting unfound in 1.7 2024-09-18T11:44:32.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.7 mark_unfound_lost delete 2024-09-18T11:44:32.725 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-18T11:44:32.745 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.8 state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:32.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.8 list_unfound '{}' 2024-09-18T11:44:33.112 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:44:33.111+0000 7f5d1284c640 0 --2- 172.21.15.71:0/74829659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d0c091b20 0x7f5d0c122260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T11:44:33.133 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:33.133 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:33.133 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:33.133 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 1, 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 1, 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:33.134 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "new_8", 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 2730327960, 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'1", 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:33.135 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:33.136 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:33.136 INFO:teuthology.orchestra.run.smithi071.stdout: "2(1)" 2024-09-18T11:44:33.136 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:33.136 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:33.136 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:33.136 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:33.137 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:33.137 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:33.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:33.137 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(0)", 2024-09-18T11:44:33.137 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:33.138 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:33.138 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:33.138 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:33.138 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:33.143 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_8', 'key': '', 'snapid': -2, 'hash': 2730327960, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-18T11:44:33.143 INFO:tasks.ec_lost_unfound:reverting unfound in 1.8 2024-09-18T11:44:33.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.8 mark_unfound_lost delete 2024-09-18T11:44:33.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 16 20 21956 21936 10.7097 7.39844 0.00539531 0.0144297 2024-09-18T11:44:33.324 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-18T11:44:33.345 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.9 state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:33.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.9 list_unfound '{}' 2024-09-18T11:44:33.566 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:33.566 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:33.566 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:33.566 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:33.566 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:33.567 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:33.567 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:33.567 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:33.567 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:33.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:33.567 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 2, 2024-09-18T11:44:33.568 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 2, 2024-09-18T11:44:33.568 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:33.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:33.568 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:33.568 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existing_5", 2024-09-18T11:44:33.568 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:33.569 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:33.569 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 2998549577, 2024-09-18T11:44:33.569 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:33.569 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:33.569 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:33.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:33.570 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'4", 2024-09-18T11:44:33.570 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "16'2", 2024-09-18T11:44:33.570 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:33.570 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0", 2024-09-18T11:44:33.570 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:33.571 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:33.571 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:33.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:33.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:33.571 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:33.571 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existing_2", 2024-09-18T11:44:33.572 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:33.572 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:33.572 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 4032093161, 2024-09-18T11:44:33.572 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:33.572 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:33.572 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:33.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:33.573 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'3", 2024-09-18T11:44:33.573 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "16'1", 2024-09-18T11:44:33.573 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:33.573 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0", 2024-09-18T11:44:33.573 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:33.573 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:33.574 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:33.574 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:33.574 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:33.574 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:33.574 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:33.574 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:33.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:33.575 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(0)", 2024-09-18T11:44:33.575 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:33.575 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:33.575 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:33.575 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:33.576 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:33.591 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_5', 'key': '', 'snapid': -2, 'hash': 2998549577, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_2', 'key': '', 'snapid': -2, 'hash': 4032093161, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-18T11:44:33.592 INFO:tasks.ec_lost_unfound:reverting unfound in 1.9 2024-09-18T11:44:33.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.9 mark_unfound_lost delete 2024-09-18T11:44:33.826 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:44:33.847 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.a state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:33.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.a list_unfound '{}' 2024-09-18T11:44:34.054 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:34.054 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:34.054 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:34.054 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:34.054 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:34.054 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 2, 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 2, 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existing_9", 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 1217469130, 2024-09-18T11:44:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'3", 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "16'1", 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0", 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "2(1)" 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "new_9", 2024-09-18T11:44:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 170713514, 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'2", 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: "2(1)" 2024-09-18T11:44:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(0)", 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:34.058 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:34.079 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_9', 'key': '', 'snapid': -2, 'hash': 1217469130, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0', 'locations': ['2(1)']}, {'oid': {'oid': 'new_9', 'key': '', 'snapid': -2, 'hash': 170713514, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-18T11:44:34.080 INFO:tasks.ec_lost_unfound:reverting unfound in 1.a 2024-09-18T11:44:34.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.a mark_unfound_lost delete 2024-09-18T11:44:34.206 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 17 20 22996 22976 10.5577 8.125 0.0163165 0.0147877 2024-09-18T11:44:34.320 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:44:34.343 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.b state active+recovery_unfound+undersized+degraded 2024-09-18T11:44:34.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.b list_unfound '{}' 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "offset": { 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "", 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": 0, 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 0, 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": -9223372036854775808, 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "num_missing": 2, 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "num_unfound": 2, 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "objects": [ 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existed_3", 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 943960171, 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'5", 2024-09-18T11:44:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "0'0", 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": { 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "oid": "existing_1", 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "snapid": -2, 2024-09-18T11:44:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "hash": 2885884443, 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 0, 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "pool": 1, 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "namespace": "" 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "need": "19'4", 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "have": "16'1", 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "flags": "none", 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "clean_regions": "clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0", 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "locations": [ 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "2(2)" 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "NotRecovering", 2024-09-18T11:44:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: "available_might_have_unfound": true, 2024-09-18T11:44:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: "might_have_unfound": [ 2024-09-18T11:44:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: "osd": "1(0)", 2024-09-18T11:44:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "osd is down" 2024-09-18T11:44:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: "more": false 2024-09-18T11:44:34.574 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:34.595 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_3', 'key': '', 'snapid': -2, 'hash': 943960171, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_1', 'key': '', 'snapid': -2, 'hash': 2885884443, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [6779~18446744073709544836], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-18T11:44:34.595 INFO:tasks.ec_lost_unfound:reverting unfound in 1.b 2024-09-18T11:44:34.595 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.b mark_unfound_lost delete 2024-09-18T11:44:34.822 INFO:teuthology.orchestra.run.smithi071.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:44:34.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-09-18T11:44:35.061 INFO:teuthology.orchestra.run.smithi071.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-18T11:44:35.081 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-09-18T11:44:35.207 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 18 20 24012 23992 10.4118 7.9375 0.023142 0.0149945 2024-09-18T11:44:35.296 INFO:teuthology.orchestra.run.smithi071.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-18T11:44:35.321 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 5 2024-09-18T11:44:35.517 INFO:teuthology.orchestra.run.smithi071.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-18T11:44:35.536 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:44:35.536 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T11:44:35.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T11:44:35.788 INFO:teuthology.orchestra.run.smithi071.stdout:98784247819 2024-09-18T11:44:35.788 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:44:35.791 INFO:teuthology.orchestra.run.smithi071.stdout:111669149706 2024-09-18T11:44:35.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T11:44:35.800 INFO:teuthology.orchestra.run.smithi071.stdout:38654705681 2024-09-18T11:44:35.800 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:44:36.207 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 19 20 24667 24647 10.1331 5.11719 0.0266925 0.0153989 2024-09-18T11:44:36.234 INFO:teuthology.orchestra.run.smithi071.stdout:98784247819 2024-09-18T11:44:36.258 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247819 got 98784247819 for osd.0 2024-09-18T11:44:36.258 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:36.278 INFO:teuthology.orchestra.run.smithi071.stdout:38654705681 2024-09-18T11:44:36.287 INFO:teuthology.orchestra.run.smithi071.stdout:111669149706 2024-09-18T11:44:36.294 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705681 got 38654705681 for osd.2 2024-09-18T11:44:36.294 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:36.299 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149706 got 111669149706 for osd.3 2024-09-18T11:44:36.300 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:36.300 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-18T11:44:36.300 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:36.811 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:36.833 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-18T11:44:35.828066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":200230260,"num_objects":24447,"num_object_clones":0,"num_object_copies":97788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24474,"num_write_kb":195674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24501,"ondisk_log_size":24501,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1520588,"kb_used_data":288620,"kb_used_omap":12,"kb_used_meta":1231923,"kb_avail":375966772,"statfs":{"total":386547056640,"available":384989974528,"internally_reserved":0,"allocated":295546880,"data_stored":295019719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12494,"internal_metadata":1261489970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,6,17,18,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":159268864,"num_objects":19442,"num_object_clones":0,"num_object_copies":77768,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":19388,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":19442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19442,"num_write_kb":155536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003737"},"pg_stats":[{"pgid":"1.f","version":"27'1560","reported_seq":1589,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.725935+0000","last_change":"2024-09-18T11:44:17.230219+0000","last_active":"2024-09-18T11:44:35.725935+0000","last_peered":"2024-09-18T11:44:35.725935+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095253+0000","last_became_peered":"2024-09-18T11:44:06.095253+0000","last_unstale":"2024-09-18T11:44:35.725935+0000","last_undegraded":"2024-09-18T11:44:17.229747+0000","last_fullsized":"2024-09-18T11:44:06.087120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:29.394125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":1560,"num_object_clones":0,"num_object_copies":6240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1560,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1560","reported_seq":1610,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.715375+0000","last_change":"2024-09-18T11:44:28.260359+0000","last_active":"2024-09-18T11:44:35.715375+0000","last_peered":"2024-09-18T11:44:35.715375+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:06.096040+0000","last_became_peered":"2024-09-18T11:44:06.096040+0000","last_unstale":"2024-09-18T11:44:35.715375+0000","last_undegraded":"2024-09-18T11:44:06.086523+0000","last_fullsized":"2024-09-18T11:44:06.086250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12743926,"num_objects":1556,"num_object_clones":0,"num_object_copies":6224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1558,"num_write_kb":12460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1556}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'1499","reported_seq":1565,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.733699+0000","last_change":"2024-09-18T11:44:28.747915+0000","last_active":"2024-09-18T11:44:35.733699+0000","last_peered":"2024-09-18T11:44:35.733699+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:06.100615+0000","last_became_peered":"2024-09-18T11:44:06.100615+0000","last_unstale":"2024-09-18T11:44:35.733699+0000","last_undegraded":"2024-09-18T11:44:06.093664+0000","last_fullsized":"2024-09-18T11:44:06.093409+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1499,"log_dups_size":0,"ondisk_log_size":1499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12253819,"num_objects":1496,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1498,"num_write_kb":11974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1529","reported_seq":1579,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.713191+0000","last_change":"2024-09-18T11:44:29.314915+0000","last_active":"2024-09-18T11:44:35.713191+0000","last_peered":"2024-09-18T11:44:35.713191+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:06.099305+0000","last_became_peered":"2024-09-18T11:44:06.099305+0000","last_unstale":"2024-09-18T11:44:35.713191+0000","last_undegraded":"2024-09-18T11:44:06.085610+0000","last_fullsized":"2024-09-18T11:44:06.085286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1529,"log_dups_size":0,"ondisk_log_size":1529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12489974,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1527,"num_write_kb":12205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1525}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'1552","reported_seq":1592,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.724407+0000","last_change":"2024-09-18T11:44:29.785182+0000","last_active":"2024-09-18T11:44:35.724407+0000","last_peered":"2024-09-18T11:44:35.724407+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:06.095158+0000","last_became_peered":"2024-09-18T11:44:06.095158+0000","last_unstale":"2024-09-18T11:44:35.724407+0000","last_undegraded":"2024-09-18T11:44:06.087666+0000","last_fullsized":"2024-09-18T11:44:06.086930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1552,"log_dups_size":0,"ondisk_log_size":1552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12686582,"num_objects":1549,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1550,"num_write_kb":12397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1596","reported_seq":1624,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.762734+0000","last_change":"2024-09-18T11:44:17.229813+0000","last_active":"2024-09-18T11:44:35.762734+0000","last_peered":"2024-09-18T11:44:35.762734+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:35.762734+0000","last_undegraded":"2024-09-18T11:44:17.216477+0000","last_fullsized":"2024-09-18T11:44:06.088316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1596,"log_dups_size":0,"ondisk_log_size":1596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:21:46.952968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13074432,"num_objects":1596,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1596,"num_write_kb":12768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'1566","reported_seq":1616,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.692637+0000","last_change":"2024-09-18T11:44:30.308647+0000","last_active":"2024-09-18T11:44:35.692637+0000","last_peered":"2024-09-18T11:44:35.692637+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:06.096306+0000","last_became_peered":"2024-09-18T11:44:06.096306+0000","last_unstale":"2024-09-18T11:44:35.692637+0000","last_undegraded":"2024-09-18T11:44:06.085856+0000","last_fullsized":"2024-09-18T11:44:06.085557+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1566,"log_dups_size":0,"ondisk_log_size":1566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12776694,"num_objects":1560,"num_object_clones":0,"num_object_copies":6240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1564,"num_write_kb":12492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1560}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1490","reported_seq":1540,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.705225+0000","last_change":"2024-09-18T11:44:30.758028+0000","last_active":"2024-09-18T11:44:35.705225+0000","last_peered":"2024-09-18T11:44:35.705225+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:06.095136+0000","last_became_peered":"2024-09-18T11:44:06.095136+0000","last_unstale":"2024-09-18T11:44:35.705225+0000","last_undegraded":"2024-09-18T11:44:06.086038+0000","last_fullsized":"2024-09-18T11:44:06.085713+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12162294,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1488,"num_write_kb":11892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1598","reported_seq":1638,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.757472+0000","last_change":"2024-09-18T11:44:31.315185+0000","last_active":"2024-09-18T11:44:35.757472+0000","last_peered":"2024-09-18T11:44:35.757472+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:06.095233+0000","last_became_peered":"2024-09-18T11:44:06.095233+0000","last_unstale":"2024-09-18T11:44:35.757472+0000","last_undegraded":"2024-09-18T11:44:06.087588+0000","last_fullsized":"2024-09-18T11:44:06.087287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1598,"log_dups_size":0,"ondisk_log_size":1598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13019628,"num_objects":1590,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1595,"num_write_kb":12737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'1463","reported_seq":1530,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.748355+0000","last_change":"2024-09-18T11:44:31.783656+0000","last_active":"2024-09-18T11:44:35.748355+0000","last_peered":"2024-09-18T11:44:35.748355+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:06.100048+0000","last_became_peered":"2024-09-18T11:44:06.100048+0000","last_unstale":"2024-09-18T11:44:35.748355+0000","last_undegraded":"2024-09-18T11:44:06.093124+0000","last_fullsized":"2024-09-18T11:44:06.092822+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1463,"log_dups_size":0,"ondisk_log_size":1463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11965686,"num_objects":1461,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":11686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1518","reported_seq":1560,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.755226+0000","last_change":"2024-09-18T11:44:32.316693+0000","last_active":"2024-09-18T11:44:35.755226+0000","last_peered":"2024-09-18T11:44:35.755226+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:06.099667+0000","last_became_peered":"2024-09-18T11:44:06.099667+0000","last_unstale":"2024-09-18T11:44:35.755226+0000","last_undegraded":"2024-09-18T11:44:06.087962+0000","last_fullsized":"2024-09-18T11:44:06.087548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12390257,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1515,"num_write_kb":12108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1525","reported_seq":1549,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.740926+0000","last_change":"2024-09-18T11:44:32.775850+0000","last_active":"2024-09-18T11:44:35.740926+0000","last_peered":"2024-09-18T11:44:35.740926+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:35.740926+0000","last_undegraded":"2024-09-18T11:44:06.087360+0000","last_fullsized":"2024-09-18T11:44:06.087125+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1525,"log_dups_size":0,"ondisk_log_size":1525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12483195,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":12191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1455","reported_seq":1520,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.706211+0000","last_change":"2024-09-18T11:44:33.362854+0000","last_active":"2024-09-18T11:44:35.706211+0000","last_peered":"2024-09-18T11:44:35.706211+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:06.095142+0000","last_became_peered":"2024-09-18T11:44:06.095142+0000","last_unstale":"2024-09-18T11:44:35.706211+0000","last_undegraded":"2024-09-18T11:44:06.092217+0000","last_fullsized":"2024-09-18T11:44:06.091922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1455,"log_dups_size":0,"ondisk_log_size":1455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11909755,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1454,"num_write_kb":11631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1454}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1580","reported_seq":1617,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.734406+0000","last_change":"2024-09-18T11:44:33.907840+0000","last_active":"2024-09-18T11:44:35.734406+0000","last_peered":"2024-09-18T11:44:35.734406+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:06.095532+0000","last_became_peered":"2024-09-18T11:44:06.095532+0000","last_unstale":"2024-09-18T11:44:35.734406+0000","last_undegraded":"2024-09-18T11:44:06.087192+0000","last_fullsized":"2024-09-18T11:44:06.086893+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1580,"log_dups_size":0,"ondisk_log_size":1580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12907766,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1578,"num_write_kb":12620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1576}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1530","reported_seq":1599,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.731082+0000","last_change":"2024-09-18T11:44:34.409540+0000","last_active":"2024-09-18T11:44:35.731082+0000","last_peered":"2024-09-18T11:44:35.731082+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:06.100417+0000","last_became_peered":"2024-09-18T11:44:06.100417+0000","last_unstale":"2024-09-18T11:44:35.731082+0000","last_undegraded":"2024-09-18T11:44:06.093073+0000","last_fullsized":"2024-09-18T11:44:06.092766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12506358,"num_objects":1527,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1527,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1528,"num_write_kb":12221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1527}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1480","reported_seq":1530,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.720732+0000","last_change":"2024-09-18T11:44:34.900312+0000","last_active":"2024-09-18T11:44:35.720732+0000","last_peered":"2024-09-18T11:44:35.720732+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:06.096352+0000","last_became_peered":"2024-09-18T11:44:06.096352+0000","last_unstale":"2024-09-18T11:44:35.720732+0000","last_undegraded":"2024-09-18T11:44:06.086622+0000","last_fullsized":"2024-09-18T11:44:06.086322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12080374,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1478,"num_write_kb":11812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":200230260,"num_objects":24447,"num_object_clones":0,"num_object_copies":97788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24474,"num_write_kb":195674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300208128,"data_stored":300208128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24501,"ondisk_log_size":24501,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408112,"kb_used_data":97768,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93963728,"statfs":{"total":96636764160,"available":96218857472,"internally_reserved":0,"allocated":100114432,"data_stored":99970843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,5,12,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5569999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409676,"kb_used_data":95684,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93962164,"statfs":{"total":96636764160,"available":96217255936,"internally_reserved":0,"allocated":97980416,"data_stored":97836827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3520000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":180224,"data_stored":83830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405272,"kb_used_data":94992,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93966568,"statfs":{"total":96636764160,"available":96221765632,"internally_reserved":0,"allocated":97271808,"data_stored":97128219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,7,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3420000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100024320,"data_stored":100024320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100093952,"data_stored":100093952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":100048896,"data_stored":100048896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:36.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:37.207 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:44:37.207239+0000 min lat: 0.00272854 max lat: 0.228235 avg lat: 0.0156568 2024-09-18T11:44:37.208 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:44:37.208 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 20 20 25550 25530 9.97131 6.89844 0.0156868 0.0156568 2024-09-18T11:44:37.348 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:37.348 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:37.366 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-18T11:44:35.828066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":200230260,"num_objects":24447,"num_object_clones":0,"num_object_copies":97788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24474,"num_write_kb":195674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24501,"ondisk_log_size":24501,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1520588,"kb_used_data":288620,"kb_used_omap":12,"kb_used_meta":1231923,"kb_avail":375966772,"statfs":{"total":386547056640,"available":384989974528,"internally_reserved":0,"allocated":295546880,"data_stored":295019719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12494,"internal_metadata":1261489970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,6,17,18,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":159268864,"num_objects":19442,"num_object_clones":0,"num_object_copies":77768,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":19388,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":19442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19442,"num_write_kb":155536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003737"},"pg_stats":[{"pgid":"1.f","version":"27'1560","reported_seq":1589,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.725935+0000","last_change":"2024-09-18T11:44:17.230219+0000","last_active":"2024-09-18T11:44:35.725935+0000","last_peered":"2024-09-18T11:44:35.725935+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095253+0000","last_became_peered":"2024-09-18T11:44:06.095253+0000","last_unstale":"2024-09-18T11:44:35.725935+0000","last_undegraded":"2024-09-18T11:44:17.229747+0000","last_fullsized":"2024-09-18T11:44:06.087120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:29.394125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":1560,"num_object_clones":0,"num_object_copies":6240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1560,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1560","reported_seq":1610,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.715375+0000","last_change":"2024-09-18T11:44:28.260359+0000","last_active":"2024-09-18T11:44:35.715375+0000","last_peered":"2024-09-18T11:44:35.715375+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:06.096040+0000","last_became_peered":"2024-09-18T11:44:06.096040+0000","last_unstale":"2024-09-18T11:44:35.715375+0000","last_undegraded":"2024-09-18T11:44:06.086523+0000","last_fullsized":"2024-09-18T11:44:06.086250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12743926,"num_objects":1556,"num_object_clones":0,"num_object_copies":6224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1558,"num_write_kb":12460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1556}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'1499","reported_seq":1565,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.733699+0000","last_change":"2024-09-18T11:44:28.747915+0000","last_active":"2024-09-18T11:44:35.733699+0000","last_peered":"2024-09-18T11:44:35.733699+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:06.100615+0000","last_became_peered":"2024-09-18T11:44:06.100615+0000","last_unstale":"2024-09-18T11:44:35.733699+0000","last_undegraded":"2024-09-18T11:44:06.093664+0000","last_fullsized":"2024-09-18T11:44:06.093409+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1499,"log_dups_size":0,"ondisk_log_size":1499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12253819,"num_objects":1496,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1498,"num_write_kb":11974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1529","reported_seq":1579,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.713191+0000","last_change":"2024-09-18T11:44:29.314915+0000","last_active":"2024-09-18T11:44:35.713191+0000","last_peered":"2024-09-18T11:44:35.713191+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:06.099305+0000","last_became_peered":"2024-09-18T11:44:06.099305+0000","last_unstale":"2024-09-18T11:44:35.713191+0000","last_undegraded":"2024-09-18T11:44:06.085610+0000","last_fullsized":"2024-09-18T11:44:06.085286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1529,"log_dups_size":0,"ondisk_log_size":1529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12489974,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1527,"num_write_kb":12205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1525}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'1552","reported_seq":1592,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.724407+0000","last_change":"2024-09-18T11:44:29.785182+0000","last_active":"2024-09-18T11:44:35.724407+0000","last_peered":"2024-09-18T11:44:35.724407+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:06.095158+0000","last_became_peered":"2024-09-18T11:44:06.095158+0000","last_unstale":"2024-09-18T11:44:35.724407+0000","last_undegraded":"2024-09-18T11:44:06.087666+0000","last_fullsized":"2024-09-18T11:44:06.086930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1552,"log_dups_size":0,"ondisk_log_size":1552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12686582,"num_objects":1549,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1550,"num_write_kb":12397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1596","reported_seq":1624,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.762734+0000","last_change":"2024-09-18T11:44:17.229813+0000","last_active":"2024-09-18T11:44:35.762734+0000","last_peered":"2024-09-18T11:44:35.762734+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:35.762734+0000","last_undegraded":"2024-09-18T11:44:17.216477+0000","last_fullsized":"2024-09-18T11:44:06.088316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1596,"log_dups_size":0,"ondisk_log_size":1596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:21:46.952968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13074432,"num_objects":1596,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1596,"num_write_kb":12768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'1566","reported_seq":1616,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.692637+0000","last_change":"2024-09-18T11:44:30.308647+0000","last_active":"2024-09-18T11:44:35.692637+0000","last_peered":"2024-09-18T11:44:35.692637+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:06.096306+0000","last_became_peered":"2024-09-18T11:44:06.096306+0000","last_unstale":"2024-09-18T11:44:35.692637+0000","last_undegraded":"2024-09-18T11:44:06.085856+0000","last_fullsized":"2024-09-18T11:44:06.085557+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1566,"log_dups_size":0,"ondisk_log_size":1566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12776694,"num_objects":1560,"num_object_clones":0,"num_object_copies":6240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1564,"num_write_kb":12492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1560}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1490","reported_seq":1540,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.705225+0000","last_change":"2024-09-18T11:44:30.758028+0000","last_active":"2024-09-18T11:44:35.705225+0000","last_peered":"2024-09-18T11:44:35.705225+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:06.095136+0000","last_became_peered":"2024-09-18T11:44:06.095136+0000","last_unstale":"2024-09-18T11:44:35.705225+0000","last_undegraded":"2024-09-18T11:44:06.086038+0000","last_fullsized":"2024-09-18T11:44:06.085713+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12162294,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1488,"num_write_kb":11892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1598","reported_seq":1638,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.757472+0000","last_change":"2024-09-18T11:44:31.315185+0000","last_active":"2024-09-18T11:44:35.757472+0000","last_peered":"2024-09-18T11:44:35.757472+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:06.095233+0000","last_became_peered":"2024-09-18T11:44:06.095233+0000","last_unstale":"2024-09-18T11:44:35.757472+0000","last_undegraded":"2024-09-18T11:44:06.087588+0000","last_fullsized":"2024-09-18T11:44:06.087287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1598,"log_dups_size":0,"ondisk_log_size":1598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13019628,"num_objects":1590,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1595,"num_write_kb":12737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'1463","reported_seq":1530,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.748355+0000","last_change":"2024-09-18T11:44:31.783656+0000","last_active":"2024-09-18T11:44:35.748355+0000","last_peered":"2024-09-18T11:44:35.748355+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:06.100048+0000","last_became_peered":"2024-09-18T11:44:06.100048+0000","last_unstale":"2024-09-18T11:44:35.748355+0000","last_undegraded":"2024-09-18T11:44:06.093124+0000","last_fullsized":"2024-09-18T11:44:06.092822+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1463,"log_dups_size":0,"ondisk_log_size":1463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11965686,"num_objects":1461,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":11686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1518","reported_seq":1560,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.755226+0000","last_change":"2024-09-18T11:44:32.316693+0000","last_active":"2024-09-18T11:44:35.755226+0000","last_peered":"2024-09-18T11:44:35.755226+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:06.099667+0000","last_became_peered":"2024-09-18T11:44:06.099667+0000","last_unstale":"2024-09-18T11:44:35.755226+0000","last_undegraded":"2024-09-18T11:44:06.087962+0000","last_fullsized":"2024-09-18T11:44:06.087548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12390257,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1515,"num_write_kb":12108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1525","reported_seq":1549,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.740926+0000","last_change":"2024-09-18T11:44:32.775850+0000","last_active":"2024-09-18T11:44:35.740926+0000","last_peered":"2024-09-18T11:44:35.740926+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:35.740926+0000","last_undegraded":"2024-09-18T11:44:06.087360+0000","last_fullsized":"2024-09-18T11:44:06.087125+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1525,"log_dups_size":0,"ondisk_log_size":1525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12483195,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":12191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1455","reported_seq":1520,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.706211+0000","last_change":"2024-09-18T11:44:33.362854+0000","last_active":"2024-09-18T11:44:35.706211+0000","last_peered":"2024-09-18T11:44:35.706211+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:06.095142+0000","last_became_peered":"2024-09-18T11:44:06.095142+0000","last_unstale":"2024-09-18T11:44:35.706211+0000","last_undegraded":"2024-09-18T11:44:06.092217+0000","last_fullsized":"2024-09-18T11:44:06.091922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1455,"log_dups_size":0,"ondisk_log_size":1455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11909755,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1454,"num_write_kb":11631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1454}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1580","reported_seq":1617,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.734406+0000","last_change":"2024-09-18T11:44:33.907840+0000","last_active":"2024-09-18T11:44:35.734406+0000","last_peered":"2024-09-18T11:44:35.734406+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:06.095532+0000","last_became_peered":"2024-09-18T11:44:06.095532+0000","last_unstale":"2024-09-18T11:44:35.734406+0000","last_undegraded":"2024-09-18T11:44:06.087192+0000","last_fullsized":"2024-09-18T11:44:06.086893+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1580,"log_dups_size":0,"ondisk_log_size":1580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12907766,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1578,"num_write_kb":12620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1576}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1530","reported_seq":1599,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.731082+0000","last_change":"2024-09-18T11:44:34.409540+0000","last_active":"2024-09-18T11:44:35.731082+0000","last_peered":"2024-09-18T11:44:35.731082+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:06.100417+0000","last_became_peered":"2024-09-18T11:44:06.100417+0000","last_unstale":"2024-09-18T11:44:35.731082+0000","last_undegraded":"2024-09-18T11:44:06.093073+0000","last_fullsized":"2024-09-18T11:44:06.092766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12506358,"num_objects":1527,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1527,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1528,"num_write_kb":12221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1527}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1480","reported_seq":1530,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:35.720732+0000","last_change":"2024-09-18T11:44:34.900312+0000","last_active":"2024-09-18T11:44:35.720732+0000","last_peered":"2024-09-18T11:44:35.720732+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:06.096352+0000","last_became_peered":"2024-09-18T11:44:06.096352+0000","last_unstale":"2024-09-18T11:44:35.720732+0000","last_undegraded":"2024-09-18T11:44:06.086622+0000","last_fullsized":"2024-09-18T11:44:06.086322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12080374,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1478,"num_write_kb":11812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":200230260,"num_objects":24447,"num_object_clones":0,"num_object_copies":97788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24474,"num_write_kb":195674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300208128,"data_stored":300208128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24501,"ondisk_log_size":24501,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408112,"kb_used_data":97768,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93963728,"statfs":{"total":96636764160,"available":96218857472,"internally_reserved":0,"allocated":100114432,"data_stored":99970843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,5,12,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5569999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409676,"kb_used_data":95684,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93962164,"statfs":{"total":96636764160,"available":96217255936,"internally_reserved":0,"allocated":97980416,"data_stored":97836827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3520000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":180224,"data_stored":83830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405272,"kb_used_data":94992,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93966568,"statfs":{"total":96636764160,"available":96221765632,"internally_reserved":0,"allocated":97271808,"data_stored":97128219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,7,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3420000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100024320,"data_stored":100024320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100093952,"data_stored":100093952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":100048896,"data_stored":100048896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:37.367 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-18T11:44:37.368 INFO:tasks.util.rados:rados -p unique_pool_0 get new_1 - 2024-09-18T11:44:37.368 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_1 - 2024-09-18T11:44:37.449 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/new_1: (2) No such file or directory 2024-09-18T11:44:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:37.465 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_1 - 2024-09-18T11:44:37.465 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_1 - 2024-09-18T11:44:37.554 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existed_1: (2) No such file or directory 2024-09-18T11:44:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:37.563 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_1 - 2024-09-18T11:44:37.563 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_1 - 2024-09-18T11:44:37.658 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existing_1: (2) No such file or directory 2024-09-18T11:44:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:37.669 INFO:tasks.util.rados:rados -p unique_pool_0 get new_2 - 2024-09-18T11:44:37.670 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_2 - 2024-09-18T11:44:37.763 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/new_2: (2) No such file or directory 2024-09-18T11:44:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:37.774 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_2 - 2024-09-18T11:44:37.774 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_2 - 2024-09-18T11:44:37.858 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existed_2: (2) No such file or directory 2024-09-18T11:44:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:37.869 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_2 - 2024-09-18T11:44:37.869 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_2 - 2024-09-18T11:44:38.156 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:44:38.154+0000 7f0b2ea4a640 0 --2- 172.21.15.71:0/1340158240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e7393b230 0x562e739e7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T11:44:38.168 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existing_2: (2) No such file or directory 2024-09-18T11:44:38.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:38.174 INFO:tasks.util.rados:rados -p unique_pool_0 get new_3 - 2024-09-18T11:44:38.174 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_3 - 2024-09-18T11:44:38.208 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 21 20 26569 26549 9.87527 7.96094 0.0182131 0.0158108 2024-09-18T11:44:38.273 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/new_3: (2) No such file or directory 2024-09-18T11:44:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:38.285 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_3 - 2024-09-18T11:44:38.285 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_3 - 2024-09-18T11:44:38.415 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existed_3: (2) No such file or directory 2024-09-18T11:44:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:38.419 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_3 - 2024-09-18T11:44:38.419 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_3 - 2024-09-18T11:44:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:38.522 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existing_3: (2) No such file or directory 2024-09-18T11:44:38.522 INFO:tasks.util.rados:rados -p unique_pool_0 get new_4 - 2024-09-18T11:44:38.522 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_4 - 2024-09-18T11:44:38.614 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/new_4: (2) No such file or directory 2024-09-18T11:44:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:38.623 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_4 - 2024-09-18T11:44:38.623 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_4 - 2024-09-18T11:44:38.727 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existed_4: (2) No such file or directory 2024-09-18T11:44:38.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:38.732 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_4 - 2024-09-18T11:44:38.732 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_4 - 2024-09-18T11:44:38.829 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existing_4: (2) No such file or directory 2024-09-18T11:44:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:38.838 INFO:tasks.util.rados:rados -p unique_pool_0 get new_5 - 2024-09-18T11:44:38.838 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_5 - 2024-09-18T11:44:38.923 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/new_5: (2) No such file or directory 2024-09-18T11:44:38.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:38.934 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_5 - 2024-09-18T11:44:38.934 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_5 - 2024-09-18T11:44:39.020 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existed_5: (2) No such file or directory 2024-09-18T11:44:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:39.023 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_5 - 2024-09-18T11:44:39.023 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_5 - 2024-09-18T11:44:39.113 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existing_5: (2) No such file or directory 2024-09-18T11:44:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:39.119 INFO:tasks.util.rados:rados -p unique_pool_0 get new_6 - 2024-09-18T11:44:39.119 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_6 - 2024-09-18T11:44:39.208 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 22 20 27564 27544 9.77972 7.77344 0.0178808 0.0159646 2024-09-18T11:44:39.224 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/new_6: (2) No such file or directory 2024-09-18T11:44:39.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:39.229 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_6 - 2024-09-18T11:44:39.229 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_6 - 2024-09-18T11:44:39.325 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existed_6: (2) No such file or directory 2024-09-18T11:44:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:39.335 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_6 - 2024-09-18T11:44:39.335 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_6 - 2024-09-18T11:44:39.441 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existing_6: (2) No such file or directory 2024-09-18T11:44:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:39.447 INFO:tasks.util.rados:rados -p unique_pool_0 get new_7 - 2024-09-18T11:44:39.447 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_7 - 2024-09-18T11:44:39.533 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/new_7: (2) No such file or directory 2024-09-18T11:44:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:39.540 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_7 - 2024-09-18T11:44:39.540 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_7 - 2024-09-18T11:44:39.638 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existed_7: (2) No such file or directory 2024-09-18T11:44:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:39.646 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_7 - 2024-09-18T11:44:39.646 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_7 - 2024-09-18T11:44:39.738 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existing_7: (2) No such file or directory 2024-09-18T11:44:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:39.750 INFO:tasks.util.rados:rados -p unique_pool_0 get new_8 - 2024-09-18T11:44:39.750 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_8 - 2024-09-18T11:44:39.841 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/new_8: (2) No such file or directory 2024-09-18T11:44:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:39.855 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_8 - 2024-09-18T11:44:39.855 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_8 - 2024-09-18T11:44:39.940 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existed_8: (2) No such file or directory 2024-09-18T11:44:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:39.947 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_8 - 2024-09-18T11:44:39.947 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_8 - 2024-09-18T11:44:40.049 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existing_8: (2) No such file or directory 2024-09-18T11:44:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:40.065 INFO:tasks.util.rados:rados -p unique_pool_0 get new_9 - 2024-09-18T11:44:40.065 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_9 - 2024-09-18T11:44:40.152 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/new_9: (2) No such file or directory 2024-09-18T11:44:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:40.160 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_9 - 2024-09-18T11:44:40.161 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_9 - 2024-09-18T11:44:40.208 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 23 20 28541 28521 9.68635 7.63281 0.0251116 0.0161212 2024-09-18T11:44:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existed_9: (2) No such file or directory 2024-09-18T11:44:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:40.263 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_9 - 2024-09-18T11:44:40.263 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_9 - 2024-09-18T11:44:40.361 INFO:teuthology.orchestra.run.smithi071.stderr:error getting unique_pool_0/existing_9: (2) No such file or directory 2024-09-18T11:44:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:40.364 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-18T11:44:40.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-18T11:44:40.366 INFO:tasks.ceph.osd.1:Started 2024-09-18T11:44:40.367 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:0 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:1 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:2 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:abort 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:assert 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:config diff 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:config diff get 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:config help [] 2024-09-18T11:44:40.527 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2024-09-18T11:44:40.528 INFO:teuthology.orchestra.run.smithi071.stderr:admin_socket: invalid command 2024-09-18T11:44:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T11:44:40.530 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-18T11:44:40.755 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-09-18T11:44:40.752+0000 7f98ff44b440 -1 Falling back to public interface 2024-09-18T11:44:41.208 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 24 20 29562 29542 9.61511 7.97656 0.0326789 0.0162424 2024-09-18T11:44:42.208 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 25 20 30604 30584 9.55608 8.14062 0.0145583 0.0163395 2024-09-18T11:44:42.776 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-09-18T11:44:42.773+0000 7f98ff44b440 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:44:43.208 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 26 20 31602 31582 9.4884 7.79688 0.0314792 0.0164577 2024-09-18T11:44:43.532 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-09-18T11:44:43.530+0000 7f98ff44b440 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:44:43.854 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-09-18T11:44:43.853+0000 7f98ff44b440 -1 osd.1 19 log_to_monitors true 2024-09-18T11:44:44.123 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-09-18T11:44:44.122+0000 7f98f53ed640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:44:44.209 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 27 20 32508 32488 9.39911 7.07812 0.0350126 0.0166154 2024-09-18T11:44:45.209 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 28 20 33404 33384 9.3134 7 0.0276315 0.0167088 2024-09-18T11:44:45.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33377 1.0s0 1.9fa49a40 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108177+0000", 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53333091700000002, 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53335215800000002, 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33377 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108177+0000", 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108177+0000", 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108176+0000", 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999981 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108189+0000", 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3131e-05 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108201+0000", 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2812e-05 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108216+0000", 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4856e-05 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108260+0000", 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4060000000000002e-05 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108268+0000", 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5499999999999997e-06 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116753+0000", 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0084851089999999994 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116754+0000", 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6300000000000005e-07 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33379 1.0s0 1.4fc9f6a0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108227+0000", 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53328150100000005, 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53335769099999997, 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.648 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33379 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108227+0000", 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108227+0000", 2024-09-18T11:44:45.649 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108226+0000", 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999995 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108243+0000", 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6358000000000002e-05 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108249+0000", 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5719999999999999e-06 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108253+0000", 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1579999999999998e-06 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108311+0000", 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7989000000000003e-05 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.651 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108316+0000", 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8189999999999999e-06 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116774+0000", 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0084573919999999993 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116774+0000", 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.44e-07 2024-09-18T11:44:45.652 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33385 1.8s0 1.90ee6d78 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108258+0000", 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53325057799999998, 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53336773900000001, 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33385 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108258+0000", 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108258+0000", 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108257+0000", 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999995 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108266+0000", 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9949999999999994e-06 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108272+0000", 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7769999999999999e-06 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108275+0000", 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2600000000000001e-06 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108338+0000", 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2277e-05 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108343+0000", 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2639999999999999e-06 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116679+0000", 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.008336431 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116682+0000", 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8339999999999999e-06 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33387 1.7s0 1.fc2751c7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108280+0000", 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53322802000000002, 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53338492000000004, 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33387 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108280+0000", 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108280+0000", 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108280+0000", 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999995 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108293+0000", 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3161e-05 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108299+0000", 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9660000000000001e-06 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108303+0000", 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.602e-06 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108395+0000", 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2298000000000003e-05 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108401+0000", 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.772e-06 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.118541+0000", 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.010140087000000001 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.118542+0000", 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3999999999999999e-06 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33390 1.8s0 1.e719688 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108315+0000", 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53319367900000003, 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53338954599999999, 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33390 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108315+0000", 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108315+0000", 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108314+0000", 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999995 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108324+0000", 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7929999999999992e-06 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108330+0000", 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1319999999999999e-06 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108333+0000", 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3400000000000002e-06 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108360+0000", 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6953000000000001e-05 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108364+0000", 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1030000000000004e-06 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.666 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116700+0000", 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0083359530000000001 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116701+0000", 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3099999999999998e-07 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33391 1.9s0 1.b1fc34d9 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108337+0000", 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53317090899999997, 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53340548799999998, 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33391 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108337+0000", 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108337+0000", 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108337+0000", 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999995 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108350+0000", 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2731999999999999e-05 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108356+0000", 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9619999999999998e-06 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108359+0000", 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.326e-06 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108854+0000", 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049522200000000002 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108859+0000", 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0699999999999997e-06 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.119491+0000", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.010631826 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.119493+0000", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.694e-06 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33392 1.8s0 1.cfa2e068 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108369+0000", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53313930899999995, 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53341264899999996, 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33392 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108369+0000", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108369+0000", 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108369+0000", 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999995 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108381+0000", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2473000000000001e-05 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108387+0000", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3230000000000003e-06 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108391+0000", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6040000000000001e-06 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108550+0000", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000159426 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108555+0000", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6840000000000004e-06 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116719+0000", 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0081636509999999992 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116719+0000", 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4900000000000001e-07 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33393 1.7s0 1.6bfc6997 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108415+0000", 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53309320299999996, 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.533404345, 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33393 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108415+0000", 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108415+0000", 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108415+0000", 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999995 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108430+0000", 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5087e-05 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108436+0000", 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5050000000000004e-06 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108440+0000", 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0380000000000001e-06 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108614+0000", 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000173418 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108618+0000", 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.929e-06 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.118559+0000", 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0099417859999999993 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.118560+0000", 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4300000000000003e-07 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33395 1.8s0 1.6925c9b8 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108448+0000", 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53306035200000002, 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53340922800000001, 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33395 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108448+0000", 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108448+0000", 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108447+0000", 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999995 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108455+0000", 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9999999999999996e-06 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108461+0000", 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4389999999999999e-06 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108464+0000", 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0520000000000002e-06 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108570+0000", 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010601499999999999 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108574+0000", 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.895e-06 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116735+0000", 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0081609149999999995 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116735+0000", 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8899999999999999e-07 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33397 1.0s0 1.c252c870 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108468+0000", 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53304068400000004, 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53342957300000005, 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.687 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33397 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108468+0000", 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108468+0000", 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108467+0000", 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999995 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108479+0000", 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2258e-05 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108485+0000", 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4920000000000001e-06 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108491+0000", 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4169999999999997e-06 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108589+0000", 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7938999999999999e-05 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108593+0000", 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8430000000000003e-06 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116794+0000", 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0082005169999999992 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.116794+0000", 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1600000000000001e-07 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.691 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33401 1.bs0 1.f089225b (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.108496+0000", 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.53301267399999996, 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53343971099999998, 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.692 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33401 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108496+0000", 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108496+0000", 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108495+0000", 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999995 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108505+0000", 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9240000000000002e-06 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108510+0000", 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2580000000000003e-06 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.694 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108514+0000", 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3280000000000002e-06 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108806+0000", 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00029260799999999997 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.108812+0000", 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4179999999999996e-06 2024-09-18T11:44:45.695 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.118399+0000", 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0095872849999999992 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.118401+0000", 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0420000000000001e-06 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.696 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.5271.0:33404 1.bs0 1.9708c8b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-18T11:44:45.120454+0000", 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.52105374500000001, 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52152007199999995, 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for peered", 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.5271", 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.71:0/326416245", 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 33404 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-18T11:44:45.697 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.120454+0000", 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.120454+0000", 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.120453+0000", 2024-09-18T11:44:45.698 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4294967295.9999981 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.120477+0000", 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4301999999999999e-05 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.120487+0000", 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0108999999999999e-05 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-18T11:44:45.699 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.120493+0000", 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.959e-06 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.121957+0000", 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014637739999999999 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-18T11:44:45.121960+0000", 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1379999999999999e-06 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-18T11:44:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.701 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-18T11:44:45.701 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-18T11:44:45.701 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 12 2024-09-18T11:44:45.701 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:44:45.702 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.1 to be up 2024-09-18T11:44:45.702 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:44:45.968 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:45.968 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:44:45.102723+0000","last_up_change":"2024-09-18T11:44:45.102723+0000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-18T11:43:35.889679+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":26,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":40343},{"type":"v1","addr":"172.21.15.71:6801","nonce":40343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":40343},{"type":"v1","addr":"172.21.15.71:6803","nonce":40343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":40343},{"type":"v1","addr":"172.21.15.71:6807","nonce":40343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":40343},{"type":"v1","addr":"172.21.15.71:6805","nonce":40343}]},"public_addr":"172.21.15.71:6801/40343","cluster_addr":"172.21.15.71:6803/40343","heartbeat_back_addr":"172.21.15.71:6807/40343","heartbeat_front_addr":"172.21.15.71:6805/40343","state":["exists","up"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":46018},{"type":"v1","addr":"172.21.15.71:6822","nonce":46018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":46018},{"type":"v1","addr":"172.21.15.71:6825","nonce":46018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6829","nonce":46018},{"type":"v1","addr":"172.21.15.71:6831","nonce":46018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":46018},{"type":"v1","addr":"172.21.15.71:6828","nonce":46018}]},"public_addr":"172.21.15.71:6822/46018","cluster_addr":"172.21.15.71:6825/46018","heartbeat_back_addr":"172.21.15.71:6831/46018","heartbeat_front_addr":"172.21.15.71:6828/46018","state":["exists","up"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":33275},{"type":"v1","addr":"172.21.15.71:6811","nonce":33275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":33275},{"type":"v1","addr":"172.21.15.71:6818","nonce":33275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6827","nonce":33275},{"type":"v1","addr":"172.21.15.71:6830","nonce":33275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":33275},{"type":"v1","addr":"172.21.15.71:6823","nonce":33275}]},"public_addr":"172.21.15.71:6811/33275","cluster_addr":"172.21.15.71:6818/33275","heartbeat_back_addr":"172.21.15.71:6830/33275","heartbeat_front_addr":"172.21.15.71:6823/33275","state":["exists","up"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":26,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":41313},{"type":"v1","addr":"172.21.15.71:6810","nonce":41313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":41313},{"type":"v1","addr":"172.21.15.71:6813","nonce":41313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6817","nonce":41313},{"type":"v1","addr":"172.21.15.71:6819","nonce":41313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":41313},{"type":"v1","addr":"172.21.15.71:6816","nonce":41313}]},"public_addr":"172.21.15.71:6810/41313","cluster_addr":"172.21.15.71:6813/41313","heartbeat_back_addr":"172.21.15.71:6819/41313","heartbeat_front_addr":"172.21.15.71:6816/41313","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.447050+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.374310+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.355034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.394307+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:44:45.980 INFO:tasks.ec_lost_unfound.ceph_manager:osd.1 is up 2024-09-18T11:44:45.980 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-18T11:44:45.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:46.209 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 29 20 33404 33384 8.99226 0 - 0.0167088 2024-09-18T11:44:46.253 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:46.253 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:46.264 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-18T11:44:46.107519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":258098548,"num_objects":31511,"num_object_clones":0,"num_object_copies":126044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31538,"num_write_kb":252186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31565,"ondisk_log_size":31565,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1616464,"kb_used_data":396516,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375870896,"statfs":{"total":386547056640,"available":384891797504,"internally_reserved":0,"allocated":406032384,"data_stored":405480249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,9,8,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46391296,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":45304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.278527"},"pg_stats":[{"pgid":"1.f","version":"28'2067","reported_seq":2097,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.160301+0000","last_change":"2024-09-18T11:44:17.230219+0000","last_active":"2024-09-18T11:44:44.160301+0000","last_peered":"2024-09-18T11:44:44.160301+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095253+0000","last_became_peered":"2024-09-18T11:44:06.095253+0000","last_unstale":"2024-09-18T11:44:44.160301+0000","last_undegraded":"2024-09-18T11:44:17.229747+0000","last_fullsized":"2024-09-18T11:44:06.087120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:29.394125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16932864,"num_objects":2067,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2067,"num_write_kb":16536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1852","reported_seq":1902,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.990614+0000","last_change":"2024-09-18T11:44:28.260359+0000","last_active":"2024-09-18T11:44:40.990614+0000","last_peered":"2024-09-18T11:44:40.990614+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:06.096040+0000","last_became_peered":"2024-09-18T11:44:06.096040+0000","last_unstale":"2024-09-18T11:44:40.990614+0000","last_undegraded":"2024-09-18T11:44:06.086523+0000","last_fullsized":"2024-09-18T11:44:06.086250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1852,"log_dups_size":0,"ondisk_log_size":1852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15135990,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1850,"num_write_kb":14796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1848}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2058","reported_seq":2128,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129270+0000","last_change":"2024-09-18T11:44:45.130506+0000","last_active":"2024-09-18T11:44:45.129239+0000","last_peered":"2024-09-18T11:44:45.099594+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:06.100615+0000","last_became_peered":"2024-09-18T11:44:06.100615+0000","last_unstale":"2024-09-18T11:44:45.129270+0000","last_undegraded":"2024-09-18T11:44:45.129270+0000","last_fullsized":"2024-09-18T11:44:45.129270+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16833147,"num_objects":2055,"num_object_clones":0,"num_object_copies":8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1870","reported_seq":1920,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.012499+0000","last_change":"2024-09-18T11:44:29.314915+0000","last_active":"2024-09-18T11:44:41.012499+0000","last_peered":"2024-09-18T11:44:41.012499+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:06.099305+0000","last_became_peered":"2024-09-18T11:44:06.099305+0000","last_unstale":"2024-09-18T11:44:41.012499+0000","last_undegraded":"2024-09-18T11:44:06.085610+0000","last_fullsized":"2024-09-18T11:44:06.085286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15283446,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1866}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'2031","reported_seq":2072,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.175494+0000","last_change":"2024-09-18T11:44:29.785182+0000","last_active":"2024-09-18T11:44:44.175494+0000","last_peered":"2024-09-18T11:44:44.175494+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:06.095158+0000","last_became_peered":"2024-09-18T11:44:06.095158+0000","last_unstale":"2024-09-18T11:44:44.175494+0000","last_undegraded":"2024-09-18T11:44:06.087666+0000","last_fullsized":"2024-09-18T11:44:06.086930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2031,"log_dups_size":0,"ondisk_log_size":2031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16610550,"num_objects":2028,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":16229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'2100","reported_seq":2130,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.171487+0000","last_change":"2024-09-18T11:44:17.229813+0000","last_active":"2024-09-18T11:44:44.171487+0000","last_peered":"2024-09-18T11:44:44.171487+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:44.171487+0000","last_undegraded":"2024-09-18T11:44:17.216477+0000","last_fullsized":"2024-09-18T11:44:06.088316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:21:46.952968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17203200,"num_objects":2100,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2100,"num_write_kb":16800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2142","reported_seq":2195,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.122860+0000","last_change":"2024-09-18T11:44:45.126986+0000","last_active":"2024-09-18T11:44:45.122834+0000","last_peered":"2024-09-18T11:44:45.089661+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:06.096306+0000","last_became_peered":"2024-09-18T11:44:06.096306+0000","last_unstale":"2024-09-18T11:44:45.122860+0000","last_undegraded":"2024-09-18T11:44:45.122860+0000","last_fullsized":"2024-09-18T11:44:45.122860+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17495286,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":17100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1819","reported_seq":1869,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.006933+0000","last_change":"2024-09-18T11:44:30.758028+0000","last_active":"2024-09-18T11:44:41.006933+0000","last_peered":"2024-09-18T11:44:41.006933+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:06.095136+0000","last_became_peered":"2024-09-18T11:44:06.095136+0000","last_unstale":"2024-09-18T11:44:41.006933+0000","last_undegraded":"2024-09-18T11:44:06.086038+0000","last_fullsized":"2024-09-18T11:44:06.085713+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1819,"log_dups_size":0,"ondisk_log_size":1819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857462,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1817,"num_write_kb":14524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2104","reported_seq":2147,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.156286+0000","last_change":"2024-09-18T11:44:31.315185+0000","last_active":"2024-09-18T11:44:44.156286+0000","last_peered":"2024-09-18T11:44:44.156286+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:06.095233+0000","last_became_peered":"2024-09-18T11:44:06.095233+0000","last_unstale":"2024-09-18T11:44:44.156286+0000","last_undegraded":"2024-09-18T11:44:06.087588+0000","last_fullsized":"2024-09-18T11:44:06.087287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2104,"log_dups_size":0,"ondisk_log_size":2104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17164780,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":16785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1979","reported_seq":2051,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129947+0000","last_change":"2024-09-18T11:44:45.131087+0000","last_active":"2024-09-18T11:44:45.129917+0000","last_peered":"2024-09-18T11:44:45.087728+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:06.100048+0000","last_became_peered":"2024-09-18T11:44:06.100048+0000","last_unstale":"2024-09-18T11:44:45.129947+0000","last_undegraded":"2024-09-18T11:44:45.129947+0000","last_fullsized":"2024-09-18T11:44:45.129947+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1979,"log_dups_size":0,"ondisk_log_size":1979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16192758,"num_objects":1977,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":15814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'2013","reported_seq":2055,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.172593+0000","last_change":"2024-09-18T11:44:32.316693+0000","last_active":"2024-09-18T11:44:44.172593+0000","last_peered":"2024-09-18T11:44:44.172593+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:06.099667+0000","last_became_peered":"2024-09-18T11:44:06.099667+0000","last_unstale":"2024-09-18T11:44:44.172593+0000","last_undegraded":"2024-09-18T11:44:06.087962+0000","last_fullsized":"2024-09-18T11:44:06.087548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16445297,"num_objects":2008,"num_object_clones":0,"num_object_copies":8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2010,"num_write_kb":16068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'2040","reported_seq":2066,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.148252+0000","last_change":"2024-09-18T11:44:32.775850+0000","last_active":"2024-09-18T11:44:44.148252+0000","last_peered":"2024-09-18T11:44:44.148252+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:44.148252+0000","last_undegraded":"2024-09-18T11:44:06.087360+0000","last_fullsized":"2024-09-18T11:44:06.087125+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16702075,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":16311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1771","reported_seq":1836,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.800566+0000","last_change":"2024-09-18T11:44:33.362854+0000","last_active":"2024-09-18T11:44:40.800566+0000","last_peered":"2024-09-18T11:44:40.800566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:06.095142+0000","last_became_peered":"2024-09-18T11:44:06.095142+0000","last_unstale":"2024-09-18T11:44:40.800566+0000","last_undegraded":"2024-09-18T11:44:06.092217+0000","last_fullsized":"2024-09-18T11:44:06.091922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1771,"log_dups_size":0,"ondisk_log_size":1771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14498427,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":14159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2108","reported_seq":2144,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.155959+0000","last_change":"2024-09-18T11:44:33.907840+0000","last_active":"2024-09-18T11:44:44.155959+0000","last_peered":"2024-09-18T11:44:44.155959+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:06.095532+0000","last_became_peered":"2024-09-18T11:44:06.095532+0000","last_unstale":"2024-09-18T11:44:44.155959+0000","last_undegraded":"2024-09-18T11:44:06.087192+0000","last_fullsized":"2024-09-18T11:44:06.086893+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2108,"log_dups_size":0,"ondisk_log_size":2108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17233142,"num_objects":2104,"num_object_clones":0,"num_object_copies":8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":16844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1837","reported_seq":1906,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.832556+0000","last_change":"2024-09-18T11:44:34.409540+0000","last_active":"2024-09-18T11:44:40.832556+0000","last_peered":"2024-09-18T11:44:40.832556+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:06.100417+0000","last_became_peered":"2024-09-18T11:44:06.100417+0000","last_unstale":"2024-09-18T11:44:40.832556+0000","last_undegraded":"2024-09-18T11:44:06.093073+0000","last_fullsized":"2024-09-18T11:44:06.092766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1837,"log_dups_size":0,"ondisk_log_size":1837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15021302,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1835,"num_write_kb":14677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1774","reported_seq":1824,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.004136+0000","last_change":"2024-09-18T11:44:34.900312+0000","last_active":"2024-09-18T11:44:41.004136+0000","last_peered":"2024-09-18T11:44:41.004136+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:06.096352+0000","last_became_peered":"2024-09-18T11:44:06.096352+0000","last_unstale":"2024-09-18T11:44:41.004136+0000","last_undegraded":"2024-09-18T11:44:06.086622+0000","last_fullsized":"2024-09-18T11:44:06.086322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1774,"log_dups_size":0,"ondisk_log_size":1774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488822,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1772,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1769}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":258098548,"num_objects":31511,"num_object_clones":0,"num_object_copies":126044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31538,"num_write_kb":252186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":406831104,"data_stored":406831104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31565,"ondisk_log_size":31565,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434960,"kb_used_data":128776,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93936880,"statfs":{"total":96636764160,"available":96191365120,"internally_reserved":0,"allocated":131866624,"data_stored":131723035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443452,"kb_used_data":133616,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93928388,"statfs":{"total":96636764160,"available":96182669312,"internally_reserved":0,"allocated":136822784,"data_stored":136675291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":278528,"data_stored":168931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439980,"kb_used_data":133852,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93931860,"statfs":{"total":96636764160,"available":96186224640,"internally_reserved":0,"allocated":137064448,"data_stored":136912992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":133079040,"data_stored":133079040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:46.266 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:46.522 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:46.523 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:46.534 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-18T11:44:46.107519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":258098548,"num_objects":31511,"num_object_clones":0,"num_object_copies":126044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31538,"num_write_kb":252186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31565,"ondisk_log_size":31565,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1616464,"kb_used_data":396516,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375870896,"statfs":{"total":386547056640,"available":384891797504,"internally_reserved":0,"allocated":406032384,"data_stored":405480249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,9,8,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46391296,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":45304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.278527"},"pg_stats":[{"pgid":"1.f","version":"28'2067","reported_seq":2097,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.160301+0000","last_change":"2024-09-18T11:44:17.230219+0000","last_active":"2024-09-18T11:44:44.160301+0000","last_peered":"2024-09-18T11:44:44.160301+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095253+0000","last_became_peered":"2024-09-18T11:44:06.095253+0000","last_unstale":"2024-09-18T11:44:44.160301+0000","last_undegraded":"2024-09-18T11:44:17.229747+0000","last_fullsized":"2024-09-18T11:44:06.087120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:29.394125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16932864,"num_objects":2067,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2067,"num_write_kb":16536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1852","reported_seq":1902,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.990614+0000","last_change":"2024-09-18T11:44:28.260359+0000","last_active":"2024-09-18T11:44:40.990614+0000","last_peered":"2024-09-18T11:44:40.990614+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:06.096040+0000","last_became_peered":"2024-09-18T11:44:06.096040+0000","last_unstale":"2024-09-18T11:44:40.990614+0000","last_undegraded":"2024-09-18T11:44:06.086523+0000","last_fullsized":"2024-09-18T11:44:06.086250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1852,"log_dups_size":0,"ondisk_log_size":1852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15135990,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1850,"num_write_kb":14796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1848}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2058","reported_seq":2128,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129270+0000","last_change":"2024-09-18T11:44:45.130506+0000","last_active":"2024-09-18T11:44:45.129239+0000","last_peered":"2024-09-18T11:44:45.099594+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:06.100615+0000","last_became_peered":"2024-09-18T11:44:06.100615+0000","last_unstale":"2024-09-18T11:44:45.129270+0000","last_undegraded":"2024-09-18T11:44:45.129270+0000","last_fullsized":"2024-09-18T11:44:45.129270+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16833147,"num_objects":2055,"num_object_clones":0,"num_object_copies":8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1870","reported_seq":1920,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.012499+0000","last_change":"2024-09-18T11:44:29.314915+0000","last_active":"2024-09-18T11:44:41.012499+0000","last_peered":"2024-09-18T11:44:41.012499+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:06.099305+0000","last_became_peered":"2024-09-18T11:44:06.099305+0000","last_unstale":"2024-09-18T11:44:41.012499+0000","last_undegraded":"2024-09-18T11:44:06.085610+0000","last_fullsized":"2024-09-18T11:44:06.085286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15283446,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1866}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'2031","reported_seq":2072,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.175494+0000","last_change":"2024-09-18T11:44:29.785182+0000","last_active":"2024-09-18T11:44:44.175494+0000","last_peered":"2024-09-18T11:44:44.175494+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:06.095158+0000","last_became_peered":"2024-09-18T11:44:06.095158+0000","last_unstale":"2024-09-18T11:44:44.175494+0000","last_undegraded":"2024-09-18T11:44:06.087666+0000","last_fullsized":"2024-09-18T11:44:06.086930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2031,"log_dups_size":0,"ondisk_log_size":2031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16610550,"num_objects":2028,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":16229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'2100","reported_seq":2130,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.171487+0000","last_change":"2024-09-18T11:44:17.229813+0000","last_active":"2024-09-18T11:44:44.171487+0000","last_peered":"2024-09-18T11:44:44.171487+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:44.171487+0000","last_undegraded":"2024-09-18T11:44:17.216477+0000","last_fullsized":"2024-09-18T11:44:06.088316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:21:46.952968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17203200,"num_objects":2100,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2100,"num_write_kb":16800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2142","reported_seq":2195,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.122860+0000","last_change":"2024-09-18T11:44:45.126986+0000","last_active":"2024-09-18T11:44:45.122834+0000","last_peered":"2024-09-18T11:44:45.089661+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:06.096306+0000","last_became_peered":"2024-09-18T11:44:06.096306+0000","last_unstale":"2024-09-18T11:44:45.122860+0000","last_undegraded":"2024-09-18T11:44:45.122860+0000","last_fullsized":"2024-09-18T11:44:45.122860+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17495286,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":17100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1819","reported_seq":1869,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.006933+0000","last_change":"2024-09-18T11:44:30.758028+0000","last_active":"2024-09-18T11:44:41.006933+0000","last_peered":"2024-09-18T11:44:41.006933+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:06.095136+0000","last_became_peered":"2024-09-18T11:44:06.095136+0000","last_unstale":"2024-09-18T11:44:41.006933+0000","last_undegraded":"2024-09-18T11:44:06.086038+0000","last_fullsized":"2024-09-18T11:44:06.085713+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1819,"log_dups_size":0,"ondisk_log_size":1819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857462,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1817,"num_write_kb":14524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2104","reported_seq":2147,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.156286+0000","last_change":"2024-09-18T11:44:31.315185+0000","last_active":"2024-09-18T11:44:44.156286+0000","last_peered":"2024-09-18T11:44:44.156286+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:06.095233+0000","last_became_peered":"2024-09-18T11:44:06.095233+0000","last_unstale":"2024-09-18T11:44:44.156286+0000","last_undegraded":"2024-09-18T11:44:06.087588+0000","last_fullsized":"2024-09-18T11:44:06.087287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2104,"log_dups_size":0,"ondisk_log_size":2104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17164780,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":16785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1979","reported_seq":2051,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129947+0000","last_change":"2024-09-18T11:44:45.131087+0000","last_active":"2024-09-18T11:44:45.129917+0000","last_peered":"2024-09-18T11:44:45.087728+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:06.100048+0000","last_became_peered":"2024-09-18T11:44:06.100048+0000","last_unstale":"2024-09-18T11:44:45.129947+0000","last_undegraded":"2024-09-18T11:44:45.129947+0000","last_fullsized":"2024-09-18T11:44:45.129947+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1979,"log_dups_size":0,"ondisk_log_size":1979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16192758,"num_objects":1977,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":15814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'2013","reported_seq":2055,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.172593+0000","last_change":"2024-09-18T11:44:32.316693+0000","last_active":"2024-09-18T11:44:44.172593+0000","last_peered":"2024-09-18T11:44:44.172593+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:06.099667+0000","last_became_peered":"2024-09-18T11:44:06.099667+0000","last_unstale":"2024-09-18T11:44:44.172593+0000","last_undegraded":"2024-09-18T11:44:06.087962+0000","last_fullsized":"2024-09-18T11:44:06.087548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16445297,"num_objects":2008,"num_object_clones":0,"num_object_copies":8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2010,"num_write_kb":16068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'2040","reported_seq":2066,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.148252+0000","last_change":"2024-09-18T11:44:32.775850+0000","last_active":"2024-09-18T11:44:44.148252+0000","last_peered":"2024-09-18T11:44:44.148252+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:44.148252+0000","last_undegraded":"2024-09-18T11:44:06.087360+0000","last_fullsized":"2024-09-18T11:44:06.087125+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16702075,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":16311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1771","reported_seq":1836,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.800566+0000","last_change":"2024-09-18T11:44:33.362854+0000","last_active":"2024-09-18T11:44:40.800566+0000","last_peered":"2024-09-18T11:44:40.800566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:06.095142+0000","last_became_peered":"2024-09-18T11:44:06.095142+0000","last_unstale":"2024-09-18T11:44:40.800566+0000","last_undegraded":"2024-09-18T11:44:06.092217+0000","last_fullsized":"2024-09-18T11:44:06.091922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1771,"log_dups_size":0,"ondisk_log_size":1771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14498427,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":14159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2108","reported_seq":2144,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.155959+0000","last_change":"2024-09-18T11:44:33.907840+0000","last_active":"2024-09-18T11:44:44.155959+0000","last_peered":"2024-09-18T11:44:44.155959+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:06.095532+0000","last_became_peered":"2024-09-18T11:44:06.095532+0000","last_unstale":"2024-09-18T11:44:44.155959+0000","last_undegraded":"2024-09-18T11:44:06.087192+0000","last_fullsized":"2024-09-18T11:44:06.086893+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2108,"log_dups_size":0,"ondisk_log_size":2108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17233142,"num_objects":2104,"num_object_clones":0,"num_object_copies":8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":16844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1837","reported_seq":1906,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.832556+0000","last_change":"2024-09-18T11:44:34.409540+0000","last_active":"2024-09-18T11:44:40.832556+0000","last_peered":"2024-09-18T11:44:40.832556+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:06.100417+0000","last_became_peered":"2024-09-18T11:44:06.100417+0000","last_unstale":"2024-09-18T11:44:40.832556+0000","last_undegraded":"2024-09-18T11:44:06.093073+0000","last_fullsized":"2024-09-18T11:44:06.092766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1837,"log_dups_size":0,"ondisk_log_size":1837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15021302,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1835,"num_write_kb":14677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1774","reported_seq":1824,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.004136+0000","last_change":"2024-09-18T11:44:34.900312+0000","last_active":"2024-09-18T11:44:41.004136+0000","last_peered":"2024-09-18T11:44:41.004136+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:06.096352+0000","last_became_peered":"2024-09-18T11:44:06.096352+0000","last_unstale":"2024-09-18T11:44:41.004136+0000","last_undegraded":"2024-09-18T11:44:06.086622+0000","last_fullsized":"2024-09-18T11:44:06.086322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1774,"log_dups_size":0,"ondisk_log_size":1774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488822,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1772,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1769}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":258098548,"num_objects":31511,"num_object_clones":0,"num_object_copies":126044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31538,"num_write_kb":252186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":406831104,"data_stored":406831104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31565,"ondisk_log_size":31565,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434960,"kb_used_data":128776,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93936880,"statfs":{"total":96636764160,"available":96191365120,"internally_reserved":0,"allocated":131866624,"data_stored":131723035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443452,"kb_used_data":133616,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93928388,"statfs":{"total":96636764160,"available":96182669312,"internally_reserved":0,"allocated":136822784,"data_stored":136675291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":278528,"data_stored":168931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439980,"kb_used_data":133852,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93931860,"statfs":{"total":96636764160,"available":96186224640,"internally_reserved":0,"allocated":137064448,"data_stored":136912992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":133079040,"data_stored":133079040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:46.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:46.789 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:46.790 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:46.801 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-18T11:44:46.107519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":258098548,"num_objects":31511,"num_object_clones":0,"num_object_copies":126044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31538,"num_write_kb":252186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31565,"ondisk_log_size":31565,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1616464,"kb_used_data":396516,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375870896,"statfs":{"total":386547056640,"available":384891797504,"internally_reserved":0,"allocated":406032384,"data_stored":405480249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,9,8,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46391296,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":45304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.278527"},"pg_stats":[{"pgid":"1.f","version":"28'2067","reported_seq":2097,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.160301+0000","last_change":"2024-09-18T11:44:17.230219+0000","last_active":"2024-09-18T11:44:44.160301+0000","last_peered":"2024-09-18T11:44:44.160301+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095253+0000","last_became_peered":"2024-09-18T11:44:06.095253+0000","last_unstale":"2024-09-18T11:44:44.160301+0000","last_undegraded":"2024-09-18T11:44:17.229747+0000","last_fullsized":"2024-09-18T11:44:06.087120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:29.394125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16932864,"num_objects":2067,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2067,"num_write_kb":16536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1852","reported_seq":1902,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.990614+0000","last_change":"2024-09-18T11:44:28.260359+0000","last_active":"2024-09-18T11:44:40.990614+0000","last_peered":"2024-09-18T11:44:40.990614+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:06.096040+0000","last_became_peered":"2024-09-18T11:44:06.096040+0000","last_unstale":"2024-09-18T11:44:40.990614+0000","last_undegraded":"2024-09-18T11:44:06.086523+0000","last_fullsized":"2024-09-18T11:44:06.086250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1852,"log_dups_size":0,"ondisk_log_size":1852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15135990,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1850,"num_write_kb":14796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1848}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2058","reported_seq":2128,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129270+0000","last_change":"2024-09-18T11:44:45.130506+0000","last_active":"2024-09-18T11:44:45.129239+0000","last_peered":"2024-09-18T11:44:45.099594+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:06.100615+0000","last_became_peered":"2024-09-18T11:44:06.100615+0000","last_unstale":"2024-09-18T11:44:45.129270+0000","last_undegraded":"2024-09-18T11:44:45.129270+0000","last_fullsized":"2024-09-18T11:44:45.129270+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16833147,"num_objects":2055,"num_object_clones":0,"num_object_copies":8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1870","reported_seq":1920,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.012499+0000","last_change":"2024-09-18T11:44:29.314915+0000","last_active":"2024-09-18T11:44:41.012499+0000","last_peered":"2024-09-18T11:44:41.012499+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:06.099305+0000","last_became_peered":"2024-09-18T11:44:06.099305+0000","last_unstale":"2024-09-18T11:44:41.012499+0000","last_undegraded":"2024-09-18T11:44:06.085610+0000","last_fullsized":"2024-09-18T11:44:06.085286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15283446,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1866}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'2031","reported_seq":2072,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.175494+0000","last_change":"2024-09-18T11:44:29.785182+0000","last_active":"2024-09-18T11:44:44.175494+0000","last_peered":"2024-09-18T11:44:44.175494+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:06.095158+0000","last_became_peered":"2024-09-18T11:44:06.095158+0000","last_unstale":"2024-09-18T11:44:44.175494+0000","last_undegraded":"2024-09-18T11:44:06.087666+0000","last_fullsized":"2024-09-18T11:44:06.086930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2031,"log_dups_size":0,"ondisk_log_size":2031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16610550,"num_objects":2028,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":16229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'2100","reported_seq":2130,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.171487+0000","last_change":"2024-09-18T11:44:17.229813+0000","last_active":"2024-09-18T11:44:44.171487+0000","last_peered":"2024-09-18T11:44:44.171487+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:44.171487+0000","last_undegraded":"2024-09-18T11:44:17.216477+0000","last_fullsized":"2024-09-18T11:44:06.088316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:21:46.952968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17203200,"num_objects":2100,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2100,"num_write_kb":16800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2142","reported_seq":2195,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.122860+0000","last_change":"2024-09-18T11:44:45.126986+0000","last_active":"2024-09-18T11:44:45.122834+0000","last_peered":"2024-09-18T11:44:45.089661+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:06.096306+0000","last_became_peered":"2024-09-18T11:44:06.096306+0000","last_unstale":"2024-09-18T11:44:45.122860+0000","last_undegraded":"2024-09-18T11:44:45.122860+0000","last_fullsized":"2024-09-18T11:44:45.122860+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17495286,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":17100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1819","reported_seq":1869,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.006933+0000","last_change":"2024-09-18T11:44:30.758028+0000","last_active":"2024-09-18T11:44:41.006933+0000","last_peered":"2024-09-18T11:44:41.006933+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:06.095136+0000","last_became_peered":"2024-09-18T11:44:06.095136+0000","last_unstale":"2024-09-18T11:44:41.006933+0000","last_undegraded":"2024-09-18T11:44:06.086038+0000","last_fullsized":"2024-09-18T11:44:06.085713+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1819,"log_dups_size":0,"ondisk_log_size":1819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857462,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1817,"num_write_kb":14524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2104","reported_seq":2147,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.156286+0000","last_change":"2024-09-18T11:44:31.315185+0000","last_active":"2024-09-18T11:44:44.156286+0000","last_peered":"2024-09-18T11:44:44.156286+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:06.095233+0000","last_became_peered":"2024-09-18T11:44:06.095233+0000","last_unstale":"2024-09-18T11:44:44.156286+0000","last_undegraded":"2024-09-18T11:44:06.087588+0000","last_fullsized":"2024-09-18T11:44:06.087287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2104,"log_dups_size":0,"ondisk_log_size":2104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17164780,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":16785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1979","reported_seq":2051,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129947+0000","last_change":"2024-09-18T11:44:45.131087+0000","last_active":"2024-09-18T11:44:45.129917+0000","last_peered":"2024-09-18T11:44:45.087728+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:06.100048+0000","last_became_peered":"2024-09-18T11:44:06.100048+0000","last_unstale":"2024-09-18T11:44:45.129947+0000","last_undegraded":"2024-09-18T11:44:45.129947+0000","last_fullsized":"2024-09-18T11:44:45.129947+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1979,"log_dups_size":0,"ondisk_log_size":1979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16192758,"num_objects":1977,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":15814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'2013","reported_seq":2055,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.172593+0000","last_change":"2024-09-18T11:44:32.316693+0000","last_active":"2024-09-18T11:44:44.172593+0000","last_peered":"2024-09-18T11:44:44.172593+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:06.099667+0000","last_became_peered":"2024-09-18T11:44:06.099667+0000","last_unstale":"2024-09-18T11:44:44.172593+0000","last_undegraded":"2024-09-18T11:44:06.087962+0000","last_fullsized":"2024-09-18T11:44:06.087548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16445297,"num_objects":2008,"num_object_clones":0,"num_object_copies":8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2010,"num_write_kb":16068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'2040","reported_seq":2066,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.148252+0000","last_change":"2024-09-18T11:44:32.775850+0000","last_active":"2024-09-18T11:44:44.148252+0000","last_peered":"2024-09-18T11:44:44.148252+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:44.148252+0000","last_undegraded":"2024-09-18T11:44:06.087360+0000","last_fullsized":"2024-09-18T11:44:06.087125+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16702075,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":16311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1771","reported_seq":1836,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.800566+0000","last_change":"2024-09-18T11:44:33.362854+0000","last_active":"2024-09-18T11:44:40.800566+0000","last_peered":"2024-09-18T11:44:40.800566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:06.095142+0000","last_became_peered":"2024-09-18T11:44:06.095142+0000","last_unstale":"2024-09-18T11:44:40.800566+0000","last_undegraded":"2024-09-18T11:44:06.092217+0000","last_fullsized":"2024-09-18T11:44:06.091922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1771,"log_dups_size":0,"ondisk_log_size":1771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14498427,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":14159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2108","reported_seq":2144,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.155959+0000","last_change":"2024-09-18T11:44:33.907840+0000","last_active":"2024-09-18T11:44:44.155959+0000","last_peered":"2024-09-18T11:44:44.155959+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:06.095532+0000","last_became_peered":"2024-09-18T11:44:06.095532+0000","last_unstale":"2024-09-18T11:44:44.155959+0000","last_undegraded":"2024-09-18T11:44:06.087192+0000","last_fullsized":"2024-09-18T11:44:06.086893+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2108,"log_dups_size":0,"ondisk_log_size":2108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17233142,"num_objects":2104,"num_object_clones":0,"num_object_copies":8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":16844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1837","reported_seq":1906,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.832556+0000","last_change":"2024-09-18T11:44:34.409540+0000","last_active":"2024-09-18T11:44:40.832556+0000","last_peered":"2024-09-18T11:44:40.832556+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:06.100417+0000","last_became_peered":"2024-09-18T11:44:06.100417+0000","last_unstale":"2024-09-18T11:44:40.832556+0000","last_undegraded":"2024-09-18T11:44:06.093073+0000","last_fullsized":"2024-09-18T11:44:06.092766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1837,"log_dups_size":0,"ondisk_log_size":1837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15021302,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1835,"num_write_kb":14677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1774","reported_seq":1824,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.004136+0000","last_change":"2024-09-18T11:44:34.900312+0000","last_active":"2024-09-18T11:44:41.004136+0000","last_peered":"2024-09-18T11:44:41.004136+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:06.096352+0000","last_became_peered":"2024-09-18T11:44:06.096352+0000","last_unstale":"2024-09-18T11:44:41.004136+0000","last_undegraded":"2024-09-18T11:44:06.086622+0000","last_fullsized":"2024-09-18T11:44:06.086322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1774,"log_dups_size":0,"ondisk_log_size":1774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488822,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1772,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1769}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":258098548,"num_objects":31511,"num_object_clones":0,"num_object_copies":126044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31538,"num_write_kb":252186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":406831104,"data_stored":406831104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31565,"ondisk_log_size":31565,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434960,"kb_used_data":128776,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93936880,"statfs":{"total":96636764160,"available":96191365120,"internally_reserved":0,"allocated":131866624,"data_stored":131723035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443452,"kb_used_data":133616,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93928388,"statfs":{"total":96636764160,"available":96182669312,"internally_reserved":0,"allocated":136822784,"data_stored":136675291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":278528,"data_stored":168931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439980,"kb_used_data":133852,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93931860,"statfs":{"total":96636764160,"available":96186224640,"internally_reserved":0,"allocated":137064448,"data_stored":136912992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":133079040,"data_stored":133079040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:46.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:44:46.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "28'2067", 'reported_seq': 2097, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:44.160301+0000', 'last_change': '2024-09-18T11:44:17.230219+0000', 'last_active': '2024-09-18T11:44:44.160301+0000', 'last_peered': '2024-09-18T11:44:44.160301+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:06.095253+0000', 'last_became_peered': '2024-09-18T11:44:06.095253+0000', 'last_unstale': '2024-09-18T11:44:44.160301+0000', 'last_undegraded': '2024-09-18T11:44:17.229747+0000', 'last_fullsized': '2024-09-18T11:44:06.087120+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2067, 'log_dups_size': 0, 'ondisk_log_size': 2067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:24:29.394125+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16932864, 'num_objects': 2067, 'num_object_clones': 0, 'num_object_copies': 8268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2067, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2067, 'num_write_kb': 16536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2147483647, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2067}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:46.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:44:46.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "27'1852", 'reported_seq': 1902, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:40.990614+0000', 'last_change': '2024-09-18T11:44:28.260359+0000', 'last_active': '2024-09-18T11:44:40.990614+0000', 'last_peered': '2024-09-18T11:44:40.990614+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:06.096040+0000', 'last_became_peered': '2024-09-18T11:44:06.096040+0000', 'last_unstale': '2024-09-18T11:44:40.990614+0000', 'last_undegraded': '2024-09-18T11:44:06.086523+0000', 'last_fullsized': '2024-09-18T11:44:06.086250+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1852, 'log_dups_size': 0, 'ondisk_log_size': 1852, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15135990, 'num_objects': 1848, 'num_object_clones': 0, 'num_object_copies': 7392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1848, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1850, 'num_write_kb': 14796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1848}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:46.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:44:46.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "28'2058", 'reported_seq': 2128, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-18T11:44:45.129270+0000', 'last_change': '2024-09-18T11:44:45.130506+0000', 'last_active': '2024-09-18T11:44:45.129239+0000', 'last_peered': '2024-09-18T11:44:45.099594+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:06.100615+0000', 'last_became_peered': '2024-09-18T11:44:06.100615+0000', 'last_unstale': '2024-09-18T11:44:45.129270+0000', 'last_undegraded': '2024-09-18T11:44:45.129270+0000', 'last_fullsized': '2024-09-18T11:44:45.129270+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2058, 'log_dups_size': 0, 'ondisk_log_size': 2058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16833147, 'num_objects': 2055, 'num_object_clones': 0, 'num_object_copies': 8220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2057, 'num_write_kb': 16446, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 1, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:46.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:44:46.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "27'1870", 'reported_seq': 1920, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:41.012499+0000', 'last_change': '2024-09-18T11:44:29.314915+0000', 'last_active': '2024-09-18T11:44:41.012499+0000', 'last_peered': '2024-09-18T11:44:41.012499+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:06.099305+0000', 'last_became_peered': '2024-09-18T11:44:06.099305+0000', 'last_unstale': '2024-09-18T11:44:41.012499+0000', 'last_undegraded': '2024-09-18T11:44:06.085610+0000', 'last_fullsized': '2024-09-18T11:44:06.085286+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1870, 'log_dups_size': 0, 'ondisk_log_size': 1870, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15283446, 'num_objects': 1866, 'num_object_clones': 0, 'num_object_copies': 7464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1866, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1868, 'num_write_kb': 14933, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1866}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:46.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:44:46.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "28'2031", 'reported_seq': 2072, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:44.175494+0000', 'last_change': '2024-09-18T11:44:29.785182+0000', 'last_active': '2024-09-18T11:44:44.175494+0000', 'last_peered': '2024-09-18T11:44:44.175494+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:06.095158+0000', 'last_became_peered': '2024-09-18T11:44:06.095158+0000', 'last_unstale': '2024-09-18T11:44:44.175494+0000', 'last_undegraded': '2024-09-18T11:44:06.087666+0000', 'last_fullsized': '2024-09-18T11:44:06.086930+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2031, 'log_dups_size': 0, 'ondisk_log_size': 2031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16610550, 'num_objects': 2028, 'num_object_clones': 0, 'num_object_copies': 8112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2029, 'num_write_kb': 16229, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2147483647, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2028}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:46.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:44:46.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "28'2100", 'reported_seq': 2130, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:44.171487+0000', 'last_change': '2024-09-18T11:44:17.229813+0000', 'last_active': '2024-09-18T11:44:44.171487+0000', 'last_peered': '2024-09-18T11:44:44.171487+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:06.095774+0000', 'last_became_peered': '2024-09-18T11:44:06.095774+0000', 'last_unstale': '2024-09-18T11:44:44.171487+0000', 'last_undegraded': '2024-09-18T11:44:17.216477+0000', 'last_fullsized': '2024-09-18T11:44:06.088316+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2100, 'log_dups_size': 0, 'ondisk_log_size': 2100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T20:21:46.952968+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17203200, 'num_objects': 2100, 'num_object_clones': 0, 'num_object_copies': 8400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2100, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2100, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2100, 'num_write_kb': 16800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 2147483647], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2100}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:46.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:44:46.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "28'2142", 'reported_seq': 2195, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-18T11:44:45.122860+0000', 'last_change': '2024-09-18T11:44:45.126986+0000', 'last_active': '2024-09-18T11:44:45.122834+0000', 'last_peered': '2024-09-18T11:44:45.089661+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:06.096306+0000', 'last_became_peered': '2024-09-18T11:44:06.096306+0000', 'last_unstale': '2024-09-18T11:44:45.122860+0000', 'last_undegraded': '2024-09-18T11:44:45.122860+0000', 'last_fullsized': '2024-09-18T11:44:45.122860+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2142, 'log_dups_size': 0, 'ondisk_log_size': 2142, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17495286, 'num_objects': 2136, 'num_object_clones': 0, 'num_object_copies': 8544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2140, 'num_write_kb': 17100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:46.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:44:46.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "27'1819", 'reported_seq': 1869, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:41.006933+0000', 'last_change': '2024-09-18T11:44:30.758028+0000', 'last_active': '2024-09-18T11:44:41.006933+0000', 'last_peered': '2024-09-18T11:44:41.006933+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:06.095136+0000', 'last_became_peered': '2024-09-18T11:44:06.095136+0000', 'last_unstale': '2024-09-18T11:44:41.006933+0000', 'last_undegraded': '2024-09-18T11:44:06.086038+0000', 'last_fullsized': '2024-09-18T11:44:06.085713+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1819, 'log_dups_size': 0, 'ondisk_log_size': 1819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14857462, 'num_objects': 1814, 'num_object_clones': 0, 'num_object_copies': 7256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1814, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1814, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1817, 'num_write_kb': 14524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1814}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:46.804 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:44:46.804 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "28'2104", 'reported_seq': 2147, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:44.156286+0000', 'last_change': '2024-09-18T11:44:31.315185+0000', 'last_active': '2024-09-18T11:44:44.156286+0000', 'last_peered': '2024-09-18T11:44:44.156286+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:06.095233+0000', 'last_became_peered': '2024-09-18T11:44:06.095233+0000', 'last_unstale': '2024-09-18T11:44:44.156286+0000', 'last_undegraded': '2024-09-18T11:44:06.087588+0000', 'last_fullsized': '2024-09-18T11:44:06.087287+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2104, 'log_dups_size': 0, 'ondisk_log_size': 2104, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17164780, 'num_objects': 2096, 'num_object_clones': 0, 'num_object_copies': 8384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2096, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2101, 'num_write_kb': 16785, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 2147483647], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2096}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:46.804 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:44:46.804 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "28'1979", 'reported_seq': 2051, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-18T11:44:45.129947+0000', 'last_change': '2024-09-18T11:44:45.131087+0000', 'last_active': '2024-09-18T11:44:45.129917+0000', 'last_peered': '2024-09-18T11:44:45.087728+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:06.100048+0000', 'last_became_peered': '2024-09-18T11:44:06.100048+0000', 'last_unstale': '2024-09-18T11:44:45.129947+0000', 'last_undegraded': '2024-09-18T11:44:45.129947+0000', 'last_fullsized': '2024-09-18T11:44:45.129947+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1979, 'log_dups_size': 0, 'ondisk_log_size': 1979, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16192758, 'num_objects': 1977, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1977, 'num_write_kb': 15814, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:46.804 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:44:46.804 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "28'2013", 'reported_seq': 2055, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:44.172593+0000', 'last_change': '2024-09-18T11:44:32.316693+0000', 'last_active': '2024-09-18T11:44:44.172593+0000', 'last_peered': '2024-09-18T11:44:44.172593+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:06.099667+0000', 'last_became_peered': '2024-09-18T11:44:06.099667+0000', 'last_unstale': '2024-09-18T11:44:44.172593+0000', 'last_undegraded': '2024-09-18T11:44:06.087962+0000', 'last_fullsized': '2024-09-18T11:44:06.087548+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2013, 'log_dups_size': 0, 'ondisk_log_size': 2013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16445297, 'num_objects': 2008, 'num_object_clones': 0, 'num_object_copies': 8032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2008, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2010, 'num_write_kb': 16068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2147483647, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2008}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:46.804 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:44:46.804 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "28'2040", 'reported_seq': 2066, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:44.148252+0000', 'last_change': '2024-09-18T11:44:32.775850+0000', 'last_active': '2024-09-18T11:44:44.148252+0000', 'last_peered': '2024-09-18T11:44:44.148252+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:06.095774+0000', 'last_became_peered': '2024-09-18T11:44:06.095774+0000', 'last_unstale': '2024-09-18T11:44:44.148252+0000', 'last_undegraded': '2024-09-18T11:44:06.087360+0000', 'last_fullsized': '2024-09-18T11:44:06.087125+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2040, 'log_dups_size': 0, 'ondisk_log_size': 2040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16702075, 'num_objects': 2039, 'num_object_clones': 0, 'num_object_copies': 8156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2039, 'num_write_kb': 16311, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2039}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:46.805 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:44:46.805 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "27'1771", 'reported_seq': 1836, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:40.800566+0000', 'last_change': '2024-09-18T11:44:33.362854+0000', 'last_active': '2024-09-18T11:44:40.800566+0000', 'last_peered': '2024-09-18T11:44:40.800566+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:06.095142+0000', 'last_became_peered': '2024-09-18T11:44:06.095142+0000', 'last_unstale': '2024-09-18T11:44:40.800566+0000', 'last_undegraded': '2024-09-18T11:44:06.092217+0000', 'last_fullsized': '2024-09-18T11:44:06.091922+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1771, 'log_dups_size': 0, 'ondisk_log_size': 1771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14498427, 'num_objects': 1770, 'num_object_clones': 0, 'num_object_copies': 7080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1770, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1770, 'num_write_kb': 14159, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1770}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:46.805 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:44:46.805 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "28'2108", 'reported_seq': 2144, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:44.155959+0000', 'last_change': '2024-09-18T11:44:33.907840+0000', 'last_active': '2024-09-18T11:44:44.155959+0000', 'last_peered': '2024-09-18T11:44:44.155959+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:06.095532+0000', 'last_became_peered': '2024-09-18T11:44:06.095532+0000', 'last_unstale': '2024-09-18T11:44:44.155959+0000', 'last_undegraded': '2024-09-18T11:44:06.087192+0000', 'last_fullsized': '2024-09-18T11:44:06.086893+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2108, 'log_dups_size': 0, 'ondisk_log_size': 2108, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17233142, 'num_objects': 2104, 'num_object_clones': 0, 'num_object_copies': 8416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2106, 'num_write_kb': 16844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2104}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:46.805 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:44:46.805 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "27'1837", 'reported_seq': 1906, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:40.832556+0000', 'last_change': '2024-09-18T11:44:34.409540+0000', 'last_active': '2024-09-18T11:44:40.832556+0000', 'last_peered': '2024-09-18T11:44:40.832556+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:06.100417+0000', 'last_became_peered': '2024-09-18T11:44:06.100417+0000', 'last_unstale': '2024-09-18T11:44:40.832556+0000', 'last_undegraded': '2024-09-18T11:44:06.093073+0000', 'last_fullsized': '2024-09-18T11:44:06.092766+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1837, 'log_dups_size': 0, 'ondisk_log_size': 1837, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15021302, 'num_objects': 1834, 'num_object_clones': 0, 'num_object_copies': 7336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1834, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1835, 'num_write_kb': 14677, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1834}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:46.805 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:44:46.805 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "27'1774", 'reported_seq': 1824, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:41.004136+0000', 'last_change': '2024-09-18T11:44:34.900312+0000', 'last_active': '2024-09-18T11:44:41.004136+0000', 'last_peered': '2024-09-18T11:44:41.004136+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:06.096352+0000', 'last_became_peered': '2024-09-18T11:44:06.096352+0000', 'last_unstale': '2024-09-18T11:44:41.004136+0000', 'last_undegraded': '2024-09-18T11:44:06.086622+0000', 'last_fullsized': '2024-09-18T11:44:06.086322+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1774, 'log_dups_size': 0, 'ondisk_log_size': 1774, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14488822, 'num_objects': 1769, 'num_object_clones': 0, 'num_object_copies': 7076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1769, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1772, 'num_write_kb': 14164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1769}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:46.806 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:44:47.127 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:47.127 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30741/122964 objects degraded (25.000%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":85,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":16}],"num_pgs":16,"num_pools":1,"num_objects":30741,"data_bytes":251790708,"bytes_used":1604755456,"bytes_avail":384942301184,"bytes_total":386547056640,"degraded_objects":30741,"degraded_total":122964,"degraded_ratio":0.25,"write_bytes_sec":6373740,"read_op_per_sec":0,"write_op_per_sec":778},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-18T11:44:31.826669+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:44:47.160 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T11:44:47.160 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:47.209 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 30 20 33404 33384 8.69253 0 - 0.0167088 2024-09-18T11:44:47.660 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:47.660 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:47.687 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-18T11:44:47.115721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":258098548,"num_objects":31511,"num_object_clones":0,"num_object_copies":126044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31538,"num_write_kb":252186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31565,"ondisk_log_size":31565,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1616464,"kb_used_data":396516,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375870896,"statfs":{"total":386547056640,"available":384891797504,"internally_reserved":0,"allocated":406032384,"data_stored":405480249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,9,8,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24338432,"num_objects":2971,"num_object_clones":0,"num_object_copies":11884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.286296"},"pg_stats":[{"pgid":"1.f","version":"28'2067","reported_seq":2097,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.160301+0000","last_change":"2024-09-18T11:44:17.230219+0000","last_active":"2024-09-18T11:44:44.160301+0000","last_peered":"2024-09-18T11:44:44.160301+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095253+0000","last_became_peered":"2024-09-18T11:44:06.095253+0000","last_unstale":"2024-09-18T11:44:44.160301+0000","last_undegraded":"2024-09-18T11:44:17.229747+0000","last_fullsized":"2024-09-18T11:44:06.087120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:29.394125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16932864,"num_objects":2067,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2067,"num_write_kb":16536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1852","reported_seq":1902,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.990614+0000","last_change":"2024-09-18T11:44:28.260359+0000","last_active":"2024-09-18T11:44:40.990614+0000","last_peered":"2024-09-18T11:44:40.990614+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:06.096040+0000","last_became_peered":"2024-09-18T11:44:06.096040+0000","last_unstale":"2024-09-18T11:44:40.990614+0000","last_undegraded":"2024-09-18T11:44:06.086523+0000","last_fullsized":"2024-09-18T11:44:06.086250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1852,"log_dups_size":0,"ondisk_log_size":1852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15135990,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1850,"num_write_kb":14796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1848}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2058","reported_seq":2128,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129270+0000","last_change":"2024-09-18T11:44:45.130506+0000","last_active":"2024-09-18T11:44:45.129239+0000","last_peered":"2024-09-18T11:44:45.099594+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:06.100615+0000","last_became_peered":"2024-09-18T11:44:06.100615+0000","last_unstale":"2024-09-18T11:44:45.129270+0000","last_undegraded":"2024-09-18T11:44:45.129270+0000","last_fullsized":"2024-09-18T11:44:45.129270+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16833147,"num_objects":2055,"num_object_clones":0,"num_object_copies":8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1870","reported_seq":1920,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.012499+0000","last_change":"2024-09-18T11:44:29.314915+0000","last_active":"2024-09-18T11:44:41.012499+0000","last_peered":"2024-09-18T11:44:41.012499+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:06.099305+0000","last_became_peered":"2024-09-18T11:44:06.099305+0000","last_unstale":"2024-09-18T11:44:41.012499+0000","last_undegraded":"2024-09-18T11:44:06.085610+0000","last_fullsized":"2024-09-18T11:44:06.085286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15283446,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1866}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'2031","reported_seq":2072,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.175494+0000","last_change":"2024-09-18T11:44:29.785182+0000","last_active":"2024-09-18T11:44:44.175494+0000","last_peered":"2024-09-18T11:44:44.175494+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:06.095158+0000","last_became_peered":"2024-09-18T11:44:06.095158+0000","last_unstale":"2024-09-18T11:44:44.175494+0000","last_undegraded":"2024-09-18T11:44:06.087666+0000","last_fullsized":"2024-09-18T11:44:06.086930+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2031,"log_dups_size":0,"ondisk_log_size":2031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16610550,"num_objects":2028,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":16229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'2100","reported_seq":2130,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.171487+0000","last_change":"2024-09-18T11:44:17.229813+0000","last_active":"2024-09-18T11:44:44.171487+0000","last_peered":"2024-09-18T11:44:44.171487+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:44.171487+0000","last_undegraded":"2024-09-18T11:44:17.216477+0000","last_fullsized":"2024-09-18T11:44:06.088316+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:21:46.952968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17203200,"num_objects":2100,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2100,"num_write_kb":16800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2142","reported_seq":2195,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.122860+0000","last_change":"2024-09-18T11:44:45.126986+0000","last_active":"2024-09-18T11:44:45.122834+0000","last_peered":"2024-09-18T11:44:45.089661+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:06.096306+0000","last_became_peered":"2024-09-18T11:44:06.096306+0000","last_unstale":"2024-09-18T11:44:45.122860+0000","last_undegraded":"2024-09-18T11:44:45.122860+0000","last_fullsized":"2024-09-18T11:44:45.122860+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17495286,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":17100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1819","reported_seq":1869,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.006933+0000","last_change":"2024-09-18T11:44:30.758028+0000","last_active":"2024-09-18T11:44:41.006933+0000","last_peered":"2024-09-18T11:44:41.006933+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:06.095136+0000","last_became_peered":"2024-09-18T11:44:06.095136+0000","last_unstale":"2024-09-18T11:44:41.006933+0000","last_undegraded":"2024-09-18T11:44:06.086038+0000","last_fullsized":"2024-09-18T11:44:06.085713+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1819,"log_dups_size":0,"ondisk_log_size":1819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857462,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1817,"num_write_kb":14524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2104","reported_seq":2147,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.156286+0000","last_change":"2024-09-18T11:44:31.315185+0000","last_active":"2024-09-18T11:44:44.156286+0000","last_peered":"2024-09-18T11:44:44.156286+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:06.095233+0000","last_became_peered":"2024-09-18T11:44:06.095233+0000","last_unstale":"2024-09-18T11:44:44.156286+0000","last_undegraded":"2024-09-18T11:44:06.087588+0000","last_fullsized":"2024-09-18T11:44:06.087287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2104,"log_dups_size":0,"ondisk_log_size":2104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17164780,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":16785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1979","reported_seq":2051,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129947+0000","last_change":"2024-09-18T11:44:45.131087+0000","last_active":"2024-09-18T11:44:45.129917+0000","last_peered":"2024-09-18T11:44:45.087728+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:06.100048+0000","last_became_peered":"2024-09-18T11:44:06.100048+0000","last_unstale":"2024-09-18T11:44:45.129947+0000","last_undegraded":"2024-09-18T11:44:45.129947+0000","last_fullsized":"2024-09-18T11:44:45.129947+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1979,"log_dups_size":0,"ondisk_log_size":1979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16192758,"num_objects":1977,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":15814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'2013","reported_seq":2055,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.172593+0000","last_change":"2024-09-18T11:44:32.316693+0000","last_active":"2024-09-18T11:44:44.172593+0000","last_peered":"2024-09-18T11:44:44.172593+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:06.099667+0000","last_became_peered":"2024-09-18T11:44:06.099667+0000","last_unstale":"2024-09-18T11:44:44.172593+0000","last_undegraded":"2024-09-18T11:44:06.087962+0000","last_fullsized":"2024-09-18T11:44:06.087548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16445297,"num_objects":2008,"num_object_clones":0,"num_object_copies":8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2010,"num_write_kb":16068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'2040","reported_seq":2066,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.148252+0000","last_change":"2024-09-18T11:44:32.775850+0000","last_active":"2024-09-18T11:44:44.148252+0000","last_peered":"2024-09-18T11:44:44.148252+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:06.095774+0000","last_became_peered":"2024-09-18T11:44:06.095774+0000","last_unstale":"2024-09-18T11:44:44.148252+0000","last_undegraded":"2024-09-18T11:44:06.087360+0000","last_fullsized":"2024-09-18T11:44:06.087125+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16702075,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":16311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1771","reported_seq":1836,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.800566+0000","last_change":"2024-09-18T11:44:33.362854+0000","last_active":"2024-09-18T11:44:40.800566+0000","last_peered":"2024-09-18T11:44:40.800566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:06.095142+0000","last_became_peered":"2024-09-18T11:44:06.095142+0000","last_unstale":"2024-09-18T11:44:40.800566+0000","last_undegraded":"2024-09-18T11:44:06.092217+0000","last_fullsized":"2024-09-18T11:44:06.091922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1771,"log_dups_size":0,"ondisk_log_size":1771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14498427,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":14159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2108","reported_seq":2144,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:44.155959+0000","last_change":"2024-09-18T11:44:33.907840+0000","last_active":"2024-09-18T11:44:44.155959+0000","last_peered":"2024-09-18T11:44:44.155959+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:06.095532+0000","last_became_peered":"2024-09-18T11:44:06.095532+0000","last_unstale":"2024-09-18T11:44:44.155959+0000","last_undegraded":"2024-09-18T11:44:06.087192+0000","last_fullsized":"2024-09-18T11:44:06.086893+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2108,"log_dups_size":0,"ondisk_log_size":2108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17233142,"num_objects":2104,"num_object_clones":0,"num_object_copies":8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":16844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1837","reported_seq":1906,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.832556+0000","last_change":"2024-09-18T11:44:34.409540+0000","last_active":"2024-09-18T11:44:40.832556+0000","last_peered":"2024-09-18T11:44:40.832556+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:06.100417+0000","last_became_peered":"2024-09-18T11:44:06.100417+0000","last_unstale":"2024-09-18T11:44:40.832556+0000","last_undegraded":"2024-09-18T11:44:06.093073+0000","last_fullsized":"2024-09-18T11:44:06.092766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1837,"log_dups_size":0,"ondisk_log_size":1837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15021302,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1835,"num_write_kb":14677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1774","reported_seq":1824,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.004136+0000","last_change":"2024-09-18T11:44:34.900312+0000","last_active":"2024-09-18T11:44:41.004136+0000","last_peered":"2024-09-18T11:44:41.004136+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:06.096352+0000","last_became_peered":"2024-09-18T11:44:06.096352+0000","last_unstale":"2024-09-18T11:44:41.004136+0000","last_undegraded":"2024-09-18T11:44:06.086622+0000","last_fullsized":"2024-09-18T11:44:06.086322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1774,"log_dups_size":0,"ondisk_log_size":1774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488822,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1772,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1769}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":258098548,"num_objects":31511,"num_object_clones":0,"num_object_copies":126044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31538,"num_write_kb":252186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":406831104,"data_stored":406831104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31565,"ondisk_log_size":31565,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434960,"kb_used_data":128776,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93936880,"statfs":{"total":96636764160,"available":96191365120,"internally_reserved":0,"allocated":131866624,"data_stored":131723035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443452,"kb_used_data":133616,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93928388,"statfs":{"total":96636764160,"available":96182669312,"internally_reserved":0,"allocated":136822784,"data_stored":136675291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":278528,"data_stored":168931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439980,"kb_used_data":133852,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93931860,"statfs":{"total":96636764160,"available":96186224640,"internally_reserved":0,"allocated":137064448,"data_stored":136912992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":133079040,"data_stored":133079040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:48.209 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 31 20 33749 33729 8.49908 0.898438 0.0417224 0.0183693 2024-09-18T11:44:49.209 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 32 20 34230 34210 8.35091 3.75781 0.0601848 0.0186965 2024-09-18T11:44:50.209 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 33 20 34720 34700 8.21385 3.82812 0.0250245 0.0190107 2024-09-18T11:44:50.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:51.209 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 34 20 35163 35143 8.07406 3.46094 0.0663749 0.0193317 2024-09-18T11:44:51.245 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:51.246 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:51.275 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-09-18T11:44:49.832322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264447348,"num_objects":32286,"num_object_clones":0,"num_object_copies":129144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32313,"num_write_kb":258386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32340,"ondisk_log_size":32340,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1626824,"kb_used_data":406868,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375860536,"statfs":{"total":386547056640,"available":384881188864,"internally_reserved":0,"allocated":416632832,"data_stored":416015927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,12,7,9,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30687232,"num_objects":3746,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":29968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.724743"},"pg_stats":[{"pgid":"1.f","version":"31'2177","reported_seq":2217,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.221160+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:44:49.221160+0000","last_peered":"2024-09-18T11:44:49.221160+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:44:49.221160+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17833984,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2177,"num_write_kb":17416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1852","reported_seq":1902,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.990614+0000","last_change":"2024-09-18T11:44:28.260359+0000","last_active":"2024-09-18T11:44:40.990614+0000","last_peered":"2024-09-18T11:44:40.990614+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:06.096040+0000","last_became_peered":"2024-09-18T11:44:06.096040+0000","last_unstale":"2024-09-18T11:44:40.990614+0000","last_undegraded":"2024-09-18T11:44:06.086523+0000","last_fullsized":"2024-09-18T11:44:06.086250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1852,"log_dups_size":0,"ondisk_log_size":1852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15135990,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1850,"num_write_kb":14796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1848}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2058","reported_seq":2128,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129270+0000","last_change":"2024-09-18T11:44:45.130506+0000","last_active":"2024-09-18T11:44:45.129239+0000","last_peered":"2024-09-18T11:44:45.099594+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:06.100615+0000","last_became_peered":"2024-09-18T11:44:06.100615+0000","last_unstale":"2024-09-18T11:44:45.129270+0000","last_undegraded":"2024-09-18T11:44:45.129270+0000","last_fullsized":"2024-09-18T11:44:45.129270+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16833147,"num_objects":2055,"num_object_clones":0,"num_object_copies":8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1870","reported_seq":1920,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.012499+0000","last_change":"2024-09-18T11:44:29.314915+0000","last_active":"2024-09-18T11:44:41.012499+0000","last_peered":"2024-09-18T11:44:41.012499+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:06.099305+0000","last_became_peered":"2024-09-18T11:44:06.099305+0000","last_unstale":"2024-09-18T11:44:41.012499+0000","last_undegraded":"2024-09-18T11:44:06.085610+0000","last_fullsized":"2024-09-18T11:44:06.085286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15283446,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1866}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2134","reported_seq":2189,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.189733+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:44:49.189733+0000","last_peered":"2024-09-18T11:44:49.189733+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:44:49.189733+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2134,"log_dups_size":0,"ondisk_log_size":2134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17454326,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":17053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2202","reported_seq":2245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.199413+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:44:49.199413+0000","last_peered":"2024-09-18T11:44:49.199413+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:44:49.199413+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18038784,"num_objects":2202,"num_object_clones":0,"num_object_copies":8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2142","reported_seq":2195,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.122860+0000","last_change":"2024-09-18T11:44:45.126986+0000","last_active":"2024-09-18T11:44:45.122834+0000","last_peered":"2024-09-18T11:44:45.089661+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:06.096306+0000","last_became_peered":"2024-09-18T11:44:06.096306+0000","last_unstale":"2024-09-18T11:44:45.122860+0000","last_undegraded":"2024-09-18T11:44:45.122860+0000","last_fullsized":"2024-09-18T11:44:45.122860+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17495286,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":17100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1819","reported_seq":1869,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.006933+0000","last_change":"2024-09-18T11:44:30.758028+0000","last_active":"2024-09-18T11:44:41.006933+0000","last_peered":"2024-09-18T11:44:41.006933+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:06.095136+0000","last_became_peered":"2024-09-18T11:44:06.095136+0000","last_unstale":"2024-09-18T11:44:41.006933+0000","last_undegraded":"2024-09-18T11:44:06.086038+0000","last_fullsized":"2024-09-18T11:44:06.085713+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1819,"log_dups_size":0,"ondisk_log_size":1819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857462,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1817,"num_write_kb":14524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2214","reported_seq":2272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.192676+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:44:49.192676+0000","last_peered":"2024-09-18T11:44:49.192676+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:44:49.192676+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18065900,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":17665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":71}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1979","reported_seq":2051,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129947+0000","last_change":"2024-09-18T11:44:45.131087+0000","last_active":"2024-09-18T11:44:45.129917+0000","last_peered":"2024-09-18T11:44:45.087728+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:06.100048+0000","last_became_peered":"2024-09-18T11:44:06.100048+0000","last_unstale":"2024-09-18T11:44:45.129947+0000","last_undegraded":"2024-09-18T11:44:45.129947+0000","last_fullsized":"2024-09-18T11:44:45.129947+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1979,"log_dups_size":0,"ondisk_log_size":1979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16192758,"num_objects":1977,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":15814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2127","reported_seq":2184,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.180502+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:44:49.180502+0000","last_peered":"2024-09-18T11:44:49.180502+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:44:49.180502+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2127,"log_dups_size":0,"ondisk_log_size":2127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17379185,"num_objects":2122,"num_object_clones":0,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2124,"num_write_kb":16980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":59}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2158","reported_seq":2195,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.189658+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:44:49.189658+0000","last_peered":"2024-09-18T11:44:49.189658+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:44:49.189658+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17668731,"num_objects":2157,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2157,"num_write_kb":17255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":48}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1771","reported_seq":1836,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.800566+0000","last_change":"2024-09-18T11:44:33.362854+0000","last_active":"2024-09-18T11:44:40.800566+0000","last_peered":"2024-09-18T11:44:40.800566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:06.095142+0000","last_became_peered":"2024-09-18T11:44:06.095142+0000","last_unstale":"2024-09-18T11:44:40.800566+0000","last_undegraded":"2024-09-18T11:44:06.092217+0000","last_fullsized":"2024-09-18T11:44:06.091922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1771,"log_dups_size":0,"ondisk_log_size":1771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14498427,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":14159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2226","reported_seq":2275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.222676+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:44:49.222676+0000","last_peered":"2024-09-18T11:44:49.222676+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:44:49.222676+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18199798,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":17788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1837","reported_seq":1906,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.832556+0000","last_change":"2024-09-18T11:44:34.409540+0000","last_active":"2024-09-18T11:44:40.832556+0000","last_peered":"2024-09-18T11:44:40.832556+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:06.100417+0000","last_became_peered":"2024-09-18T11:44:06.100417+0000","last_unstale":"2024-09-18T11:44:40.832556+0000","last_undegraded":"2024-09-18T11:44:06.093073+0000","last_fullsized":"2024-09-18T11:44:06.092766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1837,"log_dups_size":0,"ondisk_log_size":1837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15021302,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1835,"num_write_kb":14677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1774","reported_seq":1824,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.004136+0000","last_change":"2024-09-18T11:44:34.900312+0000","last_active":"2024-09-18T11:44:41.004136+0000","last_peered":"2024-09-18T11:44:41.004136+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:06.096352+0000","last_became_peered":"2024-09-18T11:44:06.096352+0000","last_unstale":"2024-09-18T11:44:41.004136+0000","last_undegraded":"2024-09-18T11:44:06.086622+0000","last_fullsized":"2024-09-18T11:44:06.086322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1774,"log_dups_size":0,"ondisk_log_size":1774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488822,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1772,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1769}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":264447348,"num_objects":32286,"num_object_clones":0,"num_object_copies":129144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32313,"num_write_kb":258386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417423360,"data_stored":417423360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32340,"ondisk_log_size":32340,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442608,"kb_used_data":136420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93929232,"statfs":{"total":96636764160,"available":96183533568,"internally_reserved":0,"allocated":139694080,"data_stored":139535102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,1,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7759999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.413}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.222}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443452,"kb_used_data":133616,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93928388,"statfs":{"total":96636764160,"available":96182669312,"internally_reserved":0,"allocated":136822784,"data_stored":136675291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]}]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300784,"kb_used_data":2980,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94071056,"statfs":{"total":96636764160,"available":96328761344,"internally_reserved":0,"allocated":3051520,"data_stored":2892542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8439999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439980,"kb_used_data":133852,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93931860,"statfs":{"total":96636764160,"available":96186224640,"internally_reserved":0,"allocated":137064448,"data_stored":136912992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3534848,"data_stored":3534848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140251136,"data_stored":140251136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:51.276 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:51.790 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:51.791 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:51.824 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-09-18T11:44:49.832322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264447348,"num_objects":32286,"num_object_clones":0,"num_object_copies":129144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32313,"num_write_kb":258386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32340,"ondisk_log_size":32340,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1626824,"kb_used_data":406868,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375860536,"statfs":{"total":386547056640,"available":384881188864,"internally_reserved":0,"allocated":416632832,"data_stored":416015927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,12,7,9,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30687232,"num_objects":3746,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":29968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.724743"},"pg_stats":[{"pgid":"1.f","version":"31'2177","reported_seq":2217,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.221160+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:44:49.221160+0000","last_peered":"2024-09-18T11:44:49.221160+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:44:49.221160+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17833984,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2177,"num_write_kb":17416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1852","reported_seq":1902,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.990614+0000","last_change":"2024-09-18T11:44:28.260359+0000","last_active":"2024-09-18T11:44:40.990614+0000","last_peered":"2024-09-18T11:44:40.990614+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:06.096040+0000","last_became_peered":"2024-09-18T11:44:06.096040+0000","last_unstale":"2024-09-18T11:44:40.990614+0000","last_undegraded":"2024-09-18T11:44:06.086523+0000","last_fullsized":"2024-09-18T11:44:06.086250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1852,"log_dups_size":0,"ondisk_log_size":1852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15135990,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1850,"num_write_kb":14796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1848}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2058","reported_seq":2128,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129270+0000","last_change":"2024-09-18T11:44:45.130506+0000","last_active":"2024-09-18T11:44:45.129239+0000","last_peered":"2024-09-18T11:44:45.099594+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:06.100615+0000","last_became_peered":"2024-09-18T11:44:06.100615+0000","last_unstale":"2024-09-18T11:44:45.129270+0000","last_undegraded":"2024-09-18T11:44:45.129270+0000","last_fullsized":"2024-09-18T11:44:45.129270+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16833147,"num_objects":2055,"num_object_clones":0,"num_object_copies":8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1870","reported_seq":1920,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.012499+0000","last_change":"2024-09-18T11:44:29.314915+0000","last_active":"2024-09-18T11:44:41.012499+0000","last_peered":"2024-09-18T11:44:41.012499+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:06.099305+0000","last_became_peered":"2024-09-18T11:44:06.099305+0000","last_unstale":"2024-09-18T11:44:41.012499+0000","last_undegraded":"2024-09-18T11:44:06.085610+0000","last_fullsized":"2024-09-18T11:44:06.085286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15283446,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1866}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2134","reported_seq":2189,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.189733+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:44:49.189733+0000","last_peered":"2024-09-18T11:44:49.189733+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:44:49.189733+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2134,"log_dups_size":0,"ondisk_log_size":2134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17454326,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":17053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2202","reported_seq":2245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.199413+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:44:49.199413+0000","last_peered":"2024-09-18T11:44:49.199413+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:44:49.199413+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18038784,"num_objects":2202,"num_object_clones":0,"num_object_copies":8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2142","reported_seq":2195,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.122860+0000","last_change":"2024-09-18T11:44:45.126986+0000","last_active":"2024-09-18T11:44:45.122834+0000","last_peered":"2024-09-18T11:44:45.089661+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:06.096306+0000","last_became_peered":"2024-09-18T11:44:06.096306+0000","last_unstale":"2024-09-18T11:44:45.122860+0000","last_undegraded":"2024-09-18T11:44:45.122860+0000","last_fullsized":"2024-09-18T11:44:45.122860+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17495286,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":17100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1819","reported_seq":1869,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.006933+0000","last_change":"2024-09-18T11:44:30.758028+0000","last_active":"2024-09-18T11:44:41.006933+0000","last_peered":"2024-09-18T11:44:41.006933+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:06.095136+0000","last_became_peered":"2024-09-18T11:44:06.095136+0000","last_unstale":"2024-09-18T11:44:41.006933+0000","last_undegraded":"2024-09-18T11:44:06.086038+0000","last_fullsized":"2024-09-18T11:44:06.085713+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1819,"log_dups_size":0,"ondisk_log_size":1819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857462,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1817,"num_write_kb":14524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2214","reported_seq":2272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.192676+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:44:49.192676+0000","last_peered":"2024-09-18T11:44:49.192676+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:44:49.192676+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18065900,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":17665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":71}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1979","reported_seq":2051,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:44:45.129947+0000","last_change":"2024-09-18T11:44:45.131087+0000","last_active":"2024-09-18T11:44:45.129917+0000","last_peered":"2024-09-18T11:44:45.087728+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:06.100048+0000","last_became_peered":"2024-09-18T11:44:06.100048+0000","last_unstale":"2024-09-18T11:44:45.129947+0000","last_undegraded":"2024-09-18T11:44:45.129947+0000","last_fullsized":"2024-09-18T11:44:45.129947+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1979,"log_dups_size":0,"ondisk_log_size":1979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16192758,"num_objects":1977,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":15814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2127","reported_seq":2184,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.180502+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:44:49.180502+0000","last_peered":"2024-09-18T11:44:49.180502+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:44:49.180502+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2127,"log_dups_size":0,"ondisk_log_size":2127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17379185,"num_objects":2122,"num_object_clones":0,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2124,"num_write_kb":16980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":59}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2158","reported_seq":2195,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.189658+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:44:49.189658+0000","last_peered":"2024-09-18T11:44:49.189658+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:44:49.189658+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17668731,"num_objects":2157,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2157,"num_write_kb":17255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":48}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1771","reported_seq":1836,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.800566+0000","last_change":"2024-09-18T11:44:33.362854+0000","last_active":"2024-09-18T11:44:40.800566+0000","last_peered":"2024-09-18T11:44:40.800566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:06.095142+0000","last_became_peered":"2024-09-18T11:44:06.095142+0000","last_unstale":"2024-09-18T11:44:40.800566+0000","last_undegraded":"2024-09-18T11:44:06.092217+0000","last_fullsized":"2024-09-18T11:44:06.091922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1771,"log_dups_size":0,"ondisk_log_size":1771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14498427,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":14159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2226","reported_seq":2275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.222676+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:44:49.222676+0000","last_peered":"2024-09-18T11:44:49.222676+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:44:49.222676+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18199798,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":17788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1837","reported_seq":1906,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:40.832556+0000","last_change":"2024-09-18T11:44:34.409540+0000","last_active":"2024-09-18T11:44:40.832556+0000","last_peered":"2024-09-18T11:44:40.832556+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:06.100417+0000","last_became_peered":"2024-09-18T11:44:06.100417+0000","last_unstale":"2024-09-18T11:44:40.832556+0000","last_undegraded":"2024-09-18T11:44:06.093073+0000","last_fullsized":"2024-09-18T11:44:06.092766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1837,"log_dups_size":0,"ondisk_log_size":1837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15021302,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1835,"num_write_kb":14677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1774","reported_seq":1824,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-18T11:44:41.004136+0000","last_change":"2024-09-18T11:44:34.900312+0000","last_active":"2024-09-18T11:44:41.004136+0000","last_peered":"2024-09-18T11:44:41.004136+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:06.096352+0000","last_became_peered":"2024-09-18T11:44:06.096352+0000","last_unstale":"2024-09-18T11:44:41.004136+0000","last_undegraded":"2024-09-18T11:44:06.086622+0000","last_fullsized":"2024-09-18T11:44:06.086322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":1774,"log_dups_size":0,"ondisk_log_size":1774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488822,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1772,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1769}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":264447348,"num_objects":32286,"num_object_clones":0,"num_object_copies":129144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32313,"num_write_kb":258386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417423360,"data_stored":417423360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32340,"ondisk_log_size":32340,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442608,"kb_used_data":136420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93929232,"statfs":{"total":96636764160,"available":96183533568,"internally_reserved":0,"allocated":139694080,"data_stored":139535102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,1,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7759999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.413}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.222}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443452,"kb_used_data":133616,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93928388,"statfs":{"total":96636764160,"available":96182669312,"internally_reserved":0,"allocated":136822784,"data_stored":136675291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]}]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300784,"kb_used_data":2980,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94071056,"statfs":{"total":96636764160,"available":96328761344,"internally_reserved":0,"allocated":3051520,"data_stored":2892542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8439999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439980,"kb_used_data":133852,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93931860,"statfs":{"total":96636764160,"available":96186224640,"internally_reserved":0,"allocated":137064448,"data_stored":136912992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3534848,"data_stored":3534848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136818688,"data_stored":136818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140251136,"data_stored":140251136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:51.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:44:51.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2177", 'reported_seq': 2217, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:49.221160+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:44:49.221160+0000', 'last_peered': '2024-09-18T11:44:49.221160+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:44:49.221160+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2177, 'log_dups_size': 0, 'ondisk_log_size': 2177, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17833984, 'num_objects': 2177, 'num_object_clones': 0, 'num_object_copies': 8708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2177, 'num_write_kb': 17416, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 47}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:51.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:44:51.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "27'1852", 'reported_seq': 1902, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:40.990614+0000', 'last_change': '2024-09-18T11:44:28.260359+0000', 'last_active': '2024-09-18T11:44:40.990614+0000', 'last_peered': '2024-09-18T11:44:40.990614+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:06.096040+0000', 'last_became_peered': '2024-09-18T11:44:06.096040+0000', 'last_unstale': '2024-09-18T11:44:40.990614+0000', 'last_undegraded': '2024-09-18T11:44:06.086523+0000', 'last_fullsized': '2024-09-18T11:44:06.086250+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1852, 'log_dups_size': 0, 'ondisk_log_size': 1852, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15135990, 'num_objects': 1848, 'num_object_clones': 0, 'num_object_copies': 7392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1848, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1850, 'num_write_kb': 14796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1848}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:51.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:44:51.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "28'2058", 'reported_seq': 2128, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-18T11:44:45.129270+0000', 'last_change': '2024-09-18T11:44:45.130506+0000', 'last_active': '2024-09-18T11:44:45.129239+0000', 'last_peered': '2024-09-18T11:44:45.099594+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:06.100615+0000', 'last_became_peered': '2024-09-18T11:44:06.100615+0000', 'last_unstale': '2024-09-18T11:44:45.129270+0000', 'last_undegraded': '2024-09-18T11:44:45.129270+0000', 'last_fullsized': '2024-09-18T11:44:45.129270+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2058, 'log_dups_size': 0, 'ondisk_log_size': 2058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16833147, 'num_objects': 2055, 'num_object_clones': 0, 'num_object_copies': 8220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2057, 'num_write_kb': 16446, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 1, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:51.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:44:51.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "27'1870", 'reported_seq': 1920, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:41.012499+0000', 'last_change': '2024-09-18T11:44:29.314915+0000', 'last_active': '2024-09-18T11:44:41.012499+0000', 'last_peered': '2024-09-18T11:44:41.012499+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:06.099305+0000', 'last_became_peered': '2024-09-18T11:44:06.099305+0000', 'last_unstale': '2024-09-18T11:44:41.012499+0000', 'last_undegraded': '2024-09-18T11:44:06.085610+0000', 'last_fullsized': '2024-09-18T11:44:06.085286+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1870, 'log_dups_size': 0, 'ondisk_log_size': 1870, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15283446, 'num_objects': 1866, 'num_object_clones': 0, 'num_object_copies': 7464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1866, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1868, 'num_write_kb': 14933, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1866}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:51.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:44:51.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2134", 'reported_seq': 2189, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:49.189733+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:44:49.189733+0000', 'last_peered': '2024-09-18T11:44:49.189733+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:44:49.189733+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2134, 'log_dups_size': 0, 'ondisk_log_size': 2134, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17454326, 'num_objects': 2131, 'num_object_clones': 0, 'num_object_copies': 8524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2132, 'num_write_kb': 17053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 49}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:51.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:44:51.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2202", 'reported_seq': 2245, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:49.199413+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:44:49.199413+0000', 'last_peered': '2024-09-18T11:44:49.199413+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:44:49.199413+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2202, 'log_dups_size': 0, 'ondisk_log_size': 2202, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18038784, 'num_objects': 2202, 'num_object_clones': 0, 'num_object_copies': 8808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2202, 'num_write_kb': 17616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 48}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:51.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:44:51.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "28'2142", 'reported_seq': 2195, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-18T11:44:45.122860+0000', 'last_change': '2024-09-18T11:44:45.126986+0000', 'last_active': '2024-09-18T11:44:45.122834+0000', 'last_peered': '2024-09-18T11:44:45.089661+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:06.096306+0000', 'last_became_peered': '2024-09-18T11:44:06.096306+0000', 'last_unstale': '2024-09-18T11:44:45.122860+0000', 'last_undegraded': '2024-09-18T11:44:45.122860+0000', 'last_fullsized': '2024-09-18T11:44:45.122860+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2142, 'log_dups_size': 0, 'ondisk_log_size': 2142, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17495286, 'num_objects': 2136, 'num_object_clones': 0, 'num_object_copies': 8544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2140, 'num_write_kb': 17100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:51.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:44:51.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "27'1819", 'reported_seq': 1869, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:41.006933+0000', 'last_change': '2024-09-18T11:44:30.758028+0000', 'last_active': '2024-09-18T11:44:41.006933+0000', 'last_peered': '2024-09-18T11:44:41.006933+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:06.095136+0000', 'last_became_peered': '2024-09-18T11:44:06.095136+0000', 'last_unstale': '2024-09-18T11:44:41.006933+0000', 'last_undegraded': '2024-09-18T11:44:06.086038+0000', 'last_fullsized': '2024-09-18T11:44:06.085713+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1819, 'log_dups_size': 0, 'ondisk_log_size': 1819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14857462, 'num_objects': 1814, 'num_object_clones': 0, 'num_object_copies': 7256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1814, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1814, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1817, 'num_write_kb': 14524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1814}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2214", 'reported_seq': 2272, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:49.192676+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:44:49.192676+0000', 'last_peered': '2024-09-18T11:44:49.192676+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:44:49.192676+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2214, 'log_dups_size': 0, 'ondisk_log_size': 2214, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18065900, 'num_objects': 2206, 'num_object_clones': 0, 'num_object_copies': 8824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2211, 'num_write_kb': 17665, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 71}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "28'1979", 'reported_seq': 2051, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-09-18T11:44:45.129947+0000', 'last_change': '2024-09-18T11:44:45.131087+0000', 'last_active': '2024-09-18T11:44:45.129917+0000', 'last_peered': '2024-09-18T11:44:45.087728+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:06.100048+0000', 'last_became_peered': '2024-09-18T11:44:06.100048+0000', 'last_unstale': '2024-09-18T11:44:45.129947+0000', 'last_undegraded': '2024-09-18T11:44:45.129947+0000', 'last_fullsized': '2024-09-18T11:44:45.129947+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1979, 'log_dups_size': 0, 'ondisk_log_size': 1979, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16192758, 'num_objects': 1977, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1977, 'num_write_kb': 15814, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2127", 'reported_seq': 2184, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:49.180502+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:44:49.180502+0000', 'last_peered': '2024-09-18T11:44:49.180502+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:44:49.180502+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2127, 'log_dups_size': 0, 'ondisk_log_size': 2127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17379185, 'num_objects': 2122, 'num_object_clones': 0, 'num_object_copies': 8488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2124, 'num_write_kb': 16980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 59}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2158", 'reported_seq': 2195, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:49.189658+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:44:49.189658+0000', 'last_peered': '2024-09-18T11:44:49.189658+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:44:49.189658+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2158, 'log_dups_size': 0, 'ondisk_log_size': 2158, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17668731, 'num_objects': 2157, 'num_object_clones': 0, 'num_object_copies': 8628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2157, 'num_write_kb': 17255, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 48}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "27'1771", 'reported_seq': 1836, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:40.800566+0000', 'last_change': '2024-09-18T11:44:33.362854+0000', 'last_active': '2024-09-18T11:44:40.800566+0000', 'last_peered': '2024-09-18T11:44:40.800566+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:06.095142+0000', 'last_became_peered': '2024-09-18T11:44:06.095142+0000', 'last_unstale': '2024-09-18T11:44:40.800566+0000', 'last_undegraded': '2024-09-18T11:44:06.092217+0000', 'last_fullsized': '2024-09-18T11:44:06.091922+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1771, 'log_dups_size': 0, 'ondisk_log_size': 1771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14498427, 'num_objects': 1770, 'num_object_clones': 0, 'num_object_copies': 7080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1770, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1770, 'num_write_kb': 14159, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1770}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:51.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:44:51.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2226", 'reported_seq': 2275, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:49.222676+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:44:49.222676+0000', 'last_peered': '2024-09-18T11:44:49.222676+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:44:49.222676+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2226, 'log_dups_size': 0, 'ondisk_log_size': 2226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18199798, 'num_objects': 2222, 'num_object_clones': 0, 'num_object_copies': 8888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2224, 'num_write_kb': 17788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 64}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:51.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:44:51.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "27'1837", 'reported_seq': 1906, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:40.832556+0000', 'last_change': '2024-09-18T11:44:34.409540+0000', 'last_active': '2024-09-18T11:44:40.832556+0000', 'last_peered': '2024-09-18T11:44:40.832556+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:06.100417+0000', 'last_became_peered': '2024-09-18T11:44:06.100417+0000', 'last_unstale': '2024-09-18T11:44:40.832556+0000', 'last_undegraded': '2024-09-18T11:44:06.093073+0000', 'last_fullsized': '2024-09-18T11:44:06.092766+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1837, 'log_dups_size': 0, 'ondisk_log_size': 1837, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15021302, 'num_objects': 1834, 'num_object_clones': 0, 'num_object_copies': 7336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1834, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1835, 'num_write_kb': 14677, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1834}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:51.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:44:51.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "27'1774", 'reported_seq': 1824, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-18T11:44:41.004136+0000', 'last_change': '2024-09-18T11:44:34.900312+0000', 'last_active': '2024-09-18T11:44:41.004136+0000', 'last_peered': '2024-09-18T11:44:41.004136+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:06.096352+0000', 'last_became_peered': '2024-09-18T11:44:06.096352+0000', 'last_unstale': '2024-09-18T11:44:41.004136+0000', 'last_undegraded': '2024-09-18T11:44:06.086622+0000', 'last_fullsized': '2024-09-18T11:44:06.086322+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 1774, 'log_dups_size': 0, 'ondisk_log_size': 1774, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14488822, 'num_objects': 1769, 'num_object_clones': 0, 'num_object_copies': 7076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1769, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1772, 'num_write_kb': 14164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1769}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:51.828 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:44:52.211 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 35 20 35576 35556 7.93557 3.22656 0.0402597 0.0196674 2024-09-18T11:44:52.513 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:52.513 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33367/138736 objects degraded (24.051%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":90,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":34684,"data_bytes":284091764,"bytes_used":1669296128,"bytes_avail":384877760512,"bytes_total":386547056640,"degraded_objects":33367,"degraded_total":138736,"degraded_ratio":0.24050715027101835,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":73146,"recovering_keys_per_sec":0,"num_objects_recovered":60,"num_bytes_recovered":491520,"num_keys_recovered":0,"write_bytes_sec":4806961,"read_op_per_sec":0,"write_op_per_sec":586},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-18T11:44:31.826669+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:44:52.538 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:44:52.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:53.052 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:53.053 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:53.072 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-09-18T11:44:51.832868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":284091764,"num_objects":34684,"num_object_clones":0,"num_object_copies":138736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34711,"num_write_kb":277570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34738,"ondisk_log_size":34738,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1630172,"kb_used_data":418548,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375857188,"statfs":{"total":386547056640,"available":384877760512,"internally_reserved":0,"allocated":428593152,"data_stored":427957240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,12,10,13,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32301056,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3943,"num_write_kb":31544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.719640"},"pg_stats":[{"pgid":"1.f","version":"31'2177","reported_seq":2217,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.221160+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:44:49.221160+0000","last_peered":"2024-09-18T11:44:49.221160+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:44:49.221160+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17833984,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2177,"num_write_kb":17416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2201","reported_seq":2263,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.933531+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:44:50.933531+0000","last_peered":"2024-09-18T11:44:50.933531+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:44:50.933531+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2201,"log_dups_size":0,"ondisk_log_size":2201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17994998,"num_objects":2197,"num_object_clones":0,"num_object_copies":8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2199,"num_write_kb":17588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2148","reported_seq":2229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.817206+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:44:50.817206+0000","last_peered":"2024-09-18T11:44:50.817206+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:44:50.817206+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17570427,"num_objects":2145,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":17166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":91}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2234","reported_seq":2296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.932777+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:44:50.932777+0000","last_peered":"2024-09-18T11:44:50.932777+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:44:50.932777+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18265334,"num_objects":2230,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":17845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2134","reported_seq":2189,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.189733+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:44:49.189733+0000","last_peered":"2024-09-18T11:44:49.189733+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:44:49.189733+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2134,"log_dups_size":0,"ondisk_log_size":2134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17454326,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":17053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2202","reported_seq":2245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.199413+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:44:49.199413+0000","last_peered":"2024-09-18T11:44:49.199413+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:44:49.199413+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18038784,"num_objects":2202,"num_object_clones":0,"num_object_copies":8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2226","reported_seq":2291,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.964518+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:44:50.964518+0000","last_peered":"2024-09-18T11:44:50.964518+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:44:50.964518+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183414,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":17772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":86}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2146","reported_seq":2209,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.993266+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:44:50.993266+0000","last_peered":"2024-09-18T11:44:50.993266+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:44:50.993266+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2146,"log_dups_size":0,"ondisk_log_size":2146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17536246,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":17140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2214","reported_seq":2272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.192676+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:44:49.192676+0000","last_peered":"2024-09-18T11:44:49.192676+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:44:49.192676+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18065900,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":17665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":71}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2073","reported_seq":2157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.700123+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:44:50.700123+0000","last_peered":"2024-09-18T11:44:50.700123+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:44:50.700123+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16962806,"num_objects":2071,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2071,"num_write_kb":16566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":96}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2127","reported_seq":2184,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.180502+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:44:49.180502+0000","last_peered":"2024-09-18T11:44:49.180502+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:44:49.180502+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2127,"log_dups_size":0,"ondisk_log_size":2127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17379185,"num_objects":2122,"num_object_clones":0,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2124,"num_write_kb":16980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":59}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2158","reported_seq":2195,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.189658+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:44:49.189658+0000","last_peered":"2024-09-18T11:44:49.189658+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:44:49.189658+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17668731,"num_objects":2157,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2157,"num_write_kb":17255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":48}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2159","reported_seq":2234,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.821068+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:44:50.821068+0000","last_peered":"2024-09-18T11:44:50.821068+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:44:50.821068+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2159,"log_dups_size":0,"ondisk_log_size":2159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17676923,"num_objects":2158,"num_object_clones":0,"num_object_copies":8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2158,"num_write_kb":17263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2226","reported_seq":2275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:49.222676+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:44:49.222676+0000","last_peered":"2024-09-18T11:44:49.222676+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:44:49.222676+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18199798,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":17788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2195","reported_seq":2398,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.845551+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:44:50.845551+0000","last_peered":"2024-09-18T11:44:50.845551+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:44:50.845551+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2195,"log_dups_size":0,"ondisk_log_size":2195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17954038,"num_objects":2192,"num_object_clones":0,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2193,"num_write_kb":17541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2118","reported_seq":2180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.950841+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:44:50.950841+0000","last_peered":"2024-09-18T11:44:50.950841+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:44:50.950841+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2118,"log_dups_size":0,"ondisk_log_size":2118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17306870,"num_objects":2113,"num_object_clones":0,"num_object_copies":8452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2116,"num_write_kb":16916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":284091764,"num_objects":34684,"num_object_clones":0,"num_object_copies":138736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34711,"num_write_kb":277570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430936064,"data_stored":430936064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34738,"ondisk_log_size":34738,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442608,"kb_used_data":136420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93929232,"statfs":{"total":96636764160,"available":96183533568,"internally_reserved":0,"allocated":139694080,"data_stored":139535102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,1,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7759999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.413}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.222}]}]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":436144,"kb_used_data":138792,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93935696,"statfs":{"total":96636764160,"available":96190152704,"internally_reserved":0,"allocated":142123008,"data_stored":141964030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.8179999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7970000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.8470000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300784,"kb_used_data":2980,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94071056,"statfs":{"total":96636764160,"available":96328761344,"internally_reserved":0,"allocated":3051520,"data_stored":2892542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8439999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450636,"kb_used_data":140356,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93921204,"statfs":{"total":96636764160,"available":96175312896,"internally_reserved":0,"allocated":143724544,"data_stored":143565566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4470000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143704064,"data_stored":143704064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3534848,"data_stored":3534848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143446016,"data_stored":143446016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140251136,"data_stored":140251136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:53.212 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 36 20 36000 35980 7.80662 3.3125 0.0177362 0.0200007 2024-09-18T11:44:54.212 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 37 20 36470 36450 7.69487 3.67188 0.0327325 0.0202864 2024-09-18T11:44:55.212 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 38 20 36948 36928 7.59065 3.73438 0.0742628 0.020564 2024-09-18T11:44:56.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:56.212 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 39 20 37417 37397 7.48997 3.66406 0.0899723 0.0208407 2024-09-18T11:44:56.588 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:56.588 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:56.613 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-09-18T11:44:55.834136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":292382068,"num_objects":35696,"num_object_clones":0,"num_object_copies":142784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35723,"num_write_kb":285666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35750,"ondisk_log_size":35750,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1660596,"kb_used_data":436500,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375826764,"statfs":{"total":386547056640,"available":384846606336,"internally_reserved":0,"allocated":446976000,"data_stored":446340088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,6,8,16,18,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34283520,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.726616"},"pg_stats":[{"pgid":"1.f","version":"31'2331","reported_seq":2372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.237048+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:44:54.237048+0000","last_peered":"2024-09-18T11:44:54.237048+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:44:54.237048+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2331,"log_dups_size":0,"ondisk_log_size":2331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19095552,"num_objects":2331,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2331,"num_write_kb":18648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2201","reported_seq":2263,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.933531+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:44:50.933531+0000","last_peered":"2024-09-18T11:44:50.933531+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:44:50.933531+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2201,"log_dups_size":0,"ondisk_log_size":2201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17994998,"num_objects":2197,"num_object_clones":0,"num_object_copies":8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2199,"num_write_kb":17588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2148","reported_seq":2229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.817206+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:44:50.817206+0000","last_peered":"2024-09-18T11:44:50.817206+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:44:50.817206+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17570427,"num_objects":2145,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":17166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":91}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2234","reported_seq":2296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.932777+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:44:50.932777+0000","last_peered":"2024-09-18T11:44:50.932777+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:44:50.932777+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18265334,"num_objects":2230,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":17845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2272","reported_seq":2325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.216716+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:44:54.216716+0000","last_peered":"2024-09-18T11:44:54.216716+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:44:54.216716+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2272,"log_dups_size":0,"ondisk_log_size":2272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584822,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2270,"num_write_kb":18157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2339","reported_seq":2382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.253670+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:44:54.253670+0000","last_peered":"2024-09-18T11:44:54.253670+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:44:54.253670+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19161088,"num_objects":2339,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2339,"num_write_kb":18712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2226","reported_seq":2291,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.964518+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:44:50.964518+0000","last_peered":"2024-09-18T11:44:50.964518+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:44:50.964518+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183414,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":17772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":86}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2146","reported_seq":2209,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.993266+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:44:50.993266+0000","last_peered":"2024-09-18T11:44:50.993266+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:44:50.993266+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2146,"log_dups_size":0,"ondisk_log_size":2146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17536246,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":17140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2361","reported_seq":2419,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.258888+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:44:54.258888+0000","last_peered":"2024-09-18T11:44:54.258888+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:44:54.258888+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2361,"log_dups_size":0,"ondisk_log_size":2361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19270124,"num_objects":2353,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2358,"num_write_kb":18841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2073","reported_seq":2157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.700123+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:44:50.700123+0000","last_peered":"2024-09-18T11:44:50.700123+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:44:50.700123+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16962806,"num_objects":2071,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2071,"num_write_kb":16566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":96}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2269","reported_seq":2326,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.144164+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:44:54.144164+0000","last_peered":"2024-09-18T11:44:54.144164+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:44:54.144164+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2269,"log_dups_size":0,"ondisk_log_size":2269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18542449,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2266,"num_write_kb":18116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2307","reported_seq":2344,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.162973+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:44:54.162973+0000","last_peered":"2024-09-18T11:44:54.162973+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:44:54.162973+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2307,"log_dups_size":0,"ondisk_log_size":2307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18889339,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2306,"num_write_kb":18447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":197}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2159","reported_seq":2234,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.821068+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:44:50.821068+0000","last_peered":"2024-09-18T11:44:50.821068+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:44:50.821068+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2159,"log_dups_size":0,"ondisk_log_size":2159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17676923,"num_objects":2158,"num_object_clones":0,"num_object_copies":8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2158,"num_write_kb":17263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2371","reported_seq":2420,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.180734+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:44:54.180734+0000","last_peered":"2024-09-18T11:44:54.180734+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:44:54.180734+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2371,"log_dups_size":0,"ondisk_log_size":2371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387638,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2369,"num_write_kb":18948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":209}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2195","reported_seq":2398,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.845551+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:44:50.845551+0000","last_peered":"2024-09-18T11:44:50.845551+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:44:50.845551+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2195,"log_dups_size":0,"ondisk_log_size":2195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17954038,"num_objects":2192,"num_object_clones":0,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2193,"num_write_kb":17541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2118","reported_seq":2180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.950841+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:44:50.950841+0000","last_peered":"2024-09-18T11:44:50.950841+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:44:50.950841+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2118,"log_dups_size":0,"ondisk_log_size":2118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17306870,"num_objects":2113,"num_object_clones":0,"num_object_copies":8452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2116,"num_write_kb":16916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":292382068,"num_objects":35696,"num_object_clones":0,"num_object_copies":142784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35723,"num_write_kb":285666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":449716224,"data_stored":449716224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35750,"ondisk_log_size":35750,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459840,"kb_used_data":145336,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93912000,"statfs":{"total":96636764160,"available":96165888000,"internally_reserved":0,"allocated":148824064,"data_stored":148665086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,8,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":436144,"kb_used_data":138792,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93935696,"statfs":{"total":96636764160,"available":96190152704,"internally_reserved":0,"allocated":142123008,"data_stored":141964030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.8179999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7970000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.8470000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313976,"kb_used_data":12016,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94057864,"statfs":{"total":96636764160,"available":96315252736,"internally_reserved":0,"allocated":12304384,"data_stored":12145406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,2,1,3,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.351}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.724}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.246}]}]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450636,"kb_used_data":140356,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93921204,"statfs":{"total":96636764160,"available":96175312896,"internally_reserved":0,"allocated":143724544,"data_stored":143565566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4470000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143704064,"data_stored":143704064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13099008,"data_stored":13099008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143446016,"data_stored":143446016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149467136,"data_stored":149467136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:56.614 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:57.145 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:57.146 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:57.172 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-09-18T11:44:55.834136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":292382068,"num_objects":35696,"num_object_clones":0,"num_object_copies":142784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35723,"num_write_kb":285666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35750,"ondisk_log_size":35750,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1660596,"kb_used_data":436500,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375826764,"statfs":{"total":386547056640,"available":384846606336,"internally_reserved":0,"allocated":446976000,"data_stored":446340088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,6,8,16,18,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34283520,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.726616"},"pg_stats":[{"pgid":"1.f","version":"31'2331","reported_seq":2372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.237048+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:44:54.237048+0000","last_peered":"2024-09-18T11:44:54.237048+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:44:54.237048+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2331,"log_dups_size":0,"ondisk_log_size":2331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19095552,"num_objects":2331,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2331,"num_write_kb":18648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2201","reported_seq":2263,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.933531+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:44:50.933531+0000","last_peered":"2024-09-18T11:44:50.933531+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:44:50.933531+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2201,"log_dups_size":0,"ondisk_log_size":2201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17994998,"num_objects":2197,"num_object_clones":0,"num_object_copies":8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2199,"num_write_kb":17588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2148","reported_seq":2229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.817206+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:44:50.817206+0000","last_peered":"2024-09-18T11:44:50.817206+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:44:50.817206+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17570427,"num_objects":2145,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":17166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":91}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2234","reported_seq":2296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.932777+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:44:50.932777+0000","last_peered":"2024-09-18T11:44:50.932777+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:44:50.932777+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18265334,"num_objects":2230,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":17845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2272","reported_seq":2325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.216716+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:44:54.216716+0000","last_peered":"2024-09-18T11:44:54.216716+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:44:54.216716+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2272,"log_dups_size":0,"ondisk_log_size":2272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584822,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2270,"num_write_kb":18157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2339","reported_seq":2382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.253670+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:44:54.253670+0000","last_peered":"2024-09-18T11:44:54.253670+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:44:54.253670+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19161088,"num_objects":2339,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2339,"num_write_kb":18712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2226","reported_seq":2291,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.964518+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:44:50.964518+0000","last_peered":"2024-09-18T11:44:50.964518+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:44:50.964518+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183414,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":17772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":86}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2146","reported_seq":2209,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.993266+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:44:50.993266+0000","last_peered":"2024-09-18T11:44:50.993266+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:44:50.993266+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2146,"log_dups_size":0,"ondisk_log_size":2146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17536246,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":17140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2361","reported_seq":2419,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.258888+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:44:54.258888+0000","last_peered":"2024-09-18T11:44:54.258888+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:44:54.258888+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2361,"log_dups_size":0,"ondisk_log_size":2361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19270124,"num_objects":2353,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2358,"num_write_kb":18841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2073","reported_seq":2157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.700123+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:44:50.700123+0000","last_peered":"2024-09-18T11:44:50.700123+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:44:50.700123+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16962806,"num_objects":2071,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2071,"num_write_kb":16566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":96}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2269","reported_seq":2326,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.144164+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:44:54.144164+0000","last_peered":"2024-09-18T11:44:54.144164+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:44:54.144164+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2269,"log_dups_size":0,"ondisk_log_size":2269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18542449,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2266,"num_write_kb":18116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2307","reported_seq":2344,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.162973+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:44:54.162973+0000","last_peered":"2024-09-18T11:44:54.162973+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:44:54.162973+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2307,"log_dups_size":0,"ondisk_log_size":2307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18889339,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2306,"num_write_kb":18447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":197}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2159","reported_seq":2234,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.821068+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:44:50.821068+0000","last_peered":"2024-09-18T11:44:50.821068+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:44:50.821068+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2159,"log_dups_size":0,"ondisk_log_size":2159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17676923,"num_objects":2158,"num_object_clones":0,"num_object_copies":8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2158,"num_write_kb":17263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2371","reported_seq":2420,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.180734+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:44:54.180734+0000","last_peered":"2024-09-18T11:44:54.180734+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:44:54.180734+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2371,"log_dups_size":0,"ondisk_log_size":2371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387638,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2369,"num_write_kb":18948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":209}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2195","reported_seq":2398,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.845551+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:44:50.845551+0000","last_peered":"2024-09-18T11:44:50.845551+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:44:50.845551+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2195,"log_dups_size":0,"ondisk_log_size":2195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17954038,"num_objects":2192,"num_object_clones":0,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2193,"num_write_kb":17541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2118","reported_seq":2180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:50.950841+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:44:50.950841+0000","last_peered":"2024-09-18T11:44:50.950841+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:44:50.950841+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2118,"log_dups_size":0,"ondisk_log_size":2118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17306870,"num_objects":2113,"num_object_clones":0,"num_object_copies":8452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2116,"num_write_kb":16916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":292382068,"num_objects":35696,"num_object_clones":0,"num_object_copies":142784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35723,"num_write_kb":285666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":449716224,"data_stored":449716224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35750,"ondisk_log_size":35750,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459840,"kb_used_data":145336,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93912000,"statfs":{"total":96636764160,"available":96165888000,"internally_reserved":0,"allocated":148824064,"data_stored":148665086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,8,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":436144,"kb_used_data":138792,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93935696,"statfs":{"total":96636764160,"available":96190152704,"internally_reserved":0,"allocated":142123008,"data_stored":141964030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.8179999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7970000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.8470000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313976,"kb_used_data":12016,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94057864,"statfs":{"total":96636764160,"available":96315252736,"internally_reserved":0,"allocated":12304384,"data_stored":12145406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,2,1,3,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.351}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.724}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.246}]}]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450636,"kb_used_data":140356,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93921204,"statfs":{"total":96636764160,"available":96175312896,"internally_reserved":0,"allocated":143724544,"data_stored":143565566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4470000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143704064,"data_stored":143704064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13099008,"data_stored":13099008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143446016,"data_stored":143446016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149467136,"data_stored":149467136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:57.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:44:57.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2331", 'reported_seq': 2372, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:54.237048+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:44:54.237048+0000', 'last_peered': '2024-09-18T11:44:54.237048+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:44:54.237048+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2331, 'log_dups_size': 0, 'ondisk_log_size': 2331, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19095552, 'num_objects': 2331, 'num_object_clones': 0, 'num_object_copies': 9324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2331, 'num_write_kb': 18648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 201}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:57.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:44:57.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2201", 'reported_seq': 2263, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:50.933531+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:44:50.933531+0000', 'last_peered': '2024-09-18T11:44:50.933531+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:44:50.933531+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2201, 'log_dups_size': 0, 'ondisk_log_size': 2201, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17994998, 'num_objects': 2197, 'num_object_clones': 0, 'num_object_copies': 8788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2199, 'num_write_kb': 17588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 95}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:57.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:44:57.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2148", 'reported_seq': 2229, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:50.817206+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:44:50.817206+0000', 'last_peered': '2024-09-18T11:44:50.817206+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:44:50.817206+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2148, 'log_dups_size': 0, 'ondisk_log_size': 2148, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17570427, 'num_objects': 2145, 'num_object_clones': 0, 'num_object_copies': 8580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2147, 'num_write_kb': 17166, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 91}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:57.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:44:57.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2234", 'reported_seq': 2296, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:50.932777+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:44:50.932777+0000', 'last_peered': '2024-09-18T11:44:50.932777+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:44:50.932777+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2234, 'log_dups_size': 0, 'ondisk_log_size': 2234, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18265334, 'num_objects': 2230, 'num_object_clones': 0, 'num_object_copies': 8920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2232, 'num_write_kb': 17845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 116}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:57.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:44:57.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2272", 'reported_seq': 2325, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:54.216716+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:44:54.216716+0000', 'last_peered': '2024-09-18T11:44:54.216716+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:44:54.216716+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2272, 'log_dups_size': 0, 'ondisk_log_size': 2272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18584822, 'num_objects': 2269, 'num_object_clones': 0, 'num_object_copies': 9076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2270, 'num_write_kb': 18157, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 187}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:57.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:44:57.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2339", 'reported_seq': 2382, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:54.253670+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:44:54.253670+0000', 'last_peered': '2024-09-18T11:44:54.253670+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:44:54.253670+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2339, 'log_dups_size': 0, 'ondisk_log_size': 2339, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19161088, 'num_objects': 2339, 'num_object_clones': 0, 'num_object_copies': 9356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2339, 'num_write_kb': 18712, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 185}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:57.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:44:57.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2226", 'reported_seq': 2291, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:50.964518+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:44:50.964518+0000', 'last_peered': '2024-09-18T11:44:50.964518+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:44:50.964518+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2226, 'log_dups_size': 0, 'ondisk_log_size': 2226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18183414, 'num_objects': 2220, 'num_object_clones': 0, 'num_object_copies': 8880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2224, 'num_write_kb': 17772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 86}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:57.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:44:57.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2146", 'reported_seq': 2209, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:50.993266+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:44:50.993266+0000', 'last_peered': '2024-09-18T11:44:50.993266+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:44:50.993266+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2146, 'log_dups_size': 0, 'ondisk_log_size': 2146, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17536246, 'num_objects': 2141, 'num_object_clones': 0, 'num_object_copies': 8564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2144, 'num_write_kb': 17140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 107}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:57.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:44:57.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2361", 'reported_seq': 2419, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:54.258888+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:44:54.258888+0000', 'last_peered': '2024-09-18T11:44:54.258888+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:44:54.258888+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2361, 'log_dups_size': 0, 'ondisk_log_size': 2361, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19270124, 'num_objects': 2353, 'num_object_clones': 0, 'num_object_copies': 9412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2358, 'num_write_kb': 18841, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 218}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:57.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:44:57.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2073", 'reported_seq': 2157, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:50.700123+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:44:50.700123+0000', 'last_peered': '2024-09-18T11:44:50.700123+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:44:50.700123+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2073, 'log_dups_size': 0, 'ondisk_log_size': 2073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16962806, 'num_objects': 2071, 'num_object_clones': 0, 'num_object_copies': 8284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2071, 'num_write_kb': 16566, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 96}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:57.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:44:57.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2269", 'reported_seq': 2326, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:54.144164+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:44:54.144164+0000', 'last_peered': '2024-09-18T11:44:54.144164+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:44:54.144164+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2269, 'log_dups_size': 0, 'ondisk_log_size': 2269, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18542449, 'num_objects': 2264, 'num_object_clones': 0, 'num_object_copies': 9056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2266, 'num_write_kb': 18116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 201}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:57.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:44:57.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2307", 'reported_seq': 2344, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:54.162973+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:44:54.162973+0000', 'last_peered': '2024-09-18T11:44:54.162973+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:44:54.162973+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2307, 'log_dups_size': 0, 'ondisk_log_size': 2307, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18889339, 'num_objects': 2306, 'num_object_clones': 0, 'num_object_copies': 9224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2306, 'num_write_kb': 18447, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 197}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:57.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:44:57.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2159", 'reported_seq': 2234, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:50.821068+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:44:50.821068+0000', 'last_peered': '2024-09-18T11:44:50.821068+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:44:50.821068+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2159, 'log_dups_size': 0, 'ondisk_log_size': 2159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17676923, 'num_objects': 2158, 'num_object_clones': 0, 'num_object_copies': 8632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2158, 'num_write_kb': 17263, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:57.176 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:44:57.176 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2371", 'reported_seq': 2420, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:54.180734+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:44:54.180734+0000', 'last_peered': '2024-09-18T11:44:54.180734+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:44:54.180734+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2371, 'log_dups_size': 0, 'ondisk_log_size': 2371, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19387638, 'num_objects': 2367, 'num_object_clones': 0, 'num_object_copies': 9468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2369, 'num_write_kb': 18948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 209}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:44:57.176 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:44:57.176 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2195", 'reported_seq': 2398, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:50.845551+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:44:50.845551+0000', 'last_peered': '2024-09-18T11:44:50.845551+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:44:50.845551+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2195, 'log_dups_size': 0, 'ondisk_log_size': 2195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17954038, 'num_objects': 2192, 'num_object_clones': 0, 'num_object_copies': 8768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2031, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2193, 'num_write_kb': 17541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 491520, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:44:57.176 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:44:57.176 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2118", 'reported_seq': 2180, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:50.950841+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:44:50.950841+0000', 'last_peered': '2024-09-18T11:44:50.950841+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:44:50.950841+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2118, 'log_dups_size': 0, 'ondisk_log_size': 2118, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17306870, 'num_objects': 2113, 'num_object_clones': 0, 'num_object_copies': 8452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2116, 'num_write_kb': 16916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 100}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:44:57.176 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:44:57.212 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:44:57.211979+0000 min lat: 0.00272854 max lat: 2.42146 avg lat: 0.0211474 2024-09-18T11:44:57.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:44:57.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 40 20 37831 37811 7.38359 3.23438 0.049368 0.0211474 2024-09-18T11:44:57.831 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:57.831 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33367/142784 objects degraded (23.369%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":95,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35696,"data_bytes":292382068,"bytes_used":1700450304,"bytes_avail":384846606336,"bytes_total":386547056640,"degraded_objects":33367,"degraded_total":142784,"degraded_ratio":0.23368864858807709,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":50533,"recovering_keys_per_sec":0,"num_objects_recovered":60,"num_bytes_recovered":491520,"num_keys_recovered":0,"write_bytes_sec":3524711,"read_op_per_sec":0,"write_op_per_sec":430},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-18T11:44:31.826669+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:44:57.864 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:44:57.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:58.212 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 41 20 38253 38233 7.28391 3.29688 0.0390245 0.0214408 2024-09-18T11:44:58.443 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:44:58.447 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:44:58.472 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-09-18T11:44:57.834624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":302974324,"num_objects":36989,"num_object_clones":0,"num_object_copies":147956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37016,"num_write_kb":296010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37043,"ondisk_log_size":37043,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1674396,"kb_used_data":454460,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375812964,"statfs":{"total":386547056640,"available":384832475136,"internally_reserved":0,"allocated":465367040,"data_stored":464731128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,9,18,17,20,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44875776,"num_objects":5478,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5478,"num_write_kb":43824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.718903"},"pg_stats":[{"pgid":"1.f","version":"31'2331","reported_seq":2372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.237048+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:44:54.237048+0000","last_peered":"2024-09-18T11:44:54.237048+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:44:54.237048+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2331,"log_dups_size":0,"ondisk_log_size":2331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19095552,"num_objects":2331,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2331,"num_write_kb":18648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2371","reported_seq":2434,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:55.954882+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:44:55.954882+0000","last_peered":"2024-09-18T11:44:55.954882+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:44:55.954882+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2371,"log_dups_size":0,"ondisk_log_size":2371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387638,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2369,"num_write_kb":18948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2281","reported_seq":2363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:55.812915+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:44:55.812915+0000","last_peered":"2024-09-18T11:44:55.812915+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:44:55.812915+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2281,"log_dups_size":0,"ondisk_log_size":2281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18659963,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2280,"num_write_kb":18230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2366","reported_seq":2428,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:55.905271+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:44:55.905271+0000","last_peered":"2024-09-18T11:44:55.905271+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:44:55.905271+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2366,"log_dups_size":0,"ondisk_log_size":2366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19346678,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2364,"num_write_kb":18901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2272","reported_seq":2325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.216716+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:44:54.216716+0000","last_peered":"2024-09-18T11:44:54.216716+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:44:54.216716+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2272,"log_dups_size":0,"ondisk_log_size":2272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584822,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2270,"num_write_kb":18157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2339","reported_seq":2382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.253670+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:44:54.253670+0000","last_peered":"2024-09-18T11:44:54.253670+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:44:54.253670+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19161088,"num_objects":2339,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2339,"num_write_kb":18712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2381","reported_seq":2446,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:55.968773+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:44:55.968773+0000","last_peered":"2024-09-18T11:44:55.968773+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:44:55.968773+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2381,"log_dups_size":0,"ondisk_log_size":2381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19453174,"num_objects":2375,"num_object_clones":0,"num_object_copies":9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":19012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":241}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2270","reported_seq":2333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:55.990558+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:44:55.990558+0000","last_peered":"2024-09-18T11:44:55.990558+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:44:55.990558+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2270,"log_dups_size":0,"ondisk_log_size":2270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18552054,"num_objects":2265,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2268,"num_write_kb":18132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2361","reported_seq":2419,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.258888+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:44:54.258888+0000","last_peered":"2024-09-18T11:44:54.258888+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:44:54.258888+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2361,"log_dups_size":0,"ondisk_log_size":2361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19270124,"num_objects":2353,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2358,"num_write_kb":18841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2221","reported_seq":2305,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:55.867511+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:44:55.867511+0000","last_peered":"2024-09-18T11:44:55.867511+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:44:55.867511+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2221,"log_dups_size":0,"ondisk_log_size":2221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18175222,"num_objects":2219,"num_object_clones":0,"num_object_copies":8876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2219,"num_write_kb":17750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":244}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2269","reported_seq":2326,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.144164+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:44:54.144164+0000","last_peered":"2024-09-18T11:44:54.144164+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:44:54.144164+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2269,"log_dups_size":0,"ondisk_log_size":2269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18542449,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2266,"num_write_kb":18116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2307","reported_seq":2344,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.162973+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:44:54.162973+0000","last_peered":"2024-09-18T11:44:54.162973+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:44:54.162973+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2307,"log_dups_size":0,"ondisk_log_size":2307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18889339,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2306,"num_write_kb":18447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":197}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2306","reported_seq":2382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:55.841370+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:44:55.841370+0000","last_peered":"2024-09-18T11:44:55.841370+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:44:55.841370+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18881147,"num_objects":2305,"num_object_clones":0,"num_object_copies":9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2305,"num_write_kb":18439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":251}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2371","reported_seq":2420,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:54.180734+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:44:54.180734+0000","last_peered":"2024-09-18T11:44:54.180734+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:44:54.180734+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2371,"log_dups_size":0,"ondisk_log_size":2371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387638,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2369,"num_write_kb":18948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":209}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2348","reported_seq":2689,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:55.832456+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:44:55.832456+0000","last_peered":"2024-09-18T11:44:55.832456+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:44:55.832456+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2348,"log_dups_size":0,"ondisk_log_size":2348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19207414,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2346,"num_write_kb":18765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2249","reported_seq":2311,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:55.935409+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:44:55.935409+0000","last_peered":"2024-09-18T11:44:55.935409+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:44:55.935409+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2249,"log_dups_size":0,"ondisk_log_size":2249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18380022,"num_objects":2244,"num_object_clones":0,"num_object_copies":8976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2247,"num_write_kb":17964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":302974324,"num_objects":36989,"num_object_clones":0,"num_object_copies":147956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37016,"num_write_kb":296010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":468168704,"data_stored":468168704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37043,"ondisk_log_size":37043,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459840,"kb_used_data":145336,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93912000,"statfs":{"total":96636764160,"available":96165888000,"internally_reserved":0,"allocated":148824064,"data_stored":148665086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,8,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449328,"kb_used_data":147816,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93922512,"statfs":{"total":96636764160,"available":96176652288,"internally_reserved":0,"allocated":151363584,"data_stored":151204606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,10,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4720000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0119999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5010000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313976,"kb_used_data":12016,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94057864,"statfs":{"total":96636764160,"available":96315252736,"internally_reserved":0,"allocated":12304384,"data_stored":12145406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,2,1,3,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.351}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.724}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.246}]}]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451252,"kb_used_data":149292,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93920588,"statfs":{"total":96636764160,"available":96174682112,"internally_reserved":0,"allocated":152875008,"data_stored":152716030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,4,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.266}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2949999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":152956928,"data_stored":152956928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13099008,"data_stored":13099008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":152645632,"data_stored":152645632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149467136,"data_stored":149467136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:59.212 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 42 20 38696 38676 7.19289 3.46094 0.0315718 0.0217042 2024-09-18T11:45:00.212 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 43 20 39158 39138 7.10955 3.60938 0.0595437 0.0219631 2024-09-18T11:45:01.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 44 20 39611 39591 7.02841 3.53906 0.0471771 0.0222139 2024-09-18T11:45:01.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:02.014 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:02.015 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:02.037 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-18T11:45:01.836016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":320906612,"num_objects":39178,"num_object_clones":0,"num_object_copies":156712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39205,"num_write_kb":313522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39232,"ondisk_log_size":39232,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1714132,"kb_used_data":490036,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375773228,"statfs":{"total":386547056640,"available":384791785472,"internally_reserved":0,"allocated":501796864,"data_stored":501160952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[14,2,13,3,13,19,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":56459264,"num_objects":6892,"num_object_clones":0,"num_object_copies":27568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6892,"num_write_kb":55136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003693"},"pg_stats":[{"pgid":"1.f","version":"31'2477","reported_seq":2517,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.184743+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:44:59.184743+0000","last_peered":"2024-09-18T11:44:59.184743+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:44:59.184743+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20291584,"num_objects":2477,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2477,"num_write_kb":19816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2509","reported_seq":2572,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.008731+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:01.008731+0000","last_peered":"2024-09-18T11:45:01.008731+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:01.008731+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2509,"log_dups_size":0,"ondisk_log_size":2509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20518134,"num_objects":2505,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":20052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2434","reported_seq":2515,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.879701+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:00.879701+0000","last_peered":"2024-09-18T11:45:00.879701+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:00.879701+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2434,"log_dups_size":0,"ondisk_log_size":2434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19913339,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2433,"num_write_kb":19454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2495","reported_seq":2557,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.851521+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:00.851521+0000","last_peered":"2024-09-18T11:45:00.851521+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:00.851521+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2495,"log_dups_size":0,"ondisk_log_size":2495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20403446,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":377}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2412","reported_seq":2467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.223446+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:44:59.223446+0000","last_peered":"2024-09-18T11:44:59.223446+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:44:59.223446+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2412,"log_dups_size":0,"ondisk_log_size":2412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19731702,"num_objects":2409,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":19277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":327}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2452","reported_seq":2495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.169671+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:44:59.169671+0000","last_peered":"2024-09-18T11:44:59.169671+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:44:59.169671+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2523","reported_seq":2587,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.012019+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:01.012019+0000","last_peered":"2024-09-18T11:45:01.012019+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:01.012019+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20616438,"num_objects":2517,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":20148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":383}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2397","reported_seq":2460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.945101+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:00.945101+0000","last_peered":"2024-09-18T11:45:00.945101+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:00.945101+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2397,"log_dups_size":0,"ondisk_log_size":2397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19592438,"num_objects":2392,"num_object_clones":0,"num_object_copies":9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2395,"num_write_kb":19148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2489","reported_seq":2547,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.237725+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:44:59.237725+0000","last_peered":"2024-09-18T11:44:59.237725+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:44:59.237725+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2489,"log_dups_size":0,"ondisk_log_size":2489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20318700,"num_objects":2481,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":19865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2357","reported_seq":2440,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.863066+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:00.863066+0000","last_peered":"2024-09-18T11:45:00.863066+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:00.863066+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2357,"log_dups_size":0,"ondisk_log_size":2357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19289334,"num_objects":2355,"num_object_clones":0,"num_object_copies":9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":18838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2424","reported_seq":2481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.187420+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:44:59.187420+0000","last_peered":"2024-09-18T11:44:59.187420+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:44:59.187420+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2424,"log_dups_size":0,"ondisk_log_size":2424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19812209,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2421,"num_write_kb":19356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2450","reported_seq":2487,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.265066+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:44:59.265066+0000","last_peered":"2024-09-18T11:44:59.265066+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:44:59.265066+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2450,"log_dups_size":0,"ondisk_log_size":2450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20060795,"num_objects":2449,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":19591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2452","reported_seq":2528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.842106+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:00.842106+0000","last_peered":"2024-09-18T11:45:00.842106+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:00.842106+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20077179,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2451,"num_write_kb":19607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":397}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2497","reported_seq":2546,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.212503+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:44:59.212503+0000","last_peered":"2024-09-18T11:44:59.212503+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:44:59.212503+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2497,"log_dups_size":0,"ondisk_log_size":2497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20419830,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":19956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":335}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2482","reported_seq":2973,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.883406+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:00.883406+0000","last_peered":"2024-09-18T11:45:00.883406+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:00.883406+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2482,"log_dups_size":0,"ondisk_log_size":2482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20305142,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2480,"num_write_kb":19837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2382","reported_seq":2443,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.004860+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:01.004860+0000","last_peered":"2024-09-18T11:45:01.004860+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:01.004860+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2382,"log_dups_size":0,"ondisk_log_size":2382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19469558,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2380,"num_write_kb":19028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":320906612,"num_objects":39178,"num_object_clones":0,"num_object_copies":156712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39205,"num_write_kb":313522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":504725504,"data_stored":504725504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39232,"ondisk_log_size":39232,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456160,"kb_used_data":154136,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93915680,"statfs":{"total":96636764160,"available":96169656320,"internally_reserved":0,"allocated":157835264,"data_stored":157676286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,0,0,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]}]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462304,"kb_used_data":156632,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93909536,"statfs":{"total":96636764160,"available":96163364864,"internally_reserved":0,"allocated":160391168,"data_stored":160232190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":9.032},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":6.5519999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5229999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331352,"kb_used_data":21072,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94040488,"statfs":{"total":96636764160,"available":96297459712,"internally_reserved":0,"allocated":21577728,"data_stored":21418750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9119999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5810000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.024}]}]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464316,"kb_used_data":158196,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93907524,"statfs":{"total":96636764160,"available":96161304576,"internally_reserved":0,"allocated":161992704,"data_stored":161833726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,7,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.898999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.41999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161923072,"data_stored":161923072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22536192,"data_stored":22536192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161619968,"data_stored":161619968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":158646272,"data_stored":158646272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:02.038 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:02.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 45 20 40022 40002 6.94358 3.21094 0.0382463 0.0224846 2024-09-18T11:45:02.569 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:02.569 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:02.590 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-18T11:45:01.836016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":320906612,"num_objects":39178,"num_object_clones":0,"num_object_copies":156712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39205,"num_write_kb":313522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39232,"ondisk_log_size":39232,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1714132,"kb_used_data":490036,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375773228,"statfs":{"total":386547056640,"available":384791785472,"internally_reserved":0,"allocated":501796864,"data_stored":501160952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[14,2,13,3,13,19,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":56459264,"num_objects":6892,"num_object_clones":0,"num_object_copies":27568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6892,"num_write_kb":55136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003693"},"pg_stats":[{"pgid":"1.f","version":"31'2477","reported_seq":2517,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.184743+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:44:59.184743+0000","last_peered":"2024-09-18T11:44:59.184743+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:44:59.184743+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20291584,"num_objects":2477,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2477,"num_write_kb":19816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2509","reported_seq":2572,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.008731+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:01.008731+0000","last_peered":"2024-09-18T11:45:01.008731+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:01.008731+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2509,"log_dups_size":0,"ondisk_log_size":2509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20518134,"num_objects":2505,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":20052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2434","reported_seq":2515,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.879701+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:00.879701+0000","last_peered":"2024-09-18T11:45:00.879701+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:00.879701+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2434,"log_dups_size":0,"ondisk_log_size":2434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19913339,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2433,"num_write_kb":19454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2495","reported_seq":2557,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.851521+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:00.851521+0000","last_peered":"2024-09-18T11:45:00.851521+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:00.851521+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2495,"log_dups_size":0,"ondisk_log_size":2495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20403446,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":377}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2412","reported_seq":2467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.223446+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:44:59.223446+0000","last_peered":"2024-09-18T11:44:59.223446+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:44:59.223446+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2412,"log_dups_size":0,"ondisk_log_size":2412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19731702,"num_objects":2409,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":19277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":327}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2452","reported_seq":2495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.169671+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:44:59.169671+0000","last_peered":"2024-09-18T11:44:59.169671+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:44:59.169671+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2523","reported_seq":2587,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.012019+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:01.012019+0000","last_peered":"2024-09-18T11:45:01.012019+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:01.012019+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20616438,"num_objects":2517,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":20148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":383}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2397","reported_seq":2460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.945101+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:00.945101+0000","last_peered":"2024-09-18T11:45:00.945101+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:00.945101+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2397,"log_dups_size":0,"ondisk_log_size":2397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19592438,"num_objects":2392,"num_object_clones":0,"num_object_copies":9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2395,"num_write_kb":19148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2489","reported_seq":2547,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.237725+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:44:59.237725+0000","last_peered":"2024-09-18T11:44:59.237725+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:44:59.237725+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2489,"log_dups_size":0,"ondisk_log_size":2489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20318700,"num_objects":2481,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":19865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2357","reported_seq":2440,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.863066+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:00.863066+0000","last_peered":"2024-09-18T11:45:00.863066+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:00.863066+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2357,"log_dups_size":0,"ondisk_log_size":2357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19289334,"num_objects":2355,"num_object_clones":0,"num_object_copies":9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":18838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2424","reported_seq":2481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.187420+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:44:59.187420+0000","last_peered":"2024-09-18T11:44:59.187420+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:44:59.187420+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2424,"log_dups_size":0,"ondisk_log_size":2424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19812209,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2421,"num_write_kb":19356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2450","reported_seq":2487,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.265066+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:44:59.265066+0000","last_peered":"2024-09-18T11:44:59.265066+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:44:59.265066+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2450,"log_dups_size":0,"ondisk_log_size":2450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20060795,"num_objects":2449,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":19591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2452","reported_seq":2528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.842106+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:00.842106+0000","last_peered":"2024-09-18T11:45:00.842106+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:00.842106+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20077179,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2451,"num_write_kb":19607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":397}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2497","reported_seq":2546,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.212503+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:44:59.212503+0000","last_peered":"2024-09-18T11:44:59.212503+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:44:59.212503+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2497,"log_dups_size":0,"ondisk_log_size":2497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20419830,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":19956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":335}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2482","reported_seq":2973,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.883406+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:00.883406+0000","last_peered":"2024-09-18T11:45:00.883406+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:00.883406+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2482,"log_dups_size":0,"ondisk_log_size":2482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20305142,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2480,"num_write_kb":19837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2382","reported_seq":2443,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.004860+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:01.004860+0000","last_peered":"2024-09-18T11:45:01.004860+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:01.004860+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2382,"log_dups_size":0,"ondisk_log_size":2382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19469558,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2380,"num_write_kb":19028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":320906612,"num_objects":39178,"num_object_clones":0,"num_object_copies":156712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39205,"num_write_kb":313522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":504725504,"data_stored":504725504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39232,"ondisk_log_size":39232,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456160,"kb_used_data":154136,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93915680,"statfs":{"total":96636764160,"available":96169656320,"internally_reserved":0,"allocated":157835264,"data_stored":157676286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,0,0,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]}]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462304,"kb_used_data":156632,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93909536,"statfs":{"total":96636764160,"available":96163364864,"internally_reserved":0,"allocated":160391168,"data_stored":160232190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":9.032},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":6.5519999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5229999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331352,"kb_used_data":21072,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94040488,"statfs":{"total":96636764160,"available":96297459712,"internally_reserved":0,"allocated":21577728,"data_stored":21418750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9119999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5810000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.024}]}]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464316,"kb_used_data":158196,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93907524,"statfs":{"total":96636764160,"available":96161304576,"internally_reserved":0,"allocated":161992704,"data_stored":161833726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,7,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.898999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.41999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161923072,"data_stored":161923072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22536192,"data_stored":22536192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161619968,"data_stored":161619968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":158646272,"data_stored":158646272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:02.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:02.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2477", 'reported_seq': 2517, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:59.184743+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:44:59.184743+0000', 'last_peered': '2024-09-18T11:44:59.184743+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:44:59.184743+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2477, 'log_dups_size': 0, 'ondisk_log_size': 2477, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20291584, 'num_objects': 2477, 'num_object_clones': 0, 'num_object_copies': 9908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2477, 'num_write_kb': 19816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 347}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:02.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:02.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2509", 'reported_seq': 2572, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:01.008731+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:01.008731+0000', 'last_peered': '2024-09-18T11:45:01.008731+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:01.008731+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2509, 'log_dups_size': 0, 'ondisk_log_size': 2509, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20518134, 'num_objects': 2505, 'num_object_clones': 0, 'num_object_copies': 10020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2505, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2507, 'num_write_kb': 20052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:02.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:02.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2434", 'reported_seq': 2515, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:00.879701+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:00.879701+0000', 'last_peered': '2024-09-18T11:45:00.879701+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:00.879701+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2434, 'log_dups_size': 0, 'ondisk_log_size': 2434, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19913339, 'num_objects': 2431, 'num_object_clones': 0, 'num_object_copies': 9724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2433, 'num_write_kb': 19454, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 377}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:02.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:02.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2495", 'reported_seq': 2557, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:00.851521+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:00.851521+0000', 'last_peered': '2024-09-18T11:45:00.851521+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:00.851521+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2495, 'log_dups_size': 0, 'ondisk_log_size': 2495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20403446, 'num_objects': 2491, 'num_object_clones': 0, 'num_object_copies': 9964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2493, 'num_write_kb': 19933, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 377}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:02.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:02.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2412", 'reported_seq': 2467, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:59.223446+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:44:59.223446+0000', 'last_peered': '2024-09-18T11:44:59.223446+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:44:59.223446+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2412, 'log_dups_size': 0, 'ondisk_log_size': 2412, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19731702, 'num_objects': 2409, 'num_object_clones': 0, 'num_object_copies': 9636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2410, 'num_write_kb': 19277, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 327}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:02.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:02.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2452", 'reported_seq': 2495, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:59.169671+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:44:59.169671+0000', 'last_peered': '2024-09-18T11:44:59.169671+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:44:59.169671+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2452, 'log_dups_size': 0, 'ondisk_log_size': 2452, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20086784, 'num_objects': 2452, 'num_object_clones': 0, 'num_object_copies': 9808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2452, 'num_write_kb': 19616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 298}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:02.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:02.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2523", 'reported_seq': 2587, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:01.012019+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:01.012019+0000', 'last_peered': '2024-09-18T11:45:01.012019+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:01.012019+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2523, 'log_dups_size': 0, 'ondisk_log_size': 2523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20616438, 'num_objects': 2517, 'num_object_clones': 0, 'num_object_copies': 10068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2521, 'num_write_kb': 20148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 383}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:02.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:02.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2397", 'reported_seq': 2460, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:00.945101+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:00.945101+0000', 'last_peered': '2024-09-18T11:45:00.945101+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:00.945101+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2397, 'log_dups_size': 0, 'ondisk_log_size': 2397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19592438, 'num_objects': 2392, 'num_object_clones': 0, 'num_object_copies': 9568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2395, 'num_write_kb': 19148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:02.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:02.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2489", 'reported_seq': 2547, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:59.237725+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:44:59.237725+0000', 'last_peered': '2024-09-18T11:44:59.237725+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:44:59.237725+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2489, 'log_dups_size': 0, 'ondisk_log_size': 2489, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20318700, 'num_objects': 2481, 'num_object_clones': 0, 'num_object_copies': 9924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2486, 'num_write_kb': 19865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 346}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:02.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:02.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2357", 'reported_seq': 2440, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:00.863066+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:00.863066+0000', 'last_peered': '2024-09-18T11:45:00.863066+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:00.863066+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2357, 'log_dups_size': 0, 'ondisk_log_size': 2357, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19289334, 'num_objects': 2355, 'num_object_clones': 0, 'num_object_copies': 9420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2355, 'num_write_kb': 18838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 380}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:02.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:02.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2424", 'reported_seq': 2481, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:59.187420+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:44:59.187420+0000', 'last_peered': '2024-09-18T11:44:59.187420+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:44:59.187420+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2424, 'log_dups_size': 0, 'ondisk_log_size': 2424, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19812209, 'num_objects': 2419, 'num_object_clones': 0, 'num_object_copies': 9676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2421, 'num_write_kb': 19356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 356}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:02.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:02.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2450", 'reported_seq': 2487, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:59.265066+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:44:59.265066+0000', 'last_peered': '2024-09-18T11:44:59.265066+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:44:59.265066+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2450, 'log_dups_size': 0, 'ondisk_log_size': 2450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20060795, 'num_objects': 2449, 'num_object_clones': 0, 'num_object_copies': 9796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2449, 'num_write_kb': 19591, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:02.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:02.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2452", 'reported_seq': 2528, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:00.842106+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:00.842106+0000', 'last_peered': '2024-09-18T11:45:00.842106+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:00.842106+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2452, 'log_dups_size': 0, 'ondisk_log_size': 2452, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20077179, 'num_objects': 2451, 'num_object_clones': 0, 'num_object_copies': 9804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2451, 'num_write_kb': 19607, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 397}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:02.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:02.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2497", 'reported_seq': 2546, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:44:59.212503+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:44:59.212503+0000', 'last_peered': '2024-09-18T11:44:59.212503+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:44:59.212503+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2497, 'log_dups_size': 0, 'ondisk_log_size': 2497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20419830, 'num_objects': 2493, 'num_object_clones': 0, 'num_object_copies': 9972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2495, 'num_write_kb': 19956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:02.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:02.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2482", 'reported_seq': 2973, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:00.883406+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:00.883406+0000', 'last_peered': '2024-09-18T11:45:00.883406+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:00.883406+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2482, 'log_dups_size': 0, 'ondisk_log_size': 2482, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20305142, 'num_objects': 2479, 'num_object_clones': 0, 'num_object_copies': 9916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1887, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2480, 'num_write_kb': 19837, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 206, 'num_bytes_recovered': 1671168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:02.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:02.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2382", 'reported_seq': 2443, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:01.004860+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:01.004860+0000', 'last_peered': '2024-09-18T11:45:01.004860+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:01.004860+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2382, 'log_dups_size': 0, 'ondisk_log_size': 2382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19469558, 'num_objects': 2377, 'num_object_clones': 0, 'num_object_copies': 9508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2380, 'num_write_kb': 19028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 364}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:02.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:03.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 46 20 40421 40401 6.86039 3.11719 0.0448636 0.0227561 2024-09-18T11:45:03.267 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:03.267 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33223/156712 objects degraded (21.200%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":101,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39178,"data_bytes":320906612,"bytes_used":1755271168,"bytes_avail":384791785472,"bytes_total":386547056640,"degraded_objects":33223,"degraded_total":156712,"degraded_ratio":0.21200035734340703,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":139221,"recovering_keys_per_sec":0,"num_objects_recovered":204,"num_bytes_recovered":1671168,"num_keys_recovered":0,"write_bytes_sec":4703490,"read_op_per_sec":0,"write_op_per_sec":574},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T11:44:57.834722+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:45:03.296 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:03.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:03.897 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:03.897 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:03.919 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-18T11:45:03.836811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":320906612,"num_objects":39178,"num_object_clones":0,"num_object_copies":156712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39205,"num_write_kb":313522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39232,"ondisk_log_size":39232,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1714132,"kb_used_data":490036,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375773228,"statfs":{"total":386547056640,"available":384791785472,"internally_reserved":0,"allocated":501796864,"data_stored":501160952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[14,2,13,3,13,19,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36814848,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4494,"num_write_kb":35952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003942"},"pg_stats":[{"pgid":"1.f","version":"31'2477","reported_seq":2517,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.184743+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:44:59.184743+0000","last_peered":"2024-09-18T11:44:59.184743+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:44:59.184743+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20291584,"num_objects":2477,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2477,"num_write_kb":19816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2509","reported_seq":2572,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.008731+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:01.008731+0000","last_peered":"2024-09-18T11:45:01.008731+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:01.008731+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2509,"log_dups_size":0,"ondisk_log_size":2509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20518134,"num_objects":2505,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":20052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2434","reported_seq":2515,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.879701+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:00.879701+0000","last_peered":"2024-09-18T11:45:00.879701+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:00.879701+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2434,"log_dups_size":0,"ondisk_log_size":2434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19913339,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2433,"num_write_kb":19454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2495","reported_seq":2557,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.851521+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:00.851521+0000","last_peered":"2024-09-18T11:45:00.851521+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:00.851521+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2495,"log_dups_size":0,"ondisk_log_size":2495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20403446,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":377}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2412","reported_seq":2467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.223446+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:44:59.223446+0000","last_peered":"2024-09-18T11:44:59.223446+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:44:59.223446+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2412,"log_dups_size":0,"ondisk_log_size":2412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19731702,"num_objects":2409,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":19277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":327}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2452","reported_seq":2495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.169671+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:44:59.169671+0000","last_peered":"2024-09-18T11:44:59.169671+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:44:59.169671+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2523","reported_seq":2587,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.012019+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:01.012019+0000","last_peered":"2024-09-18T11:45:01.012019+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:01.012019+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20616438,"num_objects":2517,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":20148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":383}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2397","reported_seq":2460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.945101+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:00.945101+0000","last_peered":"2024-09-18T11:45:00.945101+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:00.945101+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2397,"log_dups_size":0,"ondisk_log_size":2397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19592438,"num_objects":2392,"num_object_clones":0,"num_object_copies":9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2395,"num_write_kb":19148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2489","reported_seq":2547,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.237725+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:44:59.237725+0000","last_peered":"2024-09-18T11:44:59.237725+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:44:59.237725+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2489,"log_dups_size":0,"ondisk_log_size":2489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20318700,"num_objects":2481,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":19865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2357","reported_seq":2440,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.863066+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:00.863066+0000","last_peered":"2024-09-18T11:45:00.863066+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:00.863066+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2357,"log_dups_size":0,"ondisk_log_size":2357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19289334,"num_objects":2355,"num_object_clones":0,"num_object_copies":9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":18838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2424","reported_seq":2481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.187420+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:44:59.187420+0000","last_peered":"2024-09-18T11:44:59.187420+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:44:59.187420+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2424,"log_dups_size":0,"ondisk_log_size":2424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19812209,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2421,"num_write_kb":19356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2450","reported_seq":2487,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.265066+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:44:59.265066+0000","last_peered":"2024-09-18T11:44:59.265066+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:44:59.265066+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2450,"log_dups_size":0,"ondisk_log_size":2450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20060795,"num_objects":2449,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":19591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2452","reported_seq":2528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.842106+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:00.842106+0000","last_peered":"2024-09-18T11:45:00.842106+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:00.842106+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20077179,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2451,"num_write_kb":19607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":397}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2497","reported_seq":2546,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:44:59.212503+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:44:59.212503+0000","last_peered":"2024-09-18T11:44:59.212503+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:44:59.212503+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2497,"log_dups_size":0,"ondisk_log_size":2497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20419830,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":19956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":335}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2482","reported_seq":2973,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.883406+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:00.883406+0000","last_peered":"2024-09-18T11:45:00.883406+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:00.883406+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2482,"log_dups_size":0,"ondisk_log_size":2482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20305142,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2480,"num_write_kb":19837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2382","reported_seq":2443,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.004860+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:01.004860+0000","last_peered":"2024-09-18T11:45:01.004860+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:01.004860+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2382,"log_dups_size":0,"ondisk_log_size":2382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19469558,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2380,"num_write_kb":19028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":320906612,"num_objects":39178,"num_object_clones":0,"num_object_copies":156712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39205,"num_write_kb":313522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":504725504,"data_stored":504725504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39232,"ondisk_log_size":39232,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456160,"kb_used_data":154136,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93915680,"statfs":{"total":96636764160,"available":96169656320,"internally_reserved":0,"allocated":157835264,"data_stored":157676286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,0,0,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]}]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462304,"kb_used_data":156632,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93909536,"statfs":{"total":96636764160,"available":96163364864,"internally_reserved":0,"allocated":160391168,"data_stored":160232190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":9.032},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":6.5519999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5229999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331352,"kb_used_data":21072,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94040488,"statfs":{"total":96636764160,"available":96297459712,"internally_reserved":0,"allocated":21577728,"data_stored":21418750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9119999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5810000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.024}]}]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464316,"kb_used_data":158196,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93907524,"statfs":{"total":96636764160,"available":96161304576,"internally_reserved":0,"allocated":161992704,"data_stored":161833726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,7,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.898999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.41999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161923072,"data_stored":161923072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22536192,"data_stored":22536192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161619968,"data_stored":161619968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":158646272,"data_stored":158646272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:04.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 47 20 40832 40812 6.78275 3.21094 0.0437459 0.0230196 2024-09-18T11:45:05.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 48 20 41291 41271 6.71615 3.58594 0.0264638 0.0232533 2024-09-18T11:45:06.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 49 20 41744 41724 6.65131 3.53906 0.0517458 0.0234795 2024-09-18T11:45:06.921 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:07.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 50 20 42177 42157 6.58594 3.38281 0.0404962 0.0237092 2024-09-18T11:45:07.489 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:07.489 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:07.513 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-18T11:45:05.837527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328828276,"num_objects":40145,"num_object_clones":0,"num_object_copies":160580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40172,"num_write_kb":321258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40199,"ondisk_log_size":40199,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1723364,"kb_used_data":507588,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":375763996,"statfs":{"total":386547056640,"available":384782331904,"internally_reserved":0,"allocated":519770112,"data_stored":519134200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[15,3,6,10,14,13,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44736512,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5461,"num_write_kb":43688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003914"},"pg_stats":[{"pgid":"1.f","version":"31'2629","reported_seq":2670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.278733+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:04.278733+0000","last_peered":"2024-09-18T11:45:04.278733+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:04.278733+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21536768,"num_objects":2629,"num_object_clones":0,"num_object_copies":10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":21032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2509","reported_seq":2572,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.008731+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:01.008731+0000","last_peered":"2024-09-18T11:45:01.008731+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:01.008731+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2509,"log_dups_size":0,"ondisk_log_size":2509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20518134,"num_objects":2505,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":20052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2434","reported_seq":2515,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.879701+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:00.879701+0000","last_peered":"2024-09-18T11:45:00.879701+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:00.879701+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2434,"log_dups_size":0,"ondisk_log_size":2434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19913339,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2433,"num_write_kb":19454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2495","reported_seq":2557,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.851521+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:00.851521+0000","last_peered":"2024-09-18T11:45:00.851521+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:00.851521+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2495,"log_dups_size":0,"ondisk_log_size":2495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20403446,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":377}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2550","reported_seq":2605,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.245450+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:04.245450+0000","last_peered":"2024-09-18T11:45:04.245450+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:04.245450+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20862198,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2548,"num_write_kb":20381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2585","reported_seq":2628,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.259048+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:04.259048+0000","last_peered":"2024-09-18T11:45:04.259048+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:04.259048+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2585,"log_dups_size":0,"ondisk_log_size":2585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21176320,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2585,"num_write_kb":20680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2523","reported_seq":2587,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.012019+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:01.012019+0000","last_peered":"2024-09-18T11:45:01.012019+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:01.012019+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20616438,"num_objects":2517,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":20148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":383}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2397","reported_seq":2460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.945101+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:00.945101+0000","last_peered":"2024-09-18T11:45:00.945101+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:00.945101+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2397,"log_dups_size":0,"ondisk_log_size":2397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19592438,"num_objects":2392,"num_object_clones":0,"num_object_copies":9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2395,"num_write_kb":19148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2638","reported_seq":2696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.155338+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:04.155338+0000","last_peered":"2024-09-18T11:45:04.155338+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:04.155338+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2638,"log_dups_size":0,"ondisk_log_size":2638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21539308,"num_objects":2630,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2635,"num_write_kb":21057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2357","reported_seq":2440,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.863066+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:00.863066+0000","last_peered":"2024-09-18T11:45:00.863066+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:00.863066+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2357,"log_dups_size":0,"ondisk_log_size":2357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19289334,"num_objects":2355,"num_object_clones":0,"num_object_copies":9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":18838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2546","reported_seq":2603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.205277+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:04.205277+0000","last_peered":"2024-09-18T11:45:04.205277+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:04.205277+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2546,"log_dups_size":0,"ondisk_log_size":2546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20811633,"num_objects":2541,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2543,"num_write_kb":20332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2590","reported_seq":2627,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.241609+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:04.241609+0000","last_peered":"2024-09-18T11:45:04.241609+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:04.241609+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21207675,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2589,"num_write_kb":20711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2452","reported_seq":2528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.842106+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:00.842106+0000","last_peered":"2024-09-18T11:45:00.842106+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:00.842106+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20077179,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2451,"num_write_kb":19607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":397}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2630","reported_seq":2679,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.233399+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:04.233399+0000","last_peered":"2024-09-18T11:45:04.233399+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:04.233399+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21509366,"num_objects":2626,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2628,"num_write_kb":21020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2482","reported_seq":2973,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:00.883406+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:00.883406+0000","last_peered":"2024-09-18T11:45:00.883406+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:00.883406+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2482,"log_dups_size":0,"ondisk_log_size":2482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20305142,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2480,"num_write_kb":19837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2382","reported_seq":2443,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:01.004860+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:01.004860+0000","last_peered":"2024-09-18T11:45:01.004860+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:01.004860+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2382,"log_dups_size":0,"ondisk_log_size":2382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19469558,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2380,"num_write_kb":19028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":328828276,"num_objects":40145,"num_object_clones":0,"num_object_copies":160580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40172,"num_write_kb":321258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":522563584,"data_stored":522563584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40199,"ondisk_log_size":40199,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468992,"kb_used_data":162808,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93902848,"statfs":{"total":96636764160,"available":96156516352,"internally_reserved":0,"allocated":166715392,"data_stored":166556414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,5,0,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9720000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4370000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462304,"kb_used_data":156632,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93909536,"statfs":{"total":96636764160,"available":96163364864,"internally_reserved":0,"allocated":160391168,"data_stored":160232190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":9.032},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":6.5519999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5229999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327752,"kb_used_data":29952,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94044088,"statfs":{"total":96636764160,"available":96301146112,"internally_reserved":0,"allocated":30670848,"data_stored":30511870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9789999999999992}]}]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464316,"kb_used_data":158196,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93907524,"statfs":{"total":96636764160,"available":96161304576,"internally_reserved":0,"allocated":161992704,"data_stored":161833726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,7,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.898999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.41999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161923072,"data_stored":161923072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31621120,"data_stored":31621120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161619968,"data_stored":161619968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167399424,"data_stored":167399424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:07.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:08.071 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:08.072 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:08.094 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-18T11:45:07.838586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":338486644,"num_objects":41324,"num_object_clones":0,"num_object_copies":165296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41351,"num_write_kb":330690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41378,"ondisk_log_size":41378,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1753008,"kb_used_data":524752,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375734352,"statfs":{"total":386547056640,"available":384751976448,"internally_reserved":0,"allocated":537346048,"data_stored":536710136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,3,4,14,9,7,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46104576,"num_objects":5628,"num_object_clones":0,"num_object_copies":22512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":45024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1753088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004449"},"pg_stats":[{"pgid":"1.f","version":"31'2629","reported_seq":2670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.278733+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:04.278733+0000","last_peered":"2024-09-18T11:45:04.278733+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:04.278733+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21536768,"num_objects":2629,"num_object_clones":0,"num_object_copies":10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":21032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2638","reported_seq":2701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:06.070335+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:06.070335+0000","last_peered":"2024-09-18T11:45:06.070335+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:06.070335+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2638,"log_dups_size":0,"ondisk_log_size":2638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574902,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2636,"num_write_kb":21084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2537","reported_seq":2618,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:05.865616+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:05.865616+0000","last_peered":"2024-09-18T11:45:05.865616+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:05.865616+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2537,"log_dups_size":0,"ondisk_log_size":2537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20757115,"num_objects":2534,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2536,"num_write_kb":20278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2634","reported_seq":2696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:06.050577+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:06.050577+0000","last_peered":"2024-09-18T11:45:06.050577+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:06.050577+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21542134,"num_objects":2630,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2632,"num_write_kb":21045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2550","reported_seq":2605,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.245450+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:04.245450+0000","last_peered":"2024-09-18T11:45:04.245450+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:04.245450+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20862198,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2548,"num_write_kb":20381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2585","reported_seq":2628,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.259048+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:04.259048+0000","last_peered":"2024-09-18T11:45:04.259048+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:04.259048+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2585,"log_dups_size":0,"ondisk_log_size":2585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21176320,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2585,"num_write_kb":20680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2669","reported_seq":2733,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:06.047874+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:06.047874+0000","last_peered":"2024-09-18T11:45:06.047874+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:06.047874+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2669,"log_dups_size":0,"ondisk_log_size":2669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21812470,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2667,"num_write_kb":21316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":529}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2532","reported_seq":2595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:05.986915+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:05.986915+0000","last_peered":"2024-09-18T11:45:05.986915+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:05.986915+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2532,"log_dups_size":0,"ondisk_log_size":2532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20698358,"num_objects":2527,"num_object_clones":0,"num_object_copies":10108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":20228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":493}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2638","reported_seq":2696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.155338+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:04.155338+0000","last_peered":"2024-09-18T11:45:04.155338+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:04.155338+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2638,"log_dups_size":0,"ondisk_log_size":2638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21539308,"num_objects":2630,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2635,"num_write_kb":21057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2489","reported_seq":2573,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:05.826228+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:05.826228+0000","last_peered":"2024-09-18T11:45:05.826228+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:05.826228+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2489,"log_dups_size":0,"ondisk_log_size":2489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370678,"num_objects":2487,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2487,"num_write_kb":19894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":512}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2546","reported_seq":2603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.205277+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:04.205277+0000","last_peered":"2024-09-18T11:45:04.205277+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:04.205277+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2546,"log_dups_size":0,"ondisk_log_size":2546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20811633,"num_objects":2541,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2543,"num_write_kb":20332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2590","reported_seq":2627,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.241609+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:04.241609+0000","last_peered":"2024-09-18T11:45:04.241609+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:04.241609+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21207675,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2589,"num_write_kb":20711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2581","reported_seq":2656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:05.850814+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:05.850814+0000","last_peered":"2024-09-18T11:45:05.850814+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:05.850814+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2581,"log_dups_size":0,"ondisk_log_size":2581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21133947,"num_objects":2580,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2580,"num_write_kb":20639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":526}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2630","reported_seq":2679,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.233399+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:04.233399+0000","last_peered":"2024-09-18T11:45:04.233399+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:04.233399+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21509366,"num_objects":2626,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2628,"num_write_kb":21020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2617","reported_seq":3247,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:05.890922+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:05.890922+0000","last_peered":"2024-09-18T11:45:05.890922+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:05.890922+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21411062,"num_objects":2614,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":20917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2513","reported_seq":2575,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:06.030753+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:06.030753+0000","last_peered":"2024-09-18T11:45:06.030753+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:06.030753+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2513,"log_dups_size":0,"ondisk_log_size":2513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20542710,"num_objects":2508,"num_object_clones":0,"num_object_copies":10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":495}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":338486644,"num_objects":41324,"num_object_clones":0,"num_object_copies":165296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41351,"num_write_kb":330690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540135424,"data_stored":540135424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41378,"ondisk_log_size":41378,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468992,"kb_used_data":162808,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93902848,"statfs":{"total":96636764160,"available":96156516352,"internally_reserved":0,"allocated":166715392,"data_stored":166556414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,5,0,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9720000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4370000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479144,"kb_used_data":165152,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93892696,"statfs":{"total":96636764160,"available":96146120704,"internally_reserved":0,"allocated":169115648,"data_stored":168956670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":5.6509999999999998},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":2.3530000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.731}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":1.365},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":0.48799999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327752,"kb_used_data":29952,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94044088,"statfs":{"total":96636764160,"available":96301146112,"internally_reserved":0,"allocated":30670848,"data_stored":30511870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9789999999999992}]}]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477120,"kb_used_data":166840,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93894720,"statfs":{"total":96636764160,"available":96148193280,"internally_reserved":0,"allocated":170844160,"data_stored":170685182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,4,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4489999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.851}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":4.4850000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":3.5670000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":170696704,"data_stored":170696704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31621120,"data_stored":31621120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170418176,"data_stored":170418176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167399424,"data_stored":167399424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:08.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:08.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2629", 'reported_seq': 2670, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:04.278733+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:04.278733+0000', 'last_peered': '2024-09-18T11:45:04.278733+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:04.278733+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2629, 'log_dups_size': 0, 'ondisk_log_size': 2629, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21536768, 'num_objects': 2629, 'num_object_clones': 0, 'num_object_copies': 10516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2629, 'num_write_kb': 21032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 499}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:08.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:08.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2638", 'reported_seq': 2701, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:06.070335+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:06.070335+0000', 'last_peered': '2024-09-18T11:45:06.070335+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:06.070335+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2638, 'log_dups_size': 0, 'ondisk_log_size': 2638, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21574902, 'num_objects': 2634, 'num_object_clones': 0, 'num_object_copies': 10536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2636, 'num_write_kb': 21084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 532}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:08.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:08.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2537", 'reported_seq': 2618, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:05.865616+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:05.865616+0000', 'last_peered': '2024-09-18T11:45:05.865616+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:05.865616+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2537, 'log_dups_size': 0, 'ondisk_log_size': 2537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20757115, 'num_objects': 2534, 'num_object_clones': 0, 'num_object_copies': 10136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2536, 'num_write_kb': 20278, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 480}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:08.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:08.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2634", 'reported_seq': 2696, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:06.050577+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:06.050577+0000', 'last_peered': '2024-09-18T11:45:06.050577+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:06.050577+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2634, 'log_dups_size': 0, 'ondisk_log_size': 2634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21542134, 'num_objects': 2630, 'num_object_clones': 0, 'num_object_copies': 10520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2632, 'num_write_kb': 21045, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 516}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:08.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:08.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2550", 'reported_seq': 2605, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:04.245450+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:04.245450+0000', 'last_peered': '2024-09-18T11:45:04.245450+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:04.245450+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2550, 'log_dups_size': 0, 'ondisk_log_size': 2550, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20862198, 'num_objects': 2547, 'num_object_clones': 0, 'num_object_copies': 10188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2548, 'num_write_kb': 20381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 465}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:08.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:08.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2585", 'reported_seq': 2628, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:04.259048+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:04.259048+0000', 'last_peered': '2024-09-18T11:45:04.259048+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:04.259048+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2585, 'log_dups_size': 0, 'ondisk_log_size': 2585, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21176320, 'num_objects': 2585, 'num_object_clones': 0, 'num_object_copies': 10340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2585, 'num_write_kb': 20680, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 431}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:08.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:08.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2669", 'reported_seq': 2733, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:06.047874+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:06.047874+0000', 'last_peered': '2024-09-18T11:45:06.047874+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:06.047874+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2669, 'log_dups_size': 0, 'ondisk_log_size': 2669, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21812470, 'num_objects': 2663, 'num_object_clones': 0, 'num_object_copies': 10652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2667, 'num_write_kb': 21316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 529}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:08.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:08.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2532", 'reported_seq': 2595, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:05.986915+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:05.986915+0000', 'last_peered': '2024-09-18T11:45:05.986915+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:05.986915+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2532, 'log_dups_size': 0, 'ondisk_log_size': 2532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20698358, 'num_objects': 2527, 'num_object_clones': 0, 'num_object_copies': 10108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2530, 'num_write_kb': 20228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 493}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:08.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:08.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2638", 'reported_seq': 2696, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:04.155338+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:04.155338+0000', 'last_peered': '2024-09-18T11:45:04.155338+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:04.155338+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2638, 'log_dups_size': 0, 'ondisk_log_size': 2638, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21539308, 'num_objects': 2630, 'num_object_clones': 0, 'num_object_copies': 10520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2635, 'num_write_kb': 21057, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 495}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:08.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:08.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2489", 'reported_seq': 2573, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:05.826228+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:05.826228+0000', 'last_peered': '2024-09-18T11:45:05.826228+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:05.826228+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2489, 'log_dups_size': 0, 'ondisk_log_size': 2489, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370678, 'num_objects': 2487, 'num_object_clones': 0, 'num_object_copies': 9948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2487, 'num_write_kb': 19894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 512}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:08.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:08.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2546", 'reported_seq': 2603, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:04.205277+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:04.205277+0000', 'last_peered': '2024-09-18T11:45:04.205277+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:04.205277+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2546, 'log_dups_size': 0, 'ondisk_log_size': 2546, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20811633, 'num_objects': 2541, 'num_object_clones': 0, 'num_object_copies': 10164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2543, 'num_write_kb': 20332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 478}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:08.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:08.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2590", 'reported_seq': 2627, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:04.241609+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:04.241609+0000', 'last_peered': '2024-09-18T11:45:04.241609+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:04.241609+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2590, 'log_dups_size': 0, 'ondisk_log_size': 2590, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21207675, 'num_objects': 2589, 'num_object_clones': 0, 'num_object_copies': 10356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2589, 'num_write_kb': 20711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 480}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:08.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:08.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2581", 'reported_seq': 2656, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:05.850814+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:05.850814+0000', 'last_peered': '2024-09-18T11:45:05.850814+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:05.850814+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2581, 'log_dups_size': 0, 'ondisk_log_size': 2581, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21133947, 'num_objects': 2580, 'num_object_clones': 0, 'num_object_copies': 10320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2580, 'num_write_kb': 20639, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 526}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:08.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:08.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2630", 'reported_seq': 2679, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:04.233399+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:04.233399+0000', 'last_peered': '2024-09-18T11:45:04.233399+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:04.233399+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2630, 'log_dups_size': 0, 'ondisk_log_size': 2630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21509366, 'num_objects': 2626, 'num_object_clones': 0, 'num_object_copies': 10504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2628, 'num_write_kb': 21020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 468}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:08.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:08.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2617", 'reported_seq': 3247, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:05.890922+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:05.890922+0000', 'last_peered': '2024-09-18T11:45:05.890922+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:05.890922+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2617, 'log_dups_size': 0, 'ondisk_log_size': 2617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21411062, 'num_objects': 2614, 'num_object_clones': 0, 'num_object_copies': 10456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1817, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2615, 'num_write_kb': 20917, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 276, 'num_bytes_recovered': 2244608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 797}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:08.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:08.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2513", 'reported_seq': 2575, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:06.030753+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:06.030753+0000', 'last_peered': '2024-09-18T11:45:06.030753+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:06.030753+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2513, 'log_dups_size': 0, 'ondisk_log_size': 2513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20542710, 'num_objects': 2508, 'num_object_clones': 0, 'num_object_copies': 10032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2511, 'num_write_kb': 20076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 495}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:08.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:08.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 51 20 42564 42544 6.51608 3.02344 0.0409616 0.0239605 2024-09-18T11:45:08.738 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:08.738 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33153/165296 objects degraded (20.057%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":106,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41324,"data_bytes":338486644,"bytes_used":1795080192,"bytes_avail":384751976448,"bytes_total":386547056640,"degraded_objects":33153,"degraded_total":165296,"degraded_ratio":0.20056746684735263,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":146036,"recovering_keys_per_sec":0,"num_objects_recovered":214,"num_bytes_recovered":1753088,"num_keys_recovered":0,"write_bytes_sec":3840623,"read_op_per_sec":0,"write_op_per_sec":468},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T11:45:05.837774+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:45:08.766 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:08.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:09.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:45:09.152+0000 7f1aecfcf640 0 --2- 172.21.15.71:0/3776770748 >> [v2:172.21.15.71:3302/0,v1:172.21.15.71:6791/0] conn(0x7f1ae0022b90 0x7f1ae0014050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T11:45:09.213 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 52 20 42962 42942 6.45056 3.10938 0.057028 0.024205 2024-09-18T11:45:09.513 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:09.514 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:09.538 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-18T11:45:07.838586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":338486644,"num_objects":41324,"num_object_clones":0,"num_object_copies":165296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41351,"num_write_kb":330690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41378,"ondisk_log_size":41378,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1753008,"kb_used_data":524752,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375734352,"statfs":{"total":386547056640,"available":384751976448,"internally_reserved":0,"allocated":537346048,"data_stored":536710136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,3,4,14,9,7,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46104576,"num_objects":5628,"num_object_clones":0,"num_object_copies":22512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":45024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1753088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004449"},"pg_stats":[{"pgid":"1.f","version":"31'2629","reported_seq":2670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.278733+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:04.278733+0000","last_peered":"2024-09-18T11:45:04.278733+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:04.278733+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21536768,"num_objects":2629,"num_object_clones":0,"num_object_copies":10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":21032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2638","reported_seq":2701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:06.070335+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:06.070335+0000","last_peered":"2024-09-18T11:45:06.070335+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:06.070335+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2638,"log_dups_size":0,"ondisk_log_size":2638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574902,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2636,"num_write_kb":21084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2537","reported_seq":2618,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:05.865616+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:05.865616+0000","last_peered":"2024-09-18T11:45:05.865616+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:05.865616+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2537,"log_dups_size":0,"ondisk_log_size":2537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20757115,"num_objects":2534,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2536,"num_write_kb":20278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2634","reported_seq":2696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:06.050577+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:06.050577+0000","last_peered":"2024-09-18T11:45:06.050577+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:06.050577+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21542134,"num_objects":2630,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2632,"num_write_kb":21045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2550","reported_seq":2605,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.245450+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:04.245450+0000","last_peered":"2024-09-18T11:45:04.245450+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:04.245450+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20862198,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2548,"num_write_kb":20381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2585","reported_seq":2628,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.259048+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:04.259048+0000","last_peered":"2024-09-18T11:45:04.259048+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:04.259048+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2585,"log_dups_size":0,"ondisk_log_size":2585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21176320,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2585,"num_write_kb":20680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2669","reported_seq":2733,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:06.047874+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:06.047874+0000","last_peered":"2024-09-18T11:45:06.047874+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:06.047874+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2669,"log_dups_size":0,"ondisk_log_size":2669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21812470,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2667,"num_write_kb":21316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":529}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2532","reported_seq":2595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:05.986915+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:05.986915+0000","last_peered":"2024-09-18T11:45:05.986915+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:05.986915+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2532,"log_dups_size":0,"ondisk_log_size":2532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20698358,"num_objects":2527,"num_object_clones":0,"num_object_copies":10108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":20228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":493}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2638","reported_seq":2696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.155338+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:04.155338+0000","last_peered":"2024-09-18T11:45:04.155338+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:04.155338+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2638,"log_dups_size":0,"ondisk_log_size":2638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21539308,"num_objects":2630,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2635,"num_write_kb":21057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2489","reported_seq":2573,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:05.826228+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:05.826228+0000","last_peered":"2024-09-18T11:45:05.826228+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:05.826228+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2489,"log_dups_size":0,"ondisk_log_size":2489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370678,"num_objects":2487,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2487,"num_write_kb":19894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":512}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2546","reported_seq":2603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.205277+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:04.205277+0000","last_peered":"2024-09-18T11:45:04.205277+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:04.205277+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2546,"log_dups_size":0,"ondisk_log_size":2546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20811633,"num_objects":2541,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2543,"num_write_kb":20332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2590","reported_seq":2627,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.241609+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:04.241609+0000","last_peered":"2024-09-18T11:45:04.241609+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:04.241609+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21207675,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2589,"num_write_kb":20711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2581","reported_seq":2656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:05.850814+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:05.850814+0000","last_peered":"2024-09-18T11:45:05.850814+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:05.850814+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2581,"log_dups_size":0,"ondisk_log_size":2581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21133947,"num_objects":2580,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2580,"num_write_kb":20639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":526}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2630","reported_seq":2679,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:04.233399+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:04.233399+0000","last_peered":"2024-09-18T11:45:04.233399+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:04.233399+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21509366,"num_objects":2626,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2628,"num_write_kb":21020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2617","reported_seq":3247,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:05.890922+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:05.890922+0000","last_peered":"2024-09-18T11:45:05.890922+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:05.890922+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21411062,"num_objects":2614,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":20917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2513","reported_seq":2575,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:06.030753+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:06.030753+0000","last_peered":"2024-09-18T11:45:06.030753+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:06.030753+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2513,"log_dups_size":0,"ondisk_log_size":2513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20542710,"num_objects":2508,"num_object_clones":0,"num_object_copies":10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":495}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":338486644,"num_objects":41324,"num_object_clones":0,"num_object_copies":165296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41351,"num_write_kb":330690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540135424,"data_stored":540135424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41378,"ondisk_log_size":41378,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468992,"kb_used_data":162808,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93902848,"statfs":{"total":96636764160,"available":96156516352,"internally_reserved":0,"allocated":166715392,"data_stored":166556414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,5,0,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9720000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4370000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479144,"kb_used_data":165152,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93892696,"statfs":{"total":96636764160,"available":96146120704,"internally_reserved":0,"allocated":169115648,"data_stored":168956670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":5.6509999999999998},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":2.3530000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.731}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":1.365},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":0.48799999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327752,"kb_used_data":29952,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94044088,"statfs":{"total":96636764160,"available":96301146112,"internally_reserved":0,"allocated":30670848,"data_stored":30511870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9789999999999992}]}]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477120,"kb_used_data":166840,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93894720,"statfs":{"total":96636764160,"available":96148193280,"internally_reserved":0,"allocated":170844160,"data_stored":170685182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,4,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4489999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.851}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":4.4850000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":3.5670000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":170696704,"data_stored":170696704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31621120,"data_stored":31621120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170418176,"data_stored":170418176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167399424,"data_stored":167399424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:10.214 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 53 20 43386 43366 6.3913 3.3125 0.0802968 0.0244328 2024-09-18T11:45:11.214 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 54 20 43824 43804 6.33631 3.42188 0.0434284 0.0246413 2024-09-18T11:45:12.214 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 55 20 44259 44239 6.28289 3.39844 0.0651502 0.0248546 2024-09-18T11:45:12.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:13.063 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:13.063 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:13.094 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-09-18T11:45:11.839872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":355542388,"num_objects":43406,"num_object_clones":0,"num_object_copies":173624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43433,"num_write_kb":347346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":2834432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43460,"ondisk_log_size":43460,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1770160,"kb_used_data":558540,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375717200,"statfs":{"total":386547056640,"available":384734412800,"internally_reserved":0,"allocated":571944960,"data_stored":571309048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[19,0,4,16,10,16,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44777472,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5466,"num_write_kb":43728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":1777664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004604"},"pg_stats":[{"pgid":"1.f","version":"31'2768","reported_seq":2809,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.226849+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:09.226849+0000","last_peered":"2024-09-18T11:45:09.226849+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:09.226849+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2768,"log_dups_size":0,"ondisk_log_size":2768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":22144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2757","reported_seq":2820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.950716+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:10.950716+0000","last_peered":"2024-09-18T11:45:10.950716+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:10.950716+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22549750,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2755,"num_write_kb":22036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2682","reported_seq":2764,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.886685+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:10.886685+0000","last_peered":"2024-09-18T11:45:10.886685+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:10.886685+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2682,"log_dups_size":0,"ondisk_log_size":2682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21944955,"num_objects":2679,"num_object_clones":0,"num_object_copies":10716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2681,"num_write_kb":21438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2759","reported_seq":2821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.065111+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:11.065111+0000","last_peered":"2024-09-18T11:45:11.065111+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:11.065111+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2759,"log_dups_size":0,"ondisk_log_size":2759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22566134,"num_objects":2755,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2757,"num_write_kb":22045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2692","reported_seq":2747,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.175964+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:09.175964+0000","last_peered":"2024-09-18T11:45:09.175964+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:09.175964+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2692,"log_dups_size":0,"ondisk_log_size":2692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22025462,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2690,"num_write_kb":21517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2723","reported_seq":2766,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.291678+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:09.291678+0000","last_peered":"2024-09-18T11:45:09.291678+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:09.291678+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2723,"log_dups_size":0,"ondisk_log_size":2723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22306816,"num_objects":2723,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2723,"num_write_kb":21784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2802","reported_seq":2867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.012052+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:11.012052+0000","last_peered":"2024-09-18T11:45:11.012052+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:11.012052+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22902006,"num_objects":2796,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2800,"num_write_kb":22380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":662}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2659","reported_seq":2721,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.065994+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:11.065994+0000","last_peered":"2024-09-18T11:45:11.065994+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:11.065994+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2659,"log_dups_size":0,"ondisk_log_size":2659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21738742,"num_objects":2654,"num_object_clones":0,"num_object_copies":10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":21244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2770","reported_seq":2828,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.226933+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:09.226933+0000","last_peered":"2024-09-18T11:45:09.226933+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:09.226933+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22620652,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":22113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2608","reported_seq":2692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.885348+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:10.885348+0000","last_peered":"2024-09-18T11:45:10.885348+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:10.885348+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2608,"log_dups_size":0,"ondisk_log_size":2608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21345526,"num_objects":2606,"num_object_clones":0,"num_object_copies":10424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":20846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":631}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2675","reported_seq":2730,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.279723+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:09.279723+0000","last_peered":"2024-09-18T11:45:09.279723+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:09.279723+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2675,"log_dups_size":0,"ondisk_log_size":2675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21868401,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2672,"num_write_kb":21364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2722","reported_seq":2758,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.186345+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:09.186345+0000","last_peered":"2024-09-18T11:45:09.186345+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:09.186345+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22289019,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":21767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":612}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2721","reported_seq":2797,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.877184+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:10.877184+0000","last_peered":"2024-09-18T11:45:10.877184+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:10.877184+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2721,"log_dups_size":0,"ondisk_log_size":2721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22280827,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":21759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":666}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2763","reported_seq":2812,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.265742+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:09.265742+0000","last_peered":"2024-09-18T11:45:09.265742+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:09.265742+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2763,"log_dups_size":0,"ondisk_log_size":2763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22598902,"num_objects":2759,"num_object_clones":0,"num_object_copies":11036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2761,"num_write_kb":22084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2727","reported_seq":3502,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.920561+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:10.920561+0000","last_peered":"2024-09-18T11:45:10.920561+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:10.920561+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2727,"log_dups_size":0,"ondisk_log_size":2727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22312182,"num_objects":2724,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2725,"num_write_kb":21797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":348,"num_bytes_recovered":2834432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2632","reported_seq":2692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.065332+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:11.065332+0000","last_peered":"2024-09-18T11:45:11.065332+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:11.065332+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2632,"log_dups_size":0,"ondisk_log_size":2632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21517558,"num_objects":2627,"num_object_clones":0,"num_object_copies":10508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":21028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":355542388,"num_objects":43406,"num_object_clones":0,"num_object_copies":173624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43433,"num_write_kb":347346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":2834432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":575152128,"data_stored":575152128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43460,"ondisk_log_size":43460,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481728,"kb_used_data":171384,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93890112,"statfs":{"total":96636764160,"available":96143474688,"internally_reserved":0,"allocated":175497216,"data_stored":175338238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,7,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":5.7489999999999997},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":5.6740000000000004},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":6.8029999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474840,"kb_used_data":173328,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93897000,"statfs":{"total":96636764160,"available":96150528000,"internally_reserved":0,"allocated":177487872,"data_stored":177328894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":7.6050000000000004},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":7.5810000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":7.3979999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340608,"kb_used_data":38644,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94031232,"statfs":{"total":96636764160,"available":96287981568,"internally_reserved":0,"allocated":39571456,"data_stored":39412478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0279999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.162000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472984,"kb_used_data":175184,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93898856,"statfs":{"total":96636764160,"available":96152428544,"internally_reserved":0,"allocated":179388416,"data_stored":179229438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,4,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4489999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.851}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":4.4850000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":3.5670000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179261440,"data_stored":179261440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40747008,"data_stored":40747008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179003392,"data_stored":179003392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176140288,"data_stored":176140288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:13.096 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:13.216 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 56 20 44658 44638 6.22612 3.11719 0.0616469 0.0250785 2024-09-18T11:45:13.619 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:13.619 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:13.644 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-09-18T11:45:11.839872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":355542388,"num_objects":43406,"num_object_clones":0,"num_object_copies":173624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43433,"num_write_kb":347346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":2834432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43460,"ondisk_log_size":43460,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1770160,"kb_used_data":558540,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375717200,"statfs":{"total":386547056640,"available":384734412800,"internally_reserved":0,"allocated":571944960,"data_stored":571309048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[19,0,4,16,10,16,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44777472,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5466,"num_write_kb":43728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":1777664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004604"},"pg_stats":[{"pgid":"1.f","version":"31'2768","reported_seq":2809,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.226849+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:09.226849+0000","last_peered":"2024-09-18T11:45:09.226849+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:09.226849+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2768,"log_dups_size":0,"ondisk_log_size":2768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":22144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2757","reported_seq":2820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.950716+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:10.950716+0000","last_peered":"2024-09-18T11:45:10.950716+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:10.950716+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22549750,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2755,"num_write_kb":22036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2682","reported_seq":2764,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.886685+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:10.886685+0000","last_peered":"2024-09-18T11:45:10.886685+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:10.886685+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2682,"log_dups_size":0,"ondisk_log_size":2682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21944955,"num_objects":2679,"num_object_clones":0,"num_object_copies":10716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2681,"num_write_kb":21438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2759","reported_seq":2821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.065111+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:11.065111+0000","last_peered":"2024-09-18T11:45:11.065111+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:11.065111+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2759,"log_dups_size":0,"ondisk_log_size":2759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22566134,"num_objects":2755,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2757,"num_write_kb":22045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2692","reported_seq":2747,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.175964+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:09.175964+0000","last_peered":"2024-09-18T11:45:09.175964+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:09.175964+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2692,"log_dups_size":0,"ondisk_log_size":2692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22025462,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2690,"num_write_kb":21517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2723","reported_seq":2766,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.291678+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:09.291678+0000","last_peered":"2024-09-18T11:45:09.291678+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:09.291678+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2723,"log_dups_size":0,"ondisk_log_size":2723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22306816,"num_objects":2723,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2723,"num_write_kb":21784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2802","reported_seq":2867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.012052+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:11.012052+0000","last_peered":"2024-09-18T11:45:11.012052+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:11.012052+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22902006,"num_objects":2796,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2800,"num_write_kb":22380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":662}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2659","reported_seq":2721,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.065994+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:11.065994+0000","last_peered":"2024-09-18T11:45:11.065994+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:11.065994+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2659,"log_dups_size":0,"ondisk_log_size":2659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21738742,"num_objects":2654,"num_object_clones":0,"num_object_copies":10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":21244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2770","reported_seq":2828,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.226933+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:09.226933+0000","last_peered":"2024-09-18T11:45:09.226933+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:09.226933+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22620652,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":22113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2608","reported_seq":2692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.885348+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:10.885348+0000","last_peered":"2024-09-18T11:45:10.885348+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:10.885348+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2608,"log_dups_size":0,"ondisk_log_size":2608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21345526,"num_objects":2606,"num_object_clones":0,"num_object_copies":10424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":20846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":631}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2675","reported_seq":2730,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.279723+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:09.279723+0000","last_peered":"2024-09-18T11:45:09.279723+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:09.279723+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2675,"log_dups_size":0,"ondisk_log_size":2675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21868401,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2672,"num_write_kb":21364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2722","reported_seq":2758,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.186345+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:09.186345+0000","last_peered":"2024-09-18T11:45:09.186345+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:09.186345+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22289019,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":21767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":612}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2721","reported_seq":2797,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.877184+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:10.877184+0000","last_peered":"2024-09-18T11:45:10.877184+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:10.877184+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2721,"log_dups_size":0,"ondisk_log_size":2721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22280827,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":21759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":666}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2763","reported_seq":2812,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.265742+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:09.265742+0000","last_peered":"2024-09-18T11:45:09.265742+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:09.265742+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2763,"log_dups_size":0,"ondisk_log_size":2763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22598902,"num_objects":2759,"num_object_clones":0,"num_object_copies":11036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2761,"num_write_kb":22084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2727","reported_seq":3502,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.920561+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:10.920561+0000","last_peered":"2024-09-18T11:45:10.920561+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:10.920561+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2727,"log_dups_size":0,"ondisk_log_size":2727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22312182,"num_objects":2724,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2725,"num_write_kb":21797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":348,"num_bytes_recovered":2834432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2632","reported_seq":2692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.065332+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:11.065332+0000","last_peered":"2024-09-18T11:45:11.065332+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:11.065332+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2632,"log_dups_size":0,"ondisk_log_size":2632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21517558,"num_objects":2627,"num_object_clones":0,"num_object_copies":10508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":21028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":355542388,"num_objects":43406,"num_object_clones":0,"num_object_copies":173624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43433,"num_write_kb":347346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":2834432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":575152128,"data_stored":575152128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43460,"ondisk_log_size":43460,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481728,"kb_used_data":171384,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93890112,"statfs":{"total":96636764160,"available":96143474688,"internally_reserved":0,"allocated":175497216,"data_stored":175338238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,7,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":5.7489999999999997},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":5.6740000000000004},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":6.8029999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474840,"kb_used_data":173328,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93897000,"statfs":{"total":96636764160,"available":96150528000,"internally_reserved":0,"allocated":177487872,"data_stored":177328894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":7.6050000000000004},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":7.5810000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":7.3979999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340608,"kb_used_data":38644,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94031232,"statfs":{"total":96636764160,"available":96287981568,"internally_reserved":0,"allocated":39571456,"data_stored":39412478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0279999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.162000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472984,"kb_used_data":175184,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93898856,"statfs":{"total":96636764160,"available":96152428544,"internally_reserved":0,"allocated":179388416,"data_stored":179229438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,4,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4489999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.851}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":4.4850000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":3.5670000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179261440,"data_stored":179261440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40747008,"data_stored":40747008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179003392,"data_stored":179003392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176140288,"data_stored":176140288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:13.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:13.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2768", 'reported_seq': 2809, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:09.226849+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:09.226849+0000', 'last_peered': '2024-09-18T11:45:09.226849+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:09.226849+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2768, 'log_dups_size': 0, 'ondisk_log_size': 2768, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22675456, 'num_objects': 2768, 'num_object_clones': 0, 'num_object_copies': 11072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2768, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2768, 'num_write_kb': 22144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 638}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:13.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2757", 'reported_seq': 2820, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:10.950716+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:10.950716+0000', 'last_peered': '2024-09-18T11:45:10.950716+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:10.950716+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2757, 'log_dups_size': 0, 'ondisk_log_size': 2757, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22549750, 'num_objects': 2753, 'num_object_clones': 0, 'num_object_copies': 11012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2753, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2755, 'num_write_kb': 22036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2682", 'reported_seq': 2764, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:10.886685+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:10.886685+0000', 'last_peered': '2024-09-18T11:45:10.886685+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:10.886685+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2682, 'log_dups_size': 0, 'ondisk_log_size': 2682, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21944955, 'num_objects': 2679, 'num_object_clones': 0, 'num_object_copies': 10716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2681, 'num_write_kb': 21438, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 625}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2759", 'reported_seq': 2821, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:11.065111+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:11.065111+0000', 'last_peered': '2024-09-18T11:45:11.065111+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:11.065111+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2759, 'log_dups_size': 0, 'ondisk_log_size': 2759, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22566134, 'num_objects': 2755, 'num_object_clones': 0, 'num_object_copies': 11020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2757, 'num_write_kb': 22045, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 641}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2692", 'reported_seq': 2747, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:09.175964+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:09.175964+0000', 'last_peered': '2024-09-18T11:45:09.175964+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:09.175964+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2692, 'log_dups_size': 0, 'ondisk_log_size': 2692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22025462, 'num_objects': 2689, 'num_object_clones': 0, 'num_object_copies': 10756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2690, 'num_write_kb': 21517, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 607}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2723", 'reported_seq': 2766, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:09.291678+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:09.291678+0000', 'last_peered': '2024-09-18T11:45:09.291678+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:09.291678+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2723, 'log_dups_size': 0, 'ondisk_log_size': 2723, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22306816, 'num_objects': 2723, 'num_object_clones': 0, 'num_object_copies': 10892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2723, 'num_write_kb': 21784, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 569}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:13.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2802", 'reported_seq': 2867, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:11.012052+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:11.012052+0000', 'last_peered': '2024-09-18T11:45:11.012052+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:11.012052+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2802, 'log_dups_size': 0, 'ondisk_log_size': 2802, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22902006, 'num_objects': 2796, 'num_object_clones': 0, 'num_object_copies': 11184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2800, 'num_write_kb': 22380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 662}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:13.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:13.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2659", 'reported_seq': 2721, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:11.065994+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:11.065994+0000', 'last_peered': '2024-09-18T11:45:11.065994+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:11.065994+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2659, 'log_dups_size': 0, 'ondisk_log_size': 2659, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21738742, 'num_objects': 2654, 'num_object_clones': 0, 'num_object_copies': 10616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2654, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2657, 'num_write_kb': 21244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 620}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:13.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:13.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2770", 'reported_seq': 2828, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:09.226933+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:09.226933+0000', 'last_peered': '2024-09-18T11:45:09.226933+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:09.226933+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2770, 'log_dups_size': 0, 'ondisk_log_size': 2770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22620652, 'num_objects': 2762, 'num_object_clones': 0, 'num_object_copies': 11048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2767, 'num_write_kb': 22113, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 627}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:13.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:13.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2608", 'reported_seq': 2692, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:10.885348+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:10.885348+0000', 'last_peered': '2024-09-18T11:45:10.885348+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:10.885348+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2608, 'log_dups_size': 0, 'ondisk_log_size': 2608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21345526, 'num_objects': 2606, 'num_object_clones': 0, 'num_object_copies': 10424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2606, 'num_write_kb': 20846, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 631}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:13.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:13.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2675", 'reported_seq': 2730, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:09.279723+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:09.279723+0000', 'last_peered': '2024-09-18T11:45:09.279723+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:09.279723+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2675, 'log_dups_size': 0, 'ondisk_log_size': 2675, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21868401, 'num_objects': 2670, 'num_object_clones': 0, 'num_object_copies': 10680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2670, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2672, 'num_write_kb': 21364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 607}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:13.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:13.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2722", 'reported_seq': 2758, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:09.186345+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:09.186345+0000', 'last_peered': '2024-09-18T11:45:09.186345+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:09.186345+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2722, 'log_dups_size': 0, 'ondisk_log_size': 2722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22289019, 'num_objects': 2721, 'num_object_clones': 0, 'num_object_copies': 10884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2721, 'num_write_kb': 21767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 612}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:13.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:13.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2721", 'reported_seq': 2797, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:10.877184+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:10.877184+0000', 'last_peered': '2024-09-18T11:45:10.877184+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:10.877184+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2721, 'log_dups_size': 0, 'ondisk_log_size': 2721, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22280827, 'num_objects': 2720, 'num_object_clones': 0, 'num_object_copies': 10880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2720, 'num_write_kb': 21759, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 666}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:13.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:13.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2763", 'reported_seq': 2812, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:09.265742+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:09.265742+0000', 'last_peered': '2024-09-18T11:45:09.265742+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:09.265742+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2763, 'log_dups_size': 0, 'ondisk_log_size': 2763, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22598902, 'num_objects': 2759, 'num_object_clones': 0, 'num_object_copies': 11036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2761, 'num_write_kb': 22084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:13.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:13.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2727", 'reported_seq': 3502, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:10.920561+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:10.920561+0000', 'last_peered': '2024-09-18T11:45:10.920561+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:10.920561+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2727, 'log_dups_size': 0, 'ondisk_log_size': 2727, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22312182, 'num_objects': 2724, 'num_object_clones': 0, 'num_object_copies': 10896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1745, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2725, 'num_write_kb': 21797, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 348, 'num_bytes_recovered': 2834432, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 979}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:13.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:13.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2632", 'reported_seq': 2692, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:11.065332+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:11.065332+0000', 'last_peered': '2024-09-18T11:45:11.065332+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:11.065332+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2632, 'log_dups_size': 0, 'ondisk_log_size': 2632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21517558, 'num_objects': 2627, 'num_object_clones': 0, 'num_object_copies': 10508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2630, 'num_write_kb': 21028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 614}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:13.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:14.216 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 57 20 45041 45021 6.16939 2.99219 0.0689106 0.0253131 2024-09-18T11:45:14.279 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:14.279 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33081/173624 objects degraded (19.053%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":112,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43406,"data_bytes":355542388,"bytes_used":1812643840,"bytes_avail":384734412800,"bytes_total":386547056640,"degraded_objects":33081,"degraded_total":173624,"degraded_ratio":0.1905324148735198,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":96902,"recovering_keys_per_sec":0,"num_objects_recovered":142,"num_bytes_recovered":1163264,"num_keys_recovered":0,"write_bytes_sec":2885239,"read_op_per_sec":0,"write_op_per_sec":352},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T11:45:05.837774+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:45:14.309 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:14.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:14.867 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:14.867 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:14.893 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-18T11:45:13.840489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":355542388,"num_objects":43406,"num_object_clones":0,"num_object_copies":173624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43433,"num_write_kb":347346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":2834432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43460,"ondisk_log_size":43460,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1770160,"kb_used_data":558540,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375717200,"statfs":{"total":386547056640,"available":384734412800,"internally_reserved":0,"allocated":571944960,"data_stored":571309048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[19,0,4,16,10,16,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34635776,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4228,"num_write_kb":33824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":1163264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004472"},"pg_stats":[{"pgid":"1.f","version":"31'2768","reported_seq":2809,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.226849+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:09.226849+0000","last_peered":"2024-09-18T11:45:09.226849+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:09.226849+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2768,"log_dups_size":0,"ondisk_log_size":2768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":22144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2757","reported_seq":2820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.950716+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:10.950716+0000","last_peered":"2024-09-18T11:45:10.950716+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:10.950716+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22549750,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2755,"num_write_kb":22036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2682","reported_seq":2764,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.886685+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:10.886685+0000","last_peered":"2024-09-18T11:45:10.886685+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:10.886685+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2682,"log_dups_size":0,"ondisk_log_size":2682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21944955,"num_objects":2679,"num_object_clones":0,"num_object_copies":10716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2681,"num_write_kb":21438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2759","reported_seq":2821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.065111+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:11.065111+0000","last_peered":"2024-09-18T11:45:11.065111+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:11.065111+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2759,"log_dups_size":0,"ondisk_log_size":2759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22566134,"num_objects":2755,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2757,"num_write_kb":22045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2692","reported_seq":2747,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.175964+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:09.175964+0000","last_peered":"2024-09-18T11:45:09.175964+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:09.175964+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2692,"log_dups_size":0,"ondisk_log_size":2692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22025462,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2690,"num_write_kb":21517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2723","reported_seq":2766,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.291678+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:09.291678+0000","last_peered":"2024-09-18T11:45:09.291678+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:09.291678+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2723,"log_dups_size":0,"ondisk_log_size":2723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22306816,"num_objects":2723,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2723,"num_write_kb":21784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2802","reported_seq":2867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.012052+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:11.012052+0000","last_peered":"2024-09-18T11:45:11.012052+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:11.012052+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22902006,"num_objects":2796,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2800,"num_write_kb":22380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":662}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2659","reported_seq":2721,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.065994+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:11.065994+0000","last_peered":"2024-09-18T11:45:11.065994+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:11.065994+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2659,"log_dups_size":0,"ondisk_log_size":2659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21738742,"num_objects":2654,"num_object_clones":0,"num_object_copies":10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":21244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2770","reported_seq":2828,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.226933+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:09.226933+0000","last_peered":"2024-09-18T11:45:09.226933+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:09.226933+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22620652,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":22113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2608","reported_seq":2692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.885348+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:10.885348+0000","last_peered":"2024-09-18T11:45:10.885348+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:10.885348+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2608,"log_dups_size":0,"ondisk_log_size":2608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21345526,"num_objects":2606,"num_object_clones":0,"num_object_copies":10424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":20846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":631}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2675","reported_seq":2730,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.279723+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:09.279723+0000","last_peered":"2024-09-18T11:45:09.279723+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:09.279723+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2675,"log_dups_size":0,"ondisk_log_size":2675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21868401,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2672,"num_write_kb":21364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2722","reported_seq":2758,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.186345+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:09.186345+0000","last_peered":"2024-09-18T11:45:09.186345+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:09.186345+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22289019,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":21767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":612}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2721","reported_seq":2797,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.877184+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:10.877184+0000","last_peered":"2024-09-18T11:45:10.877184+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:10.877184+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2721,"log_dups_size":0,"ondisk_log_size":2721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22280827,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":21759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":666}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2763","reported_seq":2812,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:09.265742+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:09.265742+0000","last_peered":"2024-09-18T11:45:09.265742+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:09.265742+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2763,"log_dups_size":0,"ondisk_log_size":2763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22598902,"num_objects":2759,"num_object_clones":0,"num_object_copies":11036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2761,"num_write_kb":22084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2727","reported_seq":3502,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:10.920561+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:10.920561+0000","last_peered":"2024-09-18T11:45:10.920561+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:10.920561+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2727,"log_dups_size":0,"ondisk_log_size":2727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22312182,"num_objects":2724,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2725,"num_write_kb":21797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":348,"num_bytes_recovered":2834432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2632","reported_seq":2692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:11.065332+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:11.065332+0000","last_peered":"2024-09-18T11:45:11.065332+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:11.065332+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2632,"log_dups_size":0,"ondisk_log_size":2632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21517558,"num_objects":2627,"num_object_clones":0,"num_object_copies":10508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":21028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":355542388,"num_objects":43406,"num_object_clones":0,"num_object_copies":173624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43433,"num_write_kb":347346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":2834432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":575152128,"data_stored":575152128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43460,"ondisk_log_size":43460,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481728,"kb_used_data":171384,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93890112,"statfs":{"total":96636764160,"available":96143474688,"internally_reserved":0,"allocated":175497216,"data_stored":175338238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,7,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":5.7489999999999997},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":5.6740000000000004},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":6.8029999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474840,"kb_used_data":173328,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93897000,"statfs":{"total":96636764160,"available":96150528000,"internally_reserved":0,"allocated":177487872,"data_stored":177328894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":7.6050000000000004},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":7.5810000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":7.3979999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340608,"kb_used_data":38644,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94031232,"statfs":{"total":96636764160,"available":96287981568,"internally_reserved":0,"allocated":39571456,"data_stored":39412478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0279999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.162000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472984,"kb_used_data":175184,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93898856,"statfs":{"total":96636764160,"available":96152428544,"internally_reserved":0,"allocated":179388416,"data_stored":179229438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,4,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4489999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.851}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":4.4850000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":3.5670000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179261440,"data_stored":179261440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40747008,"data_stored":40747008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179003392,"data_stored":179003392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176140288,"data_stored":176140288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:15.216 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 58 20 45444 45424 6.11731 3.14844 0.0389331 0.025531 2024-09-18T11:45:16.216 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 59 20 45868 45848 6.06977 3.3125 0.0360981 0.0257252 2024-09-18T11:45:17.217 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:45:17.216349+0000 min lat: 0.00272854 max lat: 2.42146 avg lat: 0.0259203 2024-09-18T11:45:17.217 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:45:17.217 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 60 20 46302 46282 6.02512 3.39062 0.0447032 0.0259203 2024-09-18T11:45:17.895 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:18.217 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 61 20 46716 46696 5.97936 3.23438 0.0448988 0.0261146 2024-09-18T11:45:18.487 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:18.487 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:18.506 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-18T11:45:17.841358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":372917620,"num_objects":45527,"num_object_clones":0,"num_object_copies":182108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45554,"num_write_kb":364314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":3391488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45581,"ondisk_log_size":45581,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1813520,"kb_used_data":593584,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375673840,"statfs":{"total":386547056640,"available":384690012160,"internally_reserved":0,"allocated":607830016,"data_stored":607194104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,4,8,5,13,17,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44089344,"num_objects":5382,"num_object_clones":0,"num_object_copies":21528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5382,"num_write_kb":43056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003830"},"pg_stats":[{"pgid":"1.f","version":"31'2922","reported_seq":2962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.257887+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:14.257887+0000","last_peered":"2024-09-18T11:45:14.257887+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:14.257887+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23937024,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":23376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2875","reported_seq":2938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.935511+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:15.935511+0000","last_peered":"2024-09-18T11:45:15.935511+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:15.935511+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23516406,"num_objects":2871,"num_object_clones":0,"num_object_copies":11484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2873,"num_write_kb":22980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2816","reported_seq":2898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.848440+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:15.848440+0000","last_peered":"2024-09-18T11:45:15.848440+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:15.848440+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2816,"log_dups_size":0,"ondisk_log_size":2816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23042683,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2815,"num_write_kb":22510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2877","reported_seq":2939,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.982508+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:15.982508+0000","last_peered":"2024-09-18T11:45:15.982508+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:15.982508+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2877,"log_dups_size":0,"ondisk_log_size":2877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23532790,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":22989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2825","reported_seq":2879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.214599+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:14.214599+0000","last_peered":"2024-09-18T11:45:14.214599+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:14.214599+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2825,"log_dups_size":0,"ondisk_log_size":2825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23114998,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":22581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2850","reported_seq":2893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.278702+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:14.278702+0000","last_peered":"2024-09-18T11:45:14.278702+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:14.278702+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23347200,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2850,"num_write_kb":22800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2935","reported_seq":3000,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:16.088051+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:16.088051+0000","last_peered":"2024-09-18T11:45:16.088051+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:16.088051+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2935,"log_dups_size":0,"ondisk_log_size":2935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23991542,"num_objects":2929,"num_object_clones":0,"num_object_copies":11716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2933,"num_write_kb":23444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":795}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2798","reported_seq":2861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:16.059129+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:16.059129+0000","last_peered":"2024-09-18T11:45:16.059129+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:16.059129+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2798,"log_dups_size":0,"ondisk_log_size":2798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22877430,"num_objects":2793,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2796,"num_write_kb":22356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2902","reported_seq":2960,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.259890+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:14.259890+0000","last_peered":"2024-09-18T11:45:14.259890+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:14.259890+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2902,"log_dups_size":0,"ondisk_log_size":2902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23701996,"num_objects":2894,"num_object_clones":0,"num_object_copies":11576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2899,"num_write_kb":23169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2736","reported_seq":2820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.919795+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:15.919795+0000","last_peered":"2024-09-18T11:45:15.919795+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:15.919795+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22394102,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":21870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2799","reported_seq":2856,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.214817+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:14.214817+0000","last_peered":"2024-09-18T11:45:14.214817+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:14.214817+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2799,"log_dups_size":0,"ondisk_log_size":2799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22884209,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2796,"num_write_kb":22356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":731}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2866","reported_seq":2903,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.279721+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:14.279721+0000","last_peered":"2024-09-18T11:45:14.279721+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:14.279721+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2866,"log_dups_size":0,"ondisk_log_size":2866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23468667,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":756}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2860","reported_seq":2936,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.827248+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:15.827248+0000","last_peered":"2024-09-18T11:45:15.827248+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:15.827248+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2860,"log_dups_size":0,"ondisk_log_size":2860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23419515,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2889","reported_seq":2938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.278315+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:14.278315+0000","last_peered":"2024-09-18T11:45:14.278315+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:14.278315+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2889,"log_dups_size":0,"ondisk_log_size":2889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23631094,"num_objects":2885,"num_object_clones":0,"num_object_copies":11540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2887,"num_write_kb":23092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":727}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2861","reported_seq":3772,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.923292+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:15.923292+0000","last_peered":"2024-09-18T11:45:15.923292+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:15.923292+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2861,"log_dups_size":0,"ondisk_log_size":2861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409910,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":3391488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2770","reported_seq":2832,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:16.043447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:16.043447+0000","last_peered":"2024-09-18T11:45:16.043447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:16.043447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648054,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":22132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":372917620,"num_objects":45527,"num_object_clones":0,"num_object_copies":182108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45554,"num_write_kb":364314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":3391488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":609357824,"data_stored":609357824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45581,"ondisk_log_size":45581,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481808,"kb_used_data":179784,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93890032,"statfs":{"total":96636764160,"available":96143392768,"internally_reserved":0,"allocated":184098816,"data_stored":183939838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,1,7,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":13.491}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.561999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":0.90400000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488964,"kb_used_data":183292,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93882876,"statfs":{"total":96636764160,"available":96136065024,"internally_reserved":0,"allocated":187691008,"data_stored":187532030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":7.6050000000000004},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":7.5810000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":7.3979999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353340,"kb_used_data":47220,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":94018500,"statfs":{"total":96636764160,"available":96274944000,"internally_reserved":0,"allocated":48353280,"data_stored":48194302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.586}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1269999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8559999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489408,"kb_used_data":183288,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93882432,"statfs":{"total":96636764160,"available":96135610368,"internally_reserved":0,"allocated":187686912,"data_stored":187527934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,5,0,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":1.246},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.0069999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187674624,"data_stored":187674624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49582080,"data_stored":49582080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187437056,"data_stored":187437056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184664064,"data_stored":184664064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:18.507 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:19.057 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:19.084 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-18T11:45:17.841358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":372917620,"num_objects":45527,"num_object_clones":0,"num_object_copies":182108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45554,"num_write_kb":364314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":3391488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45581,"ondisk_log_size":45581,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1813520,"kb_used_data":593584,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375673840,"statfs":{"total":386547056640,"available":384690012160,"internally_reserved":0,"allocated":607830016,"data_stored":607194104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,4,8,5,13,17,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44089344,"num_objects":5382,"num_object_clones":0,"num_object_copies":21528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5382,"num_write_kb":43056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003830"},"pg_stats":[{"pgid":"1.f","version":"31'2922","reported_seq":2962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.257887+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:14.257887+0000","last_peered":"2024-09-18T11:45:14.257887+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:14.257887+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23937024,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":23376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2875","reported_seq":2938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.935511+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:15.935511+0000","last_peered":"2024-09-18T11:45:15.935511+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:15.935511+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23516406,"num_objects":2871,"num_object_clones":0,"num_object_copies":11484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2873,"num_write_kb":22980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2816","reported_seq":2898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.848440+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:15.848440+0000","last_peered":"2024-09-18T11:45:15.848440+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:15.848440+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2816,"log_dups_size":0,"ondisk_log_size":2816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23042683,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2815,"num_write_kb":22510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2877","reported_seq":2939,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.982508+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:15.982508+0000","last_peered":"2024-09-18T11:45:15.982508+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:15.982508+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2877,"log_dups_size":0,"ondisk_log_size":2877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23532790,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":22989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2825","reported_seq":2879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.214599+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:14.214599+0000","last_peered":"2024-09-18T11:45:14.214599+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:14.214599+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2825,"log_dups_size":0,"ondisk_log_size":2825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23114998,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":22581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2850","reported_seq":2893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.278702+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:14.278702+0000","last_peered":"2024-09-18T11:45:14.278702+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:14.278702+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23347200,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2850,"num_write_kb":22800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2935","reported_seq":3000,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:16.088051+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:16.088051+0000","last_peered":"2024-09-18T11:45:16.088051+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:16.088051+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2935,"log_dups_size":0,"ondisk_log_size":2935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23991542,"num_objects":2929,"num_object_clones":0,"num_object_copies":11716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2933,"num_write_kb":23444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":795}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2798","reported_seq":2861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:16.059129+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:16.059129+0000","last_peered":"2024-09-18T11:45:16.059129+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:16.059129+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2798,"log_dups_size":0,"ondisk_log_size":2798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22877430,"num_objects":2793,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2796,"num_write_kb":22356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2902","reported_seq":2960,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.259890+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:14.259890+0000","last_peered":"2024-09-18T11:45:14.259890+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:14.259890+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2902,"log_dups_size":0,"ondisk_log_size":2902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23701996,"num_objects":2894,"num_object_clones":0,"num_object_copies":11576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2899,"num_write_kb":23169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2736","reported_seq":2820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.919795+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:15.919795+0000","last_peered":"2024-09-18T11:45:15.919795+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:15.919795+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22394102,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":21870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2799","reported_seq":2856,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.214817+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:14.214817+0000","last_peered":"2024-09-18T11:45:14.214817+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:14.214817+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2799,"log_dups_size":0,"ondisk_log_size":2799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22884209,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2796,"num_write_kb":22356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":731}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2866","reported_seq":2903,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.279721+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:14.279721+0000","last_peered":"2024-09-18T11:45:14.279721+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:14.279721+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2866,"log_dups_size":0,"ondisk_log_size":2866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23468667,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":756}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2860","reported_seq":2936,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.827248+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:15.827248+0000","last_peered":"2024-09-18T11:45:15.827248+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:15.827248+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2860,"log_dups_size":0,"ondisk_log_size":2860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23419515,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2889","reported_seq":2938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:14.278315+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:14.278315+0000","last_peered":"2024-09-18T11:45:14.278315+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:14.278315+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2889,"log_dups_size":0,"ondisk_log_size":2889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23631094,"num_objects":2885,"num_object_clones":0,"num_object_copies":11540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2887,"num_write_kb":23092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":727}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2861","reported_seq":3772,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.923292+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:15.923292+0000","last_peered":"2024-09-18T11:45:15.923292+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:15.923292+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2861,"log_dups_size":0,"ondisk_log_size":2861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409910,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":3391488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2770","reported_seq":2832,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:16.043447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:16.043447+0000","last_peered":"2024-09-18T11:45:16.043447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:16.043447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648054,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":22132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":372917620,"num_objects":45527,"num_object_clones":0,"num_object_copies":182108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45554,"num_write_kb":364314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":3391488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":609357824,"data_stored":609357824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45581,"ondisk_log_size":45581,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481808,"kb_used_data":179784,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93890032,"statfs":{"total":96636764160,"available":96143392768,"internally_reserved":0,"allocated":184098816,"data_stored":183939838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,1,7,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":13.491}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.561999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":0.90400000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488964,"kb_used_data":183292,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93882876,"statfs":{"total":96636764160,"available":96136065024,"internally_reserved":0,"allocated":187691008,"data_stored":187532030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":7.6050000000000004},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":7.5810000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":7.3979999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353340,"kb_used_data":47220,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":94018500,"statfs":{"total":96636764160,"available":96274944000,"internally_reserved":0,"allocated":48353280,"data_stored":48194302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.586}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1269999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8559999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489408,"kb_used_data":183288,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93882432,"statfs":{"total":96636764160,"available":96135610368,"internally_reserved":0,"allocated":187686912,"data_stored":187527934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,5,0,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":1.246},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.0069999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187674624,"data_stored":187674624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49582080,"data_stored":49582080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187437056,"data_stored":187437056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184664064,"data_stored":184664064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:19.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:19.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2922", 'reported_seq': 2962, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:14.257887+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:14.257887+0000', 'last_peered': '2024-09-18T11:45:14.257887+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:14.257887+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2922, 'log_dups_size': 0, 'ondisk_log_size': 2922, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23937024, 'num_objects': 2922, 'num_object_clones': 0, 'num_object_copies': 11688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2922, 'num_write_kb': 23376, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 792}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:19.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:19.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2875", 'reported_seq': 2938, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:15.935511+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:15.935511+0000', 'last_peered': '2024-09-18T11:45:15.935511+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:15.935511+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2875, 'log_dups_size': 0, 'ondisk_log_size': 2875, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23516406, 'num_objects': 2871, 'num_object_clones': 0, 'num_object_copies': 11484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2873, 'num_write_kb': 22980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 769}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:19.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:19.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2816", 'reported_seq': 2898, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:15.848440+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:15.848440+0000', 'last_peered': '2024-09-18T11:45:15.848440+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:15.848440+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2816, 'log_dups_size': 0, 'ondisk_log_size': 2816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23042683, 'num_objects': 2813, 'num_object_clones': 0, 'num_object_copies': 11252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2815, 'num_write_kb': 22510, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 759}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2877", 'reported_seq': 2939, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:15.982508+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:15.982508+0000', 'last_peered': '2024-09-18T11:45:15.982508+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:15.982508+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2877, 'log_dups_size': 0, 'ondisk_log_size': 2877, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23532790, 'num_objects': 2873, 'num_object_clones': 0, 'num_object_copies': 11492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2875, 'num_write_kb': 22989, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 759}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2825", 'reported_seq': 2879, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:14.214599+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:14.214599+0000', 'last_peered': '2024-09-18T11:45:14.214599+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:14.214599+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2825, 'log_dups_size': 0, 'ondisk_log_size': 2825, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23114998, 'num_objects': 2822, 'num_object_clones': 0, 'num_object_copies': 11288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2823, 'num_write_kb': 22581, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 740}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2850", 'reported_seq': 2893, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:14.278702+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:14.278702+0000', 'last_peered': '2024-09-18T11:45:14.278702+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:14.278702+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2850, 'log_dups_size': 0, 'ondisk_log_size': 2850, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23347200, 'num_objects': 2850, 'num_object_clones': 0, 'num_object_copies': 11400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2850, 'num_write_kb': 22800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 696}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2935", 'reported_seq': 3000, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:16.088051+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:16.088051+0000', 'last_peered': '2024-09-18T11:45:16.088051+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:16.088051+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2935, 'log_dups_size': 0, 'ondisk_log_size': 2935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23991542, 'num_objects': 2929, 'num_object_clones': 0, 'num_object_copies': 11716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2933, 'num_write_kb': 23444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 795}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2798", 'reported_seq': 2861, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:16.059129+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:16.059129+0000', 'last_peered': '2024-09-18T11:45:16.059129+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:16.059129+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2798, 'log_dups_size': 0, 'ondisk_log_size': 2798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22877430, 'num_objects': 2793, 'num_object_clones': 0, 'num_object_copies': 11172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2796, 'num_write_kb': 22356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 759}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:19.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2902", 'reported_seq': 2960, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:14.259890+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:14.259890+0000', 'last_peered': '2024-09-18T11:45:14.259890+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:14.259890+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2902, 'log_dups_size': 0, 'ondisk_log_size': 2902, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23701996, 'num_objects': 2894, 'num_object_clones': 0, 'num_object_copies': 11576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2894, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2899, 'num_write_kb': 23169, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 759}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2736", 'reported_seq': 2820, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:15.919795+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:15.919795+0000', 'last_peered': '2024-09-18T11:45:15.919795+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:15.919795+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2736, 'log_dups_size': 0, 'ondisk_log_size': 2736, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22394102, 'num_objects': 2734, 'num_object_clones': 0, 'num_object_copies': 10936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2734, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2734, 'num_write_kb': 21870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 759}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2799", 'reported_seq': 2856, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:14.214817+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:14.214817+0000', 'last_peered': '2024-09-18T11:45:14.214817+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:14.214817+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2799, 'log_dups_size': 0, 'ondisk_log_size': 2799, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22884209, 'num_objects': 2794, 'num_object_clones': 0, 'num_object_copies': 11176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2796, 'num_write_kb': 22356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 731}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2866", 'reported_seq': 2903, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:14.279721+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:14.279721+0000', 'last_peered': '2024-09-18T11:45:14.279721+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:14.279721+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2866, 'log_dups_size': 0, 'ondisk_log_size': 2866, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23468667, 'num_objects': 2865, 'num_object_clones': 0, 'num_object_copies': 11460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2865, 'num_write_kb': 22919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 756}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2860", 'reported_seq': 2936, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:15.827248+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:15.827248+0000', 'last_peered': '2024-09-18T11:45:15.827248+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:15.827248+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2860, 'log_dups_size': 0, 'ondisk_log_size': 2860, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23419515, 'num_objects': 2859, 'num_object_clones': 0, 'num_object_copies': 11436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2859, 'num_write_kb': 22871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 805}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:19.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2889", 'reported_seq': 2938, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:14.278315+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:14.278315+0000', 'last_peered': '2024-09-18T11:45:14.278315+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:14.278315+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2889, 'log_dups_size': 0, 'ondisk_log_size': 2889, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23631094, 'num_objects': 2885, 'num_object_clones': 0, 'num_object_copies': 11540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2887, 'num_write_kb': 23092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 727}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:19.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:19.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2861", 'reported_seq': 3772, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:15.923292+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:15.923292+0000', 'last_peered': '2024-09-18T11:45:15.923292+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:15.923292+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2861, 'log_dups_size': 0, 'ondisk_log_size': 2861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23409910, 'num_objects': 2858, 'num_object_clones': 0, 'num_object_copies': 11432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1677, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2859, 'num_write_kb': 22869, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 416, 'num_bytes_recovered': 3391488, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1181}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:19.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:19.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2770", 'reported_seq': 2832, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:16.043447+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:16.043447+0000', 'last_peered': '2024-09-18T11:45:16.043447+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:16.043447+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2770, 'log_dups_size': 0, 'ondisk_log_size': 2770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22648054, 'num_objects': 2765, 'num_object_clones': 0, 'num_object_copies': 11060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2768, 'num_write_kb': 22132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:19.088 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:19.218 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 62 20 47090 47070 5.93005 2.92188 0.0563086 0.0263309 2024-09-18T11:45:19.759 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:19.759 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33013/182108 objects degraded (18.128%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":117,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45527,"data_bytes":372917620,"bytes_used":1857044480,"bytes_avail":384690012160,"bytes_total":386547056640,"degraded_objects":33013,"degraded_total":182108,"degraded_ratio":0.18128253563819272,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":143314,"recovering_keys_per_sec":0,"num_objects_recovered":210,"num_bytes_recovered":1720320,"num_keys_recovered":0,"write_bytes_sec":3672939,"read_op_per_sec":0,"write_op_per_sec":448},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T11:45:05.837774+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:45:19.786 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:19.786 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:20.217 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 63 20 47467 47447 5.88268 2.94531 0.047266 0.0265425 2024-09-18T11:45:20.334 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:20.334 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:20.362 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-18T11:45:19.841814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":379913588,"num_objects":46381,"num_object_clones":0,"num_object_copies":185524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46408,"num_write_kb":371146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":3391488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46435,"ondisk_log_size":46435,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1842612,"kb_used_data":610196,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":375644748,"statfs":{"total":386547056640,"available":384660221952,"internally_reserved":0,"allocated":624840704,"data_stored":624204792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,4,8,7,12,17,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41426944,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5057,"num_write_kb":40456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003227"},"pg_stats":[{"pgid":"1.f","version":"31'3047","reported_seq":3088,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.270210+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:19.270210+0000","last_peered":"2024-09-18T11:45:19.270210+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:19.270210+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3047,"log_dups_size":0,"ondisk_log_size":3047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24961024,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3047,"num_write_kb":24376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2875","reported_seq":2938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.935511+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:15.935511+0000","last_peered":"2024-09-18T11:45:15.935511+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:15.935511+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23516406,"num_objects":2871,"num_object_clones":0,"num_object_copies":11484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2873,"num_write_kb":22980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2816","reported_seq":2898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.848440+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:15.848440+0000","last_peered":"2024-09-18T11:45:15.848440+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:15.848440+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2816,"log_dups_size":0,"ondisk_log_size":2816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23042683,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2815,"num_write_kb":22510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2877","reported_seq":2939,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.982508+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:15.982508+0000","last_peered":"2024-09-18T11:45:15.982508+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:15.982508+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2877,"log_dups_size":0,"ondisk_log_size":2877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23532790,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":22989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2950","reported_seq":3005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.315687+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:19.315687+0000","last_peered":"2024-09-18T11:45:19.315687+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:19.315687+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24138998,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2948,"num_write_kb":23581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2976","reported_seq":3019,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.298772+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:19.298772+0000","last_peered":"2024-09-18T11:45:19.298772+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:19.298772+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2976,"log_dups_size":0,"ondisk_log_size":2976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379392,"num_objects":2976,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2935","reported_seq":3000,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:16.088051+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:16.088051+0000","last_peered":"2024-09-18T11:45:16.088051+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:16.088051+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2935,"log_dups_size":0,"ondisk_log_size":2935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23991542,"num_objects":2929,"num_object_clones":0,"num_object_copies":11716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2933,"num_write_kb":23444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":795}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2798","reported_seq":2861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:16.059129+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:16.059129+0000","last_peered":"2024-09-18T11:45:16.059129+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:16.059129+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2798,"log_dups_size":0,"ondisk_log_size":2798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22877430,"num_objects":2793,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2796,"num_write_kb":22356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3020","reported_seq":3078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.247324+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:19.247324+0000","last_peered":"2024-09-18T11:45:19.247324+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:19.247324+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24668652,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":877}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2736","reported_seq":2820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.919795+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:15.919795+0000","last_peered":"2024-09-18T11:45:15.919795+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:15.919795+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22394102,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":21870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2919","reported_seq":2975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.199847+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:19.199847+0000","last_peered":"2024-09-18T11:45:19.199847+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:19.199847+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2919,"log_dups_size":0,"ondisk_log_size":2919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23867249,"num_objects":2914,"num_object_clones":0,"num_object_copies":11656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2916,"num_write_kb":23316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2980","reported_seq":3015,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.248270+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:19.248270+0000","last_peered":"2024-09-18T11:45:19.248270+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:19.248270+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2980,"log_dups_size":0,"ondisk_log_size":2980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24402555,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2979,"num_write_kb":23831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2860","reported_seq":2936,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.827248+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:15.827248+0000","last_peered":"2024-09-18T11:45:15.827248+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:15.827248+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2860,"log_dups_size":0,"ondisk_log_size":2860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23419515,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3015","reported_seq":3064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.284326+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:19.284326+0000","last_peered":"2024-09-18T11:45:19.284326+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:19.284326+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24663286,"num_objects":3011,"num_object_clones":0,"num_object_copies":12044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3013,"num_write_kb":24100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":853}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2861","reported_seq":3772,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:15.923292+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:15.923292+0000","last_peered":"2024-09-18T11:45:15.923292+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:15.923292+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2861,"log_dups_size":0,"ondisk_log_size":2861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409910,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":3391488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2770","reported_seq":2832,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:16.043447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:16.043447+0000","last_peered":"2024-09-18T11:45:16.043447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:16.043447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648054,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":22132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":379913588,"num_objects":46381,"num_object_clones":0,"num_object_copies":185524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46408,"num_write_kb":371146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":3391488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626556928,"data_stored":626556928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46435,"ondisk_log_size":46435,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494008,"kb_used_data":187824,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93877832,"statfs":{"total":96636764160,"available":96130899968,"internally_reserved":0,"allocated":192331776,"data_stored":192172798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.254},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":5.133}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0640000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":7.5910000000000002},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":0.48899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488964,"kb_used_data":183292,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93882876,"statfs":{"total":96636764160,"available":96136065024,"internally_reserved":0,"allocated":187691008,"data_stored":187532030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":7.6050000000000004},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":7.5810000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":7.3979999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370232,"kb_used_data":55792,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":94001608,"statfs":{"total":96636764160,"available":96257646592,"internally_reserved":0,"allocated":57131008,"data_stored":56972030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,9,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4380000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489408,"kb_used_data":183288,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93882432,"statfs":{"total":96636764160,"available":96135610368,"internally_reserved":0,"allocated":187686912,"data_stored":187527934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,5,0,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":1.246},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.0069999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187674624,"data_stored":187674624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58327040,"data_stored":58327040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187437056,"data_stored":187437056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193118208,"data_stored":193118208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:21.217 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 64 20 47885 47865 5.84178 3.26562 0.030654 0.0267323 2024-09-18T11:45:22.217 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 65 20 48306 48286 5.80251 3.28906 0.0341699 0.0269126 2024-09-18T11:45:23.217 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 66 20 48734 48714 5.76526 3.34375 0.0513108 0.0270853 2024-09-18T11:45:23.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:23.906 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:23.938 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-18T11:45:23.843849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389309812,"num_objects":47528,"num_object_clones":0,"num_object_copies":190112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47555,"num_write_kb":380322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47582,"ondisk_log_size":47582,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1871088,"kb_used_data":626192,"kb_used_omap":14,"kb_used_meta":1244849,"kb_avail":375616272,"statfs":{"total":386547056640,"available":384631062528,"internally_reserved":0,"allocated":641220608,"data_stored":640584696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1274725403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,10,13,11,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33767424,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4122,"num_write_kb":32976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003977"},"pg_stats":[{"pgid":"1.f","version":"31'3047","reported_seq":3088,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.270210+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:19.270210+0000","last_peered":"2024-09-18T11:45:19.270210+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:19.270210+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3047,"log_dups_size":0,"ondisk_log_size":3047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24961024,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3047,"num_write_kb":24376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3001","reported_seq":3063,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.083186+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:21.083186+0000","last_peered":"2024-09-18T11:45:21.083186+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:21.083186+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3001,"log_dups_size":0,"ondisk_log_size":3001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24548598,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2999,"num_write_kb":23988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2972","reported_seq":3051,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.962410+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:20.962410+0000","last_peered":"2024-09-18T11:45:20.962410+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:20.962410+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24320635,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":915}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2981","reported_seq":3043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.909756+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:20.909756+0000","last_peered":"2024-09-18T11:45:20.909756+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:20.909756+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24384758,"num_objects":2977,"num_object_clones":0,"num_object_copies":11908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2979,"num_write_kb":23821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2950","reported_seq":3005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.315687+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:19.315687+0000","last_peered":"2024-09-18T11:45:19.315687+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:19.315687+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24138998,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2948,"num_write_kb":23581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2976","reported_seq":3019,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.298772+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:19.298772+0000","last_peered":"2024-09-18T11:45:19.298772+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:19.298772+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2976,"log_dups_size":0,"ondisk_log_size":2976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379392,"num_objects":2976,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3064","reported_seq":3129,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.026812+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:21.026812+0000","last_peered":"2024-09-18T11:45:21.026812+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:21.026812+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3064,"log_dups_size":0,"ondisk_log_size":3064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25048310,"num_objects":3058,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3062,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":924}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2933","reported_seq":2996,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.008756+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:21.008756+0000","last_peered":"2024-09-18T11:45:21.008756+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:21.008756+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2933,"log_dups_size":0,"ondisk_log_size":2933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23983350,"num_objects":2928,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2931,"num_write_kb":23436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3020","reported_seq":3078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.247324+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:19.247324+0000","last_peered":"2024-09-18T11:45:19.247324+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:19.247324+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24668652,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":877}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2871","reported_seq":2955,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.872280+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:20.872280+0000","last_peered":"2024-09-18T11:45:20.872280+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:20.872280+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2871,"log_dups_size":0,"ondisk_log_size":2871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23500022,"num_objects":2869,"num_object_clones":0,"num_object_copies":11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":22950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":894}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2919","reported_seq":2975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.199847+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:19.199847+0000","last_peered":"2024-09-18T11:45:19.199847+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:19.199847+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2919,"log_dups_size":0,"ondisk_log_size":2919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23867249,"num_objects":2914,"num_object_clones":0,"num_object_copies":11656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2916,"num_write_kb":23316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2980","reported_seq":3015,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.248270+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:19.248270+0000","last_peered":"2024-09-18T11:45:19.248270+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:19.248270+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2980,"log_dups_size":0,"ondisk_log_size":2980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24402555,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2979,"num_write_kb":23831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2978","reported_seq":3054,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.939201+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:20.939201+0000","last_peered":"2024-09-18T11:45:20.939201+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:20.939201+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24386171,"num_objects":2977,"num_object_clones":0,"num_object_copies":11908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2977,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":923}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3015","reported_seq":3064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.284326+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:19.284326+0000","last_peered":"2024-09-18T11:45:19.284326+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:19.284326+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24663286,"num_objects":3011,"num_object_clones":0,"num_object_copies":12044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3013,"num_write_kb":24100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":853}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2983","reported_seq":4048,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.953293+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:20.953293+0000","last_peered":"2024-09-18T11:45:20.953293+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:20.953293+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2983,"log_dups_size":0,"ondisk_log_size":2983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409334,"num_objects":2980,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2981,"num_write_kb":23845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1380}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2892","reported_seq":2954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.058209+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:21.058209+0000","last_peered":"2024-09-18T11:45:21.058209+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:21.058209+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2892,"log_dups_size":0,"ondisk_log_size":2892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23647478,"num_objects":2887,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2890,"num_write_kb":23108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":874}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389309812,"num_objects":47528,"num_object_clones":0,"num_object_copies":190112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47555,"num_write_kb":380322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":643080192,"data_stored":643080192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47582,"ondisk_log_size":47582,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494008,"kb_used_data":187824,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93877832,"statfs":{"total":96636764160,"available":96130899968,"internally_reserved":0,"allocated":192331776,"data_stored":192172798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.254},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":5.133}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0640000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":7.5910000000000002},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":0.48899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505272,"kb_used_data":191280,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93866568,"statfs":{"total":96636764160,"available":96119365632,"internally_reserved":0,"allocated":195870720,"data_stored":195711742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":2.8860000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":2.8540000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.752},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":0.38}]}]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370232,"kb_used_data":55792,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":94001608,"statfs":{"total":96636764160,"available":96257646592,"internally_reserved":0,"allocated":57131008,"data_stored":56972030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,9,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4380000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501576,"kb_used_data":191296,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93870264,"statfs":{"total":96636764160,"available":96123150336,"internally_reserved":0,"allocated":195887104,"data_stored":195728126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9390000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":8.2189999999999994},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":2.677}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.9109999999999996},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":9.3689999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195936256,"data_stored":195936256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58327040,"data_stored":58327040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195698688,"data_stored":195698688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193118208,"data_stored":193118208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:23.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:24.217 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 67 20 49108 49088 5.72279 2.92188 0.062678 0.0272848 2024-09-18T11:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:24.460 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-18T11:45:23.843849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389309812,"num_objects":47528,"num_object_clones":0,"num_object_copies":190112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47555,"num_write_kb":380322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47582,"ondisk_log_size":47582,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1871088,"kb_used_data":626192,"kb_used_omap":14,"kb_used_meta":1244849,"kb_avail":375616272,"statfs":{"total":386547056640,"available":384631062528,"internally_reserved":0,"allocated":641220608,"data_stored":640584696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1274725403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,10,13,11,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33767424,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4122,"num_write_kb":32976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003977"},"pg_stats":[{"pgid":"1.f","version":"31'3047","reported_seq":3088,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.270210+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:19.270210+0000","last_peered":"2024-09-18T11:45:19.270210+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:19.270210+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3047,"log_dups_size":0,"ondisk_log_size":3047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24961024,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3047,"num_write_kb":24376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3001","reported_seq":3063,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.083186+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:21.083186+0000","last_peered":"2024-09-18T11:45:21.083186+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:21.083186+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3001,"log_dups_size":0,"ondisk_log_size":3001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24548598,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2999,"num_write_kb":23988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2972","reported_seq":3051,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.962410+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:20.962410+0000","last_peered":"2024-09-18T11:45:20.962410+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:20.962410+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24320635,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":915}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2981","reported_seq":3043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.909756+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:20.909756+0000","last_peered":"2024-09-18T11:45:20.909756+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:20.909756+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24384758,"num_objects":2977,"num_object_clones":0,"num_object_copies":11908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2979,"num_write_kb":23821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2950","reported_seq":3005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.315687+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:19.315687+0000","last_peered":"2024-09-18T11:45:19.315687+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:19.315687+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24138998,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2948,"num_write_kb":23581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2976","reported_seq":3019,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.298772+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:19.298772+0000","last_peered":"2024-09-18T11:45:19.298772+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:19.298772+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2976,"log_dups_size":0,"ondisk_log_size":2976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379392,"num_objects":2976,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3064","reported_seq":3129,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.026812+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:21.026812+0000","last_peered":"2024-09-18T11:45:21.026812+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:21.026812+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3064,"log_dups_size":0,"ondisk_log_size":3064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25048310,"num_objects":3058,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3062,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":924}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2933","reported_seq":2996,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.008756+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:21.008756+0000","last_peered":"2024-09-18T11:45:21.008756+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:21.008756+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2933,"log_dups_size":0,"ondisk_log_size":2933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23983350,"num_objects":2928,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2931,"num_write_kb":23436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3020","reported_seq":3078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.247324+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:19.247324+0000","last_peered":"2024-09-18T11:45:19.247324+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:19.247324+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24668652,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":877}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2871","reported_seq":2955,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.872280+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:20.872280+0000","last_peered":"2024-09-18T11:45:20.872280+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:20.872280+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2871,"log_dups_size":0,"ondisk_log_size":2871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23500022,"num_objects":2869,"num_object_clones":0,"num_object_copies":11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":22950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":894}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2919","reported_seq":2975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.199847+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:19.199847+0000","last_peered":"2024-09-18T11:45:19.199847+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:19.199847+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2919,"log_dups_size":0,"ondisk_log_size":2919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23867249,"num_objects":2914,"num_object_clones":0,"num_object_copies":11656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2916,"num_write_kb":23316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2980","reported_seq":3015,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.248270+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:19.248270+0000","last_peered":"2024-09-18T11:45:19.248270+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:19.248270+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2980,"log_dups_size":0,"ondisk_log_size":2980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24402555,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2979,"num_write_kb":23831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2978","reported_seq":3054,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.939201+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:20.939201+0000","last_peered":"2024-09-18T11:45:20.939201+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:20.939201+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24386171,"num_objects":2977,"num_object_clones":0,"num_object_copies":11908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2977,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":923}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3015","reported_seq":3064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.284326+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:19.284326+0000","last_peered":"2024-09-18T11:45:19.284326+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:19.284326+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24663286,"num_objects":3011,"num_object_clones":0,"num_object_copies":12044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3013,"num_write_kb":24100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":853}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2983","reported_seq":4048,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.953293+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:20.953293+0000","last_peered":"2024-09-18T11:45:20.953293+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:20.953293+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2983,"log_dups_size":0,"ondisk_log_size":2983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409334,"num_objects":2980,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2981,"num_write_kb":23845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1380}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2892","reported_seq":2954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.058209+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:21.058209+0000","last_peered":"2024-09-18T11:45:21.058209+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:21.058209+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2892,"log_dups_size":0,"ondisk_log_size":2892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23647478,"num_objects":2887,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2890,"num_write_kb":23108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":874}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389309812,"num_objects":47528,"num_object_clones":0,"num_object_copies":190112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47555,"num_write_kb":380322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":643080192,"data_stored":643080192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47582,"ondisk_log_size":47582,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494008,"kb_used_data":187824,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93877832,"statfs":{"total":96636764160,"available":96130899968,"internally_reserved":0,"allocated":192331776,"data_stored":192172798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.254},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":5.133}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0640000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":7.5910000000000002},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":0.48899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505272,"kb_used_data":191280,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93866568,"statfs":{"total":96636764160,"available":96119365632,"internally_reserved":0,"allocated":195870720,"data_stored":195711742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":2.8860000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":2.8540000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.752},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":0.38}]}]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370232,"kb_used_data":55792,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":94001608,"statfs":{"total":96636764160,"available":96257646592,"internally_reserved":0,"allocated":57131008,"data_stored":56972030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,9,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4380000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501576,"kb_used_data":191296,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93870264,"statfs":{"total":96636764160,"available":96123150336,"internally_reserved":0,"allocated":195887104,"data_stored":195728126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9390000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":8.2189999999999994},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":2.677}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.9109999999999996},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":9.3689999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195936256,"data_stored":195936256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58327040,"data_stored":58327040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195698688,"data_stored":195698688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193118208,"data_stored":193118208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:24.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:24.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3047", 'reported_seq': 3088, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:19.270210+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:19.270210+0000', 'last_peered': '2024-09-18T11:45:19.270210+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:19.270210+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3047, 'log_dups_size': 0, 'ondisk_log_size': 3047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24961024, 'num_objects': 3047, 'num_object_clones': 0, 'num_object_copies': 12188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3047, 'num_write_kb': 24376, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 917}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:24.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:24.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3001", 'reported_seq': 3063, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:21.083186+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:21.083186+0000', 'last_peered': '2024-09-18T11:45:21.083186+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:21.083186+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3001, 'log_dups_size': 0, 'ondisk_log_size': 3001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24548598, 'num_objects': 2997, 'num_object_clones': 0, 'num_object_copies': 11988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2999, 'num_write_kb': 23988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 895}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:24.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:24.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2972", 'reported_seq': 3051, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:20.962410+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:20.962410+0000', 'last_peered': '2024-09-18T11:45:20.962410+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:20.962410+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2972, 'log_dups_size': 0, 'ondisk_log_size': 2972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24320635, 'num_objects': 2969, 'num_object_clones': 0, 'num_object_copies': 11876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2971, 'num_write_kb': 23758, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 915}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:24.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:24.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2981", 'reported_seq': 3043, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:20.909756+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:20.909756+0000', 'last_peered': '2024-09-18T11:45:20.909756+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:20.909756+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2981, 'log_dups_size': 0, 'ondisk_log_size': 2981, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24384758, 'num_objects': 2977, 'num_object_clones': 0, 'num_object_copies': 11908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2979, 'num_write_kb': 23821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 863}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:24.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:24.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2950", 'reported_seq': 3005, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:19.315687+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:19.315687+0000', 'last_peered': '2024-09-18T11:45:19.315687+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:19.315687+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2950, 'log_dups_size': 0, 'ondisk_log_size': 2950, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24138998, 'num_objects': 2947, 'num_object_clones': 0, 'num_object_copies': 11788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2948, 'num_write_kb': 23581, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 865}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:24.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:24.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2976", 'reported_seq': 3019, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:19.298772+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:19.298772+0000', 'last_peered': '2024-09-18T11:45:19.298772+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:19.298772+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2976, 'log_dups_size': 0, 'ondisk_log_size': 2976, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24379392, 'num_objects': 2976, 'num_object_clones': 0, 'num_object_copies': 11904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2976, 'num_write_kb': 23808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 822}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:24.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:24.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3064", 'reported_seq': 3129, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:21.026812+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:21.026812+0000', 'last_peered': '2024-09-18T11:45:21.026812+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:21.026812+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3064, 'log_dups_size': 0, 'ondisk_log_size': 3064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25048310, 'num_objects': 3058, 'num_object_clones': 0, 'num_object_copies': 12232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3062, 'num_write_kb': 24476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 924}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:24.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:24.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2933", 'reported_seq': 2996, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:21.008756+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:21.008756+0000', 'last_peered': '2024-09-18T11:45:21.008756+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:21.008756+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2933, 'log_dups_size': 0, 'ondisk_log_size': 2933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23983350, 'num_objects': 2928, 'num_object_clones': 0, 'num_object_copies': 11712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2931, 'num_write_kb': 23436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 894}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:24.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:24.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3020", 'reported_seq': 3078, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:19.247324+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:19.247324+0000', 'last_peered': '2024-09-18T11:45:19.247324+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:19.247324+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3020, 'log_dups_size': 0, 'ondisk_log_size': 3020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24668652, 'num_objects': 3012, 'num_object_clones': 0, 'num_object_copies': 12048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3017, 'num_write_kb': 24113, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 877}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:24.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:24.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2871", 'reported_seq': 2955, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:20.872280+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:20.872280+0000', 'last_peered': '2024-09-18T11:45:20.872280+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:20.872280+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2871, 'log_dups_size': 0, 'ondisk_log_size': 2871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23500022, 'num_objects': 2869, 'num_object_clones': 0, 'num_object_copies': 11476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2869, 'num_write_kb': 22950, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 894}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:24.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:24.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2919", 'reported_seq': 2975, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:19.199847+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:19.199847+0000', 'last_peered': '2024-09-18T11:45:19.199847+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:19.199847+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2919, 'log_dups_size': 0, 'ondisk_log_size': 2919, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23867249, 'num_objects': 2914, 'num_object_clones': 0, 'num_object_copies': 11656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2914, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2916, 'num_write_kb': 23316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 851}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:24.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:24.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2980", 'reported_seq': 3015, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:19.248270+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:19.248270+0000', 'last_peered': '2024-09-18T11:45:19.248270+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:19.248270+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2980, 'log_dups_size': 0, 'ondisk_log_size': 2980, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24402555, 'num_objects': 2979, 'num_object_clones': 0, 'num_object_copies': 11916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2979, 'num_write_kb': 23831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 870}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:24.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:24.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2978", 'reported_seq': 3054, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:20.939201+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:20.939201+0000', 'last_peered': '2024-09-18T11:45:20.939201+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:20.939201+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2978, 'log_dups_size': 0, 'ondisk_log_size': 2978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24386171, 'num_objects': 2977, 'num_object_clones': 0, 'num_object_copies': 11908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2977, 'num_write_kb': 23815, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 923}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:24.486 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:24.486 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3015", 'reported_seq': 3064, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:19.284326+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:19.284326+0000', 'last_peered': '2024-09-18T11:45:19.284326+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:19.284326+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3015, 'log_dups_size': 0, 'ondisk_log_size': 3015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24663286, 'num_objects': 3011, 'num_object_clones': 0, 'num_object_copies': 12044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3013, 'num_write_kb': 24100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 853}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:24.486 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:24.486 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2983", 'reported_seq': 4048, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:20.953293+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:20.953293+0000', 'last_peered': '2024-09-18T11:45:20.953293+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:20.953293+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2983, 'log_dups_size': 0, 'ondisk_log_size': 2983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24409334, 'num_objects': 2980, 'num_object_clones': 0, 'num_object_copies': 11920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1600, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2981, 'num_write_kb': 23845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 493, 'num_bytes_recovered': 4022272, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1380}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:24.486 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:24.486 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2892", 'reported_seq': 2954, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:21.058209+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:21.058209+0000', 'last_peered': '2024-09-18T11:45:21.058209+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:21.058209+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2892, 'log_dups_size': 0, 'ondisk_log_size': 2892, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23647478, 'num_objects': 2887, 'num_object_clones': 0, 'num_object_copies': 11548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2890, 'num_write_kb': 23108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 874}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:24.486 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:25.172 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:25.172 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32936/190112 objects degraded (17.325%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":123,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47528,"data_bytes":389309812,"bytes_used":1915994112,"bytes_avail":384631062528,"bytes_total":386547056640,"degraded_objects":32936,"degraded_total":190112,"degraded_ratio":0.1732452449082646,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":98953,"recovering_keys_per_sec":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"write_bytes_sec":2813019,"read_op_per_sec":0,"write_op_per_sec":343},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T11:45:05.837774+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:45:25.203 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:25.203 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:25.217 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 68 20 49474 49454 5.68068 2.85938 0.0576916 0.0274889 2024-09-18T11:45:25.722 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:25.722 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:25.745 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-18T11:45:23.843849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389309812,"num_objects":47528,"num_object_clones":0,"num_object_copies":190112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47555,"num_write_kb":380322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47582,"ondisk_log_size":47582,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1871088,"kb_used_data":626192,"kb_used_omap":14,"kb_used_meta":1244849,"kb_avail":375616272,"statfs":{"total":386547056640,"available":384631062528,"internally_reserved":0,"allocated":641220608,"data_stored":640584696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1274725403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,10,13,11,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33767424,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4122,"num_write_kb":32976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003977"},"pg_stats":[{"pgid":"1.f","version":"31'3047","reported_seq":3088,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.270210+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:19.270210+0000","last_peered":"2024-09-18T11:45:19.270210+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:19.270210+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3047,"log_dups_size":0,"ondisk_log_size":3047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24961024,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3047,"num_write_kb":24376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3001","reported_seq":3063,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.083186+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:21.083186+0000","last_peered":"2024-09-18T11:45:21.083186+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:21.083186+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3001,"log_dups_size":0,"ondisk_log_size":3001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24548598,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2999,"num_write_kb":23988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2972","reported_seq":3051,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.962410+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:20.962410+0000","last_peered":"2024-09-18T11:45:20.962410+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:20.962410+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24320635,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":915}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2981","reported_seq":3043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.909756+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:20.909756+0000","last_peered":"2024-09-18T11:45:20.909756+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:20.909756+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24384758,"num_objects":2977,"num_object_clones":0,"num_object_copies":11908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2979,"num_write_kb":23821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2950","reported_seq":3005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.315687+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:19.315687+0000","last_peered":"2024-09-18T11:45:19.315687+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:19.315687+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24138998,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2948,"num_write_kb":23581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2976","reported_seq":3019,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.298772+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:19.298772+0000","last_peered":"2024-09-18T11:45:19.298772+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:19.298772+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2976,"log_dups_size":0,"ondisk_log_size":2976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379392,"num_objects":2976,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3064","reported_seq":3129,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.026812+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:21.026812+0000","last_peered":"2024-09-18T11:45:21.026812+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:21.026812+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3064,"log_dups_size":0,"ondisk_log_size":3064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25048310,"num_objects":3058,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3062,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":924}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2933","reported_seq":2996,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.008756+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:21.008756+0000","last_peered":"2024-09-18T11:45:21.008756+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:21.008756+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2933,"log_dups_size":0,"ondisk_log_size":2933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23983350,"num_objects":2928,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2931,"num_write_kb":23436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3020","reported_seq":3078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.247324+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:19.247324+0000","last_peered":"2024-09-18T11:45:19.247324+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:19.247324+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24668652,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":877}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2871","reported_seq":2955,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.872280+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:20.872280+0000","last_peered":"2024-09-18T11:45:20.872280+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:20.872280+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2871,"log_dups_size":0,"ondisk_log_size":2871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23500022,"num_objects":2869,"num_object_clones":0,"num_object_copies":11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":22950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":894}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2919","reported_seq":2975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.199847+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:19.199847+0000","last_peered":"2024-09-18T11:45:19.199847+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:19.199847+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2919,"log_dups_size":0,"ondisk_log_size":2919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23867249,"num_objects":2914,"num_object_clones":0,"num_object_copies":11656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2916,"num_write_kb":23316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2980","reported_seq":3015,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.248270+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:19.248270+0000","last_peered":"2024-09-18T11:45:19.248270+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:19.248270+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2980,"log_dups_size":0,"ondisk_log_size":2980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24402555,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2979,"num_write_kb":23831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2978","reported_seq":3054,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.939201+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:20.939201+0000","last_peered":"2024-09-18T11:45:20.939201+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:20.939201+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24386171,"num_objects":2977,"num_object_clones":0,"num_object_copies":11908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2977,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":923}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3015","reported_seq":3064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:19.284326+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:19.284326+0000","last_peered":"2024-09-18T11:45:19.284326+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:19.284326+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24663286,"num_objects":3011,"num_object_clones":0,"num_object_copies":12044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3013,"num_write_kb":24100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":853}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2983","reported_seq":4048,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:20.953293+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:20.953293+0000","last_peered":"2024-09-18T11:45:20.953293+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:20.953293+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2983,"log_dups_size":0,"ondisk_log_size":2983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409334,"num_objects":2980,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2981,"num_write_kb":23845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1380}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2892","reported_seq":2954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:21.058209+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:21.058209+0000","last_peered":"2024-09-18T11:45:21.058209+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:21.058209+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2892,"log_dups_size":0,"ondisk_log_size":2892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23647478,"num_objects":2887,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2890,"num_write_kb":23108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":874}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389309812,"num_objects":47528,"num_object_clones":0,"num_object_copies":190112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47555,"num_write_kb":380322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":643080192,"data_stored":643080192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47582,"ondisk_log_size":47582,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494008,"kb_used_data":187824,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93877832,"statfs":{"total":96636764160,"available":96130899968,"internally_reserved":0,"allocated":192331776,"data_stored":192172798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.254},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":5.133}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0640000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":7.5910000000000002},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":0.48899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505272,"kb_used_data":191280,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93866568,"statfs":{"total":96636764160,"available":96119365632,"internally_reserved":0,"allocated":195870720,"data_stored":195711742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":2.8860000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":2.8540000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.752},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":0.38}]}]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370232,"kb_used_data":55792,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":94001608,"statfs":{"total":96636764160,"available":96257646592,"internally_reserved":0,"allocated":57131008,"data_stored":56972030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,9,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4380000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501576,"kb_used_data":191296,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93870264,"statfs":{"total":96636764160,"available":96123150336,"internally_reserved":0,"allocated":195887104,"data_stored":195728126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9390000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":8.2189999999999994},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":2.677}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.9109999999999996},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":9.3689999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195936256,"data_stored":195936256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58327040,"data_stored":58327040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195698688,"data_stored":195698688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193118208,"data_stored":193118208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:26.218 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 69 20 49865 49845 5.64262 3.05469 0.0445641 0.0276773 2024-09-18T11:45:27.218 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 70 19 50278 50259 5.60822 3.23438 0.0287194 0.0278445 2024-09-18T11:45:28.218 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 71 20 50697 50677 5.57523 3.26562 0.0571562 0.0280102 2024-09-18T11:45:28.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:29.218 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 72 20 51087 51067 5.54011 3.04688 0.0743944 0.0281859 2024-09-18T11:45:29.272 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:29.272 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:29.301 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-18T11:45:27.844899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405587316,"num_objects":49515,"num_object_clones":0,"num_object_copies":198060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49542,"num_write_kb":396218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49569,"ondisk_log_size":49569,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1868764,"kb_used_data":657148,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375618596,"statfs":{"total":386547056640,"available":384633442304,"internally_reserved":0,"allocated":672919552,"data_stored":672283640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,22,15,13,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42344448,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":41352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004013"},"pg_stats":[{"pgid":"1.f","version":"31'3175","reported_seq":3216,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:24.318146+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:24.318146+0000","last_peered":"2024-09-18T11:45:24.318146+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:24.318146+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3175,"log_dups_size":0,"ondisk_log_size":3175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26009600,"num_objects":3175,"num_object_clones":0,"num_object_copies":12700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3175,"num_write_kb":25400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1045}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3117","reported_seq":3180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.960182+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:25.960182+0000","last_peered":"2024-09-18T11:45:25.960182+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:25.960182+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498870,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3115,"num_write_kb":24916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1011}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3093","reported_seq":3175,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.840899+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:25.840899+0000","last_peered":"2024-09-18T11:45:25.840899+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:25.840899+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3093,"log_dups_size":0,"ondisk_log_size":3093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25311867,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3124","reported_seq":3186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.032989+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:26.032989+0000","last_peered":"2024-09-18T11:45:26.032989+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:26.032989+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3124,"log_dups_size":0,"ondisk_log_size":3124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25556214,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3122,"num_write_kb":24965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1006}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3072","reported_seq":3127,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:24.166587+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:24.166587+0000","last_peered":"2024-09-18T11:45:24.166587+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:24.166587+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3072,"log_dups_size":0,"ondisk_log_size":3072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138422,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3070,"num_write_kb":24557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3099","reported_seq":3142,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:24.230045+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:24.230045+0000","last_peered":"2024-09-18T11:45:24.230045+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:24.230045+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3099,"log_dups_size":0,"ondisk_log_size":3099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25387008,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3099,"num_write_kb":24792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":945}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3186","reported_seq":3250,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.050862+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:26.050862+0000","last_peered":"2024-09-18T11:45:26.050862+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:26.050862+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26047734,"num_objects":3180,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3184,"num_write_kb":25452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1046}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3051","reported_seq":3114,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.085927+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:26.085927+0000","last_peered":"2024-09-18T11:45:26.085927+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:26.085927+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24950006,"num_objects":3046,"num_object_clones":0,"num_object_copies":12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":24380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1012}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3140","reported_seq":3198,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:24.176276+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:24.176276+0000","last_peered":"2024-09-18T11:45:24.176276+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:24.176276+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3140,"log_dups_size":0,"ondisk_log_size":3140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25651692,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":25073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2992","reported_seq":3076,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.920920+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:25.920920+0000","last_peered":"2024-09-18T11:45:25.920920+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:25.920920+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2992,"log_dups_size":0,"ondisk_log_size":2992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24491254,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2990,"num_write_kb":23918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1015}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3043","reported_seq":3100,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:24.306588+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:24.306588+0000","last_peered":"2024-09-18T11:45:24.306588+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:24.306588+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3043,"log_dups_size":0,"ondisk_log_size":3043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24883057,"num_objects":3038,"num_object_clones":0,"num_object_copies":12152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":24308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":975}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3113","reported_seq":3150,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:24.320709+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:24.320709+0000","last_peered":"2024-09-18T11:45:24.320709+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:24.320709+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25492091,"num_objects":3112,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3112,"num_write_kb":24895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3097","reported_seq":3173,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.824442+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:25.824442+0000","last_peered":"2024-09-18T11:45:25.824442+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:25.824442+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3097,"log_dups_size":0,"ondisk_log_size":3097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25361019,"num_objects":3096,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3096,"num_write_kb":24767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3135","reported_seq":3181,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:24.273375+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:24.273375+0000","last_peered":"2024-09-18T11:45:24.273375+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:24.273375+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25646326,"num_objects":3131,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3133,"num_write_kb":25060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3115","reported_seq":4328,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.967422+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:25.967422+0000","last_peered":"2024-09-18T11:45:25.967422+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:25.967422+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25490678,"num_objects":3112,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":567,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1586}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3017","reported_seq":3079,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.003458+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:26.003458+0000","last_peered":"2024-09-18T11:45:26.003458+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:26.003458+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3017,"log_dups_size":0,"ondisk_log_size":3017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24671478,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3015,"num_write_kb":24108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":405587316,"num_objects":49515,"num_object_clones":0,"num_object_copies":198060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49542,"num_write_kb":396218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":676143104,"data_stored":676143104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49569,"ondisk_log_size":49569,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506252,"kb_used_data":195908,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93865588,"statfs":{"total":96636764160,"available":96118362112,"internally_reserved":0,"allocated":200609792,"data_stored":200450814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,4,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":4.0830000000000002},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":31.675000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.68}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":5.0129999999999999},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":4.109}]}]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499284,"kb_used_data":197772,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93872556,"statfs":{"total":96636764160,"available":96125497344,"internally_reserved":0,"allocated":202518528,"data_stored":202359550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,11,5,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":12.529999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":18.047000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.427}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":12.5},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":11.362}]}]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":366164,"kb_used_data":64204,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94005676,"statfs":{"total":96636764160,"available":96261812224,"internally_reserved":0,"allocated":65744896,"data_stored":65585918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2570000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0469999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4169999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497064,"kb_used_data":199264,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93874776,"statfs":{"total":96636764160,"available":96127770624,"internally_reserved":0,"allocated":204046336,"data_stored":203887358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1749999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":1.988},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.774}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":2.0179999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204054528,"data_stored":204054528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66899968,"data_stored":66899968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203808768,"data_stored":203808768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":201379840,"data_stored":201379840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:29.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:29.852 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:29.852 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:29.881 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-18T11:45:29.845352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":412960116,"num_objects":50415,"num_object_clones":0,"num_object_copies":201660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50442,"num_write_kb":403418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50469,"ondisk_log_size":50469,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1876584,"kb_used_data":673288,"kb_used_omap":14,"kb_used_meta":1203249,"kb_avail":375610776,"statfs":{"total":386547056640,"available":384625434624,"internally_reserved":0,"allocated":689446912,"data_stored":688811000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232127003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,22,11,20,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40042496,"num_objects":4888,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003994"},"pg_stats":[{"pgid":"1.f","version":"31'3296","reported_seq":3337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.341668+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:29.341668+0000","last_peered":"2024-09-18T11:45:29.341668+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:29.341668+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27000832,"num_objects":3296,"num_object_clones":0,"num_object_copies":13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3117","reported_seq":3180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.960182+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:25.960182+0000","last_peered":"2024-09-18T11:45:25.960182+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:25.960182+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498870,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3115,"num_write_kb":24916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1011}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3093","reported_seq":3175,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.840899+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:25.840899+0000","last_peered":"2024-09-18T11:45:25.840899+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:25.840899+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3093,"log_dups_size":0,"ondisk_log_size":3093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25311867,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3124","reported_seq":3186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.032989+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:26.032989+0000","last_peered":"2024-09-18T11:45:26.032989+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:26.032989+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3124,"log_dups_size":0,"ondisk_log_size":3124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25556214,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3122,"num_write_kb":24965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1006}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3205","reported_seq":3260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.226820+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:29.226820+0000","last_peered":"2024-09-18T11:45:29.226820+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:29.226820+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26227958,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":25621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3220","reported_seq":3263,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.263969+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:29.263969+0000","last_peered":"2024-09-18T11:45:29.263969+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:29.263969+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26378240,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":25760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3186","reported_seq":3250,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.050862+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:26.050862+0000","last_peered":"2024-09-18T11:45:26.050862+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:26.050862+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26047734,"num_objects":3180,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3184,"num_write_kb":25452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1046}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3051","reported_seq":3114,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.085927+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:26.085927+0000","last_peered":"2024-09-18T11:45:26.085927+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:26.085927+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24950006,"num_objects":3046,"num_object_clones":0,"num_object_copies":12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":24380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1012}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3298","reported_seq":3356,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.318234+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:29.318234+0000","last_peered":"2024-09-18T11:45:29.318234+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:29.318234+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26946028,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2992","reported_seq":3076,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.920920+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:25.920920+0000","last_peered":"2024-09-18T11:45:25.920920+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:25.920920+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2992,"log_dups_size":0,"ondisk_log_size":2992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24491254,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2990,"num_write_kb":23918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1015}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3156","reported_seq":3212,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.156216+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:29.156216+0000","last_peered":"2024-09-18T11:45:29.156216+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:29.156216+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3156,"log_dups_size":0,"ondisk_log_size":3156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25808753,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3235","reported_seq":3272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.274841+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:29.274841+0000","last_peered":"2024-09-18T11:45:29.274841+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:29.274841+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3235,"log_dups_size":0,"ondisk_log_size":3235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26491515,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3234,"num_write_kb":25871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3097","reported_seq":3173,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.824442+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:25.824442+0000","last_peered":"2024-09-18T11:45:25.824442+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:25.824442+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3097,"log_dups_size":0,"ondisk_log_size":3097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25361019,"num_objects":3096,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3096,"num_write_kb":24767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3267","reported_seq":3316,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.360949+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:29.360949+0000","last_peered":"2024-09-18T11:45:29.360949+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:29.360949+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26727670,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":26116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3115","reported_seq":4328,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.967422+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:25.967422+0000","last_peered":"2024-09-18T11:45:25.967422+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:25.967422+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25490678,"num_objects":3112,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":567,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1586}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3017","reported_seq":3079,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.003458+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:26.003458+0000","last_peered":"2024-09-18T11:45:26.003458+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:26.003458+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3017,"log_dups_size":0,"ondisk_log_size":3017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24671478,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3015,"num_write_kb":24108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":412960116,"num_objects":50415,"num_object_clones":0,"num_object_copies":201660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50442,"num_write_kb":403418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":692658176,"data_stored":692658176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50469,"ondisk_log_size":50469,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501696,"kb_used_data":203832,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93870144,"statfs":{"total":96636764160,"available":96123027456,"internally_reserved":0,"allocated":208723968,"data_stored":208564990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,3,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":9.5199999999999996},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":2.2519999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":5.3070000000000004},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":6.5229999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499284,"kb_used_data":197772,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93872556,"statfs":{"total":96636764160,"available":96125497344,"internally_reserved":0,"allocated":202518528,"data_stored":202359550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,11,5,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":12.529999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":18.047000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.427}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":12.5},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":11.362}]}]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378540,"kb_used_data":72420,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93993300,"statfs":{"total":96636764160,"available":96249139200,"internally_reserved":0,"allocated":74158080,"data_stored":73999102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.532}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.223}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.258}]}]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497064,"kb_used_data":199264,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93874776,"statfs":{"total":96636764160,"available":96127770624,"internally_reserved":0,"allocated":204046336,"data_stored":203887358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1749999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":1.988},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.774}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":2.0179999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204054528,"data_stored":204054528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75296768,"data_stored":75296768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203808768,"data_stored":203808768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209498112,"data_stored":209498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:29.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:29.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3296", 'reported_seq': 3337, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.341668+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:29.341668+0000', 'last_peered': '2024-09-18T11:45:29.341668+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:29.341668+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3296, 'log_dups_size': 0, 'ondisk_log_size': 3296, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27000832, 'num_objects': 3296, 'num_object_clones': 0, 'num_object_copies': 13184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3296, 'num_write_kb': 26368, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1166}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:29.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:29.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3117", 'reported_seq': 3180, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:25.960182+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:25.960182+0000', 'last_peered': '2024-09-18T11:45:25.960182+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:25.960182+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3117, 'log_dups_size': 0, 'ondisk_log_size': 3117, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25498870, 'num_objects': 3113, 'num_object_clones': 0, 'num_object_copies': 12452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3115, 'num_write_kb': 24916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:29.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:29.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3093", 'reported_seq': 3175, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:25.840899+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:25.840899+0000', 'last_peered': '2024-09-18T11:45:25.840899+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:25.840899+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3093, 'log_dups_size': 0, 'ondisk_log_size': 3093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25311867, 'num_objects': 3090, 'num_object_clones': 0, 'num_object_copies': 12360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3092, 'num_write_kb': 24726, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1036}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:29.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:29.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3124", 'reported_seq': 3186, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:26.032989+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:26.032989+0000', 'last_peered': '2024-09-18T11:45:26.032989+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:26.032989+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3124, 'log_dups_size': 0, 'ondisk_log_size': 3124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25556214, 'num_objects': 3120, 'num_object_clones': 0, 'num_object_copies': 12480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3122, 'num_write_kb': 24965, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1006}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:29.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:29.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3205", 'reported_seq': 3260, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.226820+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:29.226820+0000', 'last_peered': '2024-09-18T11:45:29.226820+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:29.226820+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3205, 'log_dups_size': 0, 'ondisk_log_size': 3205, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26227958, 'num_objects': 3202, 'num_object_clones': 0, 'num_object_copies': 12808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3203, 'num_write_kb': 25621, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1120}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:29.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:29.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3220", 'reported_seq': 3263, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.263969+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:29.263969+0000', 'last_peered': '2024-09-18T11:45:29.263969+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:29.263969+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3220, 'log_dups_size': 0, 'ondisk_log_size': 3220, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26378240, 'num_objects': 3220, 'num_object_clones': 0, 'num_object_copies': 12880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3220, 'num_write_kb': 25760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1066}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:29.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:29.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3186", 'reported_seq': 3250, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:26.050862+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:26.050862+0000', 'last_peered': '2024-09-18T11:45:26.050862+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:26.050862+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3186, 'log_dups_size': 0, 'ondisk_log_size': 3186, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26047734, 'num_objects': 3180, 'num_object_clones': 0, 'num_object_copies': 12720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3184, 'num_write_kb': 25452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1046}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:29.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:29.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3051", 'reported_seq': 3114, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:26.085927+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:26.085927+0000', 'last_peered': '2024-09-18T11:45:26.085927+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:26.085927+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3051, 'log_dups_size': 0, 'ondisk_log_size': 3051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24950006, 'num_objects': 3046, 'num_object_clones': 0, 'num_object_copies': 12184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3049, 'num_write_kb': 24380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1012}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:29.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:29.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3298", 'reported_seq': 3356, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.318234+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:29.318234+0000', 'last_peered': '2024-09-18T11:45:29.318234+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:29.318234+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3298, 'log_dups_size': 0, 'ondisk_log_size': 3298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26946028, 'num_objects': 3290, 'num_object_clones': 0, 'num_object_copies': 13160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3295, 'num_write_kb': 26337, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1155}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:29.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:29.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2992", 'reported_seq': 3076, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:25.920920+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:25.920920+0000', 'last_peered': '2024-09-18T11:45:25.920920+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:25.920920+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 2992, 'log_dups_size': 0, 'ondisk_log_size': 2992, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24491254, 'num_objects': 2990, 'num_object_clones': 0, 'num_object_copies': 11960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2990, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2990, 'num_write_kb': 23918, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1015}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3156", 'reported_seq': 3212, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.156216+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:29.156216+0000', 'last_peered': '2024-09-18T11:45:29.156216+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:29.156216+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3156, 'log_dups_size': 0, 'ondisk_log_size': 3156, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25808753, 'num_objects': 3151, 'num_object_clones': 0, 'num_object_copies': 12604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3153, 'num_write_kb': 25212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1088}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3235", 'reported_seq': 3272, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.274841+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:29.274841+0000', 'last_peered': '2024-09-18T11:45:29.274841+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:29.274841+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3235, 'log_dups_size': 0, 'ondisk_log_size': 3235, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26491515, 'num_objects': 3234, 'num_object_clones': 0, 'num_object_copies': 12936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3234, 'num_write_kb': 25871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3097", 'reported_seq': 3173, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:25.824442+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:25.824442+0000', 'last_peered': '2024-09-18T11:45:25.824442+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:25.824442+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3097, 'log_dups_size': 0, 'ondisk_log_size': 3097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25361019, 'num_objects': 3096, 'num_object_clones': 0, 'num_object_copies': 12384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3096, 'num_write_kb': 24767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3267", 'reported_seq': 3316, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.360949+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:29.360949+0000', 'last_peered': '2024-09-18T11:45:29.360949+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:29.360949+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3267, 'log_dups_size': 0, 'ondisk_log_size': 3267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26727670, 'num_objects': 3263, 'num_object_clones': 0, 'num_object_copies': 13052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3265, 'num_write_kb': 26116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3115", 'reported_seq': 4328, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:25.967422+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:25.967422+0000', 'last_peered': '2024-09-18T11:45:25.967422+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:25.967422+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3115, 'log_dups_size': 0, 'ondisk_log_size': 3115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25490678, 'num_objects': 3112, 'num_object_clones': 0, 'num_object_copies': 12448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3113, 'num_write_kb': 24901, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 567, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:29.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:29.886 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3017", 'reported_seq': 3079, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:26.003458+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:26.003458+0000', 'last_peered': '2024-09-18T11:45:26.003458+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:26.003458+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3017, 'log_dups_size': 0, 'ondisk_log_size': 3017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24671478, 'num_objects': 3012, 'num_object_clones': 0, 'num_object_copies': 12048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3015, 'num_write_kb': 24108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 999}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:29.886 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:30.218 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 73 20 51445 51425 5.50253 2.79688 0.0312249 0.0283827 2024-09-18T11:45:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:30.543 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32862/201660 objects degraded (16.296%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":128,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":50415,"data_bytes":412960116,"bytes_used":1921622016,"bytes_avail":384625434624,"bytes_total":386547056640,"degraded_objects":32862,"degraded_total":201660,"degraded_ratio":0.16295745313894675,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":103048,"recovering_keys_per_sec":0,"num_objects_recovered":151,"num_bytes_recovered":1236992,"num_keys_recovered":0,"write_bytes_sec":3335764,"read_op_per_sec":0,"write_op_per_sec":407},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T11:45:05.837774+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:45:30.573 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:30.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:31.215 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:31.215 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:31.218 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 74 20 51773 51753 5.4628 2.5625 0.0397685 0.0285819 2024-09-18T11:45:31.236 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-18T11:45:29.845352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":412960116,"num_objects":50415,"num_object_clones":0,"num_object_copies":201660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50442,"num_write_kb":403418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50469,"ondisk_log_size":50469,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1876584,"kb_used_data":673288,"kb_used_omap":14,"kb_used_meta":1203249,"kb_avail":375610776,"statfs":{"total":386547056640,"available":384625434624,"internally_reserved":0,"allocated":689446912,"data_stored":688811000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232127003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,22,11,20,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40042496,"num_objects":4888,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003994"},"pg_stats":[{"pgid":"1.f","version":"31'3296","reported_seq":3337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.341668+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:29.341668+0000","last_peered":"2024-09-18T11:45:29.341668+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:29.341668+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27000832,"num_objects":3296,"num_object_clones":0,"num_object_copies":13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3117","reported_seq":3180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.960182+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:25.960182+0000","last_peered":"2024-09-18T11:45:25.960182+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:25.960182+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498870,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3115,"num_write_kb":24916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1011}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3093","reported_seq":3175,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.840899+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:25.840899+0000","last_peered":"2024-09-18T11:45:25.840899+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:25.840899+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3093,"log_dups_size":0,"ondisk_log_size":3093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25311867,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3124","reported_seq":3186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.032989+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:26.032989+0000","last_peered":"2024-09-18T11:45:26.032989+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:26.032989+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3124,"log_dups_size":0,"ondisk_log_size":3124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25556214,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3122,"num_write_kb":24965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1006}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3205","reported_seq":3260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.226820+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:29.226820+0000","last_peered":"2024-09-18T11:45:29.226820+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:29.226820+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26227958,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":25621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3220","reported_seq":3263,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.263969+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:29.263969+0000","last_peered":"2024-09-18T11:45:29.263969+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:29.263969+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26378240,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":25760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3186","reported_seq":3250,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.050862+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:26.050862+0000","last_peered":"2024-09-18T11:45:26.050862+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:26.050862+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26047734,"num_objects":3180,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3184,"num_write_kb":25452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1046}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3051","reported_seq":3114,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.085927+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:26.085927+0000","last_peered":"2024-09-18T11:45:26.085927+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:26.085927+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24950006,"num_objects":3046,"num_object_clones":0,"num_object_copies":12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":24380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1012}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3298","reported_seq":3356,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.318234+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:29.318234+0000","last_peered":"2024-09-18T11:45:29.318234+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:29.318234+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26946028,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2992","reported_seq":3076,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.920920+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:25.920920+0000","last_peered":"2024-09-18T11:45:25.920920+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:25.920920+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":2992,"log_dups_size":0,"ondisk_log_size":2992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24491254,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2990,"num_write_kb":23918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1015}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3156","reported_seq":3212,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.156216+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:29.156216+0000","last_peered":"2024-09-18T11:45:29.156216+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:29.156216+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3156,"log_dups_size":0,"ondisk_log_size":3156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25808753,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3235","reported_seq":3272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.274841+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:29.274841+0000","last_peered":"2024-09-18T11:45:29.274841+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:29.274841+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3235,"log_dups_size":0,"ondisk_log_size":3235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26491515,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3234,"num_write_kb":25871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3097","reported_seq":3173,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.824442+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:25.824442+0000","last_peered":"2024-09-18T11:45:25.824442+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:25.824442+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3097,"log_dups_size":0,"ondisk_log_size":3097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25361019,"num_objects":3096,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3096,"num_write_kb":24767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3267","reported_seq":3316,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.360949+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:29.360949+0000","last_peered":"2024-09-18T11:45:29.360949+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:29.360949+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26727670,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":26116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3115","reported_seq":4328,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:25.967422+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:25.967422+0000","last_peered":"2024-09-18T11:45:25.967422+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:25.967422+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25490678,"num_objects":3112,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":567,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1586}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3017","reported_seq":3079,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:26.003458+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:26.003458+0000","last_peered":"2024-09-18T11:45:26.003458+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:26.003458+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3017,"log_dups_size":0,"ondisk_log_size":3017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24671478,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3015,"num_write_kb":24108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":412960116,"num_objects":50415,"num_object_clones":0,"num_object_copies":201660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50442,"num_write_kb":403418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":692658176,"data_stored":692658176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50469,"ondisk_log_size":50469,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501696,"kb_used_data":203832,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93870144,"statfs":{"total":96636764160,"available":96123027456,"internally_reserved":0,"allocated":208723968,"data_stored":208564990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,3,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":9.5199999999999996},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":2.2519999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":5.3070000000000004},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":6.5229999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499284,"kb_used_data":197772,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93872556,"statfs":{"total":96636764160,"available":96125497344,"internally_reserved":0,"allocated":202518528,"data_stored":202359550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,11,5,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":12.529999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":18.047000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.427}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":12.5},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":11.362}]}]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378540,"kb_used_data":72420,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93993300,"statfs":{"total":96636764160,"available":96249139200,"internally_reserved":0,"allocated":74158080,"data_stored":73999102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.532}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.223}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.258}]}]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497064,"kb_used_data":199264,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93874776,"statfs":{"total":96636764160,"available":96127770624,"internally_reserved":0,"allocated":204046336,"data_stored":203887358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1749999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":1.988},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":1.774}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":2.0179999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204054528,"data_stored":204054528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75296768,"data_stored":75296768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203808768,"data_stored":203808768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209498112,"data_stored":209498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:32.218 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 75 20 52178 52158 5.43215 3.16406 0.0739663 0.0287502 2024-09-18T11:45:33.219 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 76 20 52573 52553 5.40125 3.08594 0.105964 0.0289102 2024-09-18T11:45:34.219 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 77 20 52980 52960 5.37239 3.17969 0.0397348 0.0290744 2024-09-18T11:45:34.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:34.819 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:34.819 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:34.845 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-18T11:45:33.846253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":421889396,"num_objects":51505,"num_object_clones":0,"num_object_copies":206020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51532,"num_write_kb":412138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51559,"ondisk_log_size":51559,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1901876,"kb_used_data":690260,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375585484,"statfs":{"total":386547056640,"available":384599535616,"internally_reserved":0,"allocated":706826240,"data_stored":706190328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,11,8,19,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32579584,"num_objects":3977,"num_object_clones":0,"num_object_copies":15908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3977,"num_write_kb":31816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003804"},"pg_stats":[{"pgid":"1.f","version":"31'3296","reported_seq":3337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.341668+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:29.341668+0000","last_peered":"2024-09-18T11:45:29.341668+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:29.341668+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27000832,"num_objects":3296,"num_object_clones":0,"num_object_copies":13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3247","reported_seq":3309,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.017674+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:31.017674+0000","last_peered":"2024-09-18T11:45:31.017674+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:31.017674+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26563830,"num_objects":3243,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3245,"num_write_kb":25956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3225","reported_seq":3305,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.021373+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:31.021373+0000","last_peered":"2024-09-18T11:45:31.021373+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:31.021373+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26393211,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3230","reported_seq":3292,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.023055+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:31.023055+0000","last_peered":"2024-09-18T11:45:31.023055+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:31.023055+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3230,"log_dups_size":0,"ondisk_log_size":3230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26424566,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":25813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1112}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3205","reported_seq":3260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.226820+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:29.226820+0000","last_peered":"2024-09-18T11:45:29.226820+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:29.226820+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26227958,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":25621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3220","reported_seq":3263,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.263969+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:29.263969+0000","last_peered":"2024-09-18T11:45:29.263969+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:29.263969+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26378240,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":25760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3307","reported_seq":3372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:30.943000+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:30.943000+0000","last_peered":"2024-09-18T11:45:30.943000+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:30.943000+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038966,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3305,"num_write_kb":26420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1167}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3157","reported_seq":3220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.048921+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:31.048921+0000","last_peered":"2024-09-18T11:45:31.048921+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:31.048921+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3157,"log_dups_size":0,"ondisk_log_size":3157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25818358,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3155,"num_write_kb":25228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3298","reported_seq":3356,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.318234+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:29.318234+0000","last_peered":"2024-09-18T11:45:29.318234+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:29.318234+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26946028,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3115","reported_seq":3199,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:30.904926+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:30.904926+0000","last_peered":"2024-09-18T11:45:30.904926+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:30.904926+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498870,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1138}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3156","reported_seq":3212,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.156216+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:29.156216+0000","last_peered":"2024-09-18T11:45:29.156216+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:29.156216+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3156,"log_dups_size":0,"ondisk_log_size":3156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25808753,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3235","reported_seq":3272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.274841+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:29.274841+0000","last_peered":"2024-09-18T11:45:29.274841+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:29.274841+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3235,"log_dups_size":0,"ondisk_log_size":3235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26491515,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3234,"num_write_kb":25871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3214","reported_seq":3290,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.022480+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:31.022480+0000","last_peered":"2024-09-18T11:45:31.022480+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:31.022480+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26319483,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":25703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1159}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3267","reported_seq":3316,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.360949+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:29.360949+0000","last_peered":"2024-09-18T11:45:29.360949+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:29.360949+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26727670,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":26116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3248","reported_seq":4590,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:30.976802+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:30.976802+0000","last_peered":"2024-09-18T11:45:30.976802+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:30.976802+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3248,"log_dups_size":0,"ondisk_log_size":3248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26580214,"num_objects":3245,"num_object_clones":0,"num_object_copies":12980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3246,"num_write_kb":25965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3139","reported_seq":3201,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.112561+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:31.112561+0000","last_peered":"2024-09-18T11:45:31.112561+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:31.112561+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25670902,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":25084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":421889396,"num_objects":51505,"num_object_clones":0,"num_object_copies":206020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51532,"num_write_kb":412138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":708509696,"data_stored":708509696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51559,"ondisk_log_size":51559,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501696,"kb_used_data":203832,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93870144,"statfs":{"total":96636764160,"available":96123027456,"internally_reserved":0,"allocated":208723968,"data_stored":208564990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,3,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":9.5199999999999996},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":2.2519999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":5.3070000000000004},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":6.5229999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512668,"kb_used_data":206996,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93859172,"statfs":{"total":96636764160,"available":96111792128,"internally_reserved":0,"allocated":211963904,"data_stored":211804926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0410000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":3.9660000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378540,"kb_used_data":72420,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93993300,"statfs":{"total":96636764160,"available":96249139200,"internally_reserved":0,"allocated":74158080,"data_stored":73999102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.532}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.223}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.258}]}]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508972,"kb_used_data":207012,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93862868,"statfs":{"total":96636764160,"available":96115576832,"internally_reserved":0,"allocated":211980288,"data_stored":211821310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,1,2,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.14399999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":7.867}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":1.4630000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211943424,"data_stored":211943424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75296768,"data_stored":75296768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211771392,"data_stored":211771392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209498112,"data_stored":209498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:34.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:35.222 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 78 20 53329 53309 5.33847 2.72656 0.0595071 0.0292519 2024-09-18T11:45:35.367 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:35.368 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:35.391 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-18T11:45:33.846253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":421889396,"num_objects":51505,"num_object_clones":0,"num_object_copies":206020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51532,"num_write_kb":412138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51559,"ondisk_log_size":51559,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1901876,"kb_used_data":690260,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375585484,"statfs":{"total":386547056640,"available":384599535616,"internally_reserved":0,"allocated":706826240,"data_stored":706190328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,11,8,19,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32579584,"num_objects":3977,"num_object_clones":0,"num_object_copies":15908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3977,"num_write_kb":31816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003804"},"pg_stats":[{"pgid":"1.f","version":"31'3296","reported_seq":3337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.341668+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:29.341668+0000","last_peered":"2024-09-18T11:45:29.341668+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:29.341668+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27000832,"num_objects":3296,"num_object_clones":0,"num_object_copies":13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3247","reported_seq":3309,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.017674+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:31.017674+0000","last_peered":"2024-09-18T11:45:31.017674+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:31.017674+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26563830,"num_objects":3243,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3245,"num_write_kb":25956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3225","reported_seq":3305,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.021373+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:31.021373+0000","last_peered":"2024-09-18T11:45:31.021373+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:31.021373+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26393211,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3230","reported_seq":3292,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.023055+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:31.023055+0000","last_peered":"2024-09-18T11:45:31.023055+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:31.023055+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3230,"log_dups_size":0,"ondisk_log_size":3230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26424566,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":25813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1112}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3205","reported_seq":3260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.226820+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:29.226820+0000","last_peered":"2024-09-18T11:45:29.226820+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:29.226820+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26227958,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":25621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3220","reported_seq":3263,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.263969+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:29.263969+0000","last_peered":"2024-09-18T11:45:29.263969+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:29.263969+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26378240,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":25760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3307","reported_seq":3372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:30.943000+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:30.943000+0000","last_peered":"2024-09-18T11:45:30.943000+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:30.943000+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038966,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3305,"num_write_kb":26420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1167}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3157","reported_seq":3220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.048921+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:31.048921+0000","last_peered":"2024-09-18T11:45:31.048921+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:31.048921+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3157,"log_dups_size":0,"ondisk_log_size":3157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25818358,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3155,"num_write_kb":25228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3298","reported_seq":3356,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.318234+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:29.318234+0000","last_peered":"2024-09-18T11:45:29.318234+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:29.318234+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26946028,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3115","reported_seq":3199,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:30.904926+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:30.904926+0000","last_peered":"2024-09-18T11:45:30.904926+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:30.904926+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498870,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1138}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3156","reported_seq":3212,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.156216+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:29.156216+0000","last_peered":"2024-09-18T11:45:29.156216+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:29.156216+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3156,"log_dups_size":0,"ondisk_log_size":3156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25808753,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3235","reported_seq":3272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.274841+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:29.274841+0000","last_peered":"2024-09-18T11:45:29.274841+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:29.274841+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3235,"log_dups_size":0,"ondisk_log_size":3235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26491515,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3234,"num_write_kb":25871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3214","reported_seq":3290,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.022480+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:31.022480+0000","last_peered":"2024-09-18T11:45:31.022480+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:31.022480+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26319483,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":25703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1159}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3267","reported_seq":3316,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:29.360949+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:29.360949+0000","last_peered":"2024-09-18T11:45:29.360949+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:29.360949+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26727670,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":26116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3248","reported_seq":4590,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:30.976802+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:30.976802+0000","last_peered":"2024-09-18T11:45:30.976802+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:30.976802+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3248,"log_dups_size":0,"ondisk_log_size":3248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26580214,"num_objects":3245,"num_object_clones":0,"num_object_copies":12980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3246,"num_write_kb":25965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3139","reported_seq":3201,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.112561+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:31.112561+0000","last_peered":"2024-09-18T11:45:31.112561+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:31.112561+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25670902,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":25084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":421889396,"num_objects":51505,"num_object_clones":0,"num_object_copies":206020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51532,"num_write_kb":412138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":708509696,"data_stored":708509696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51559,"ondisk_log_size":51559,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501696,"kb_used_data":203832,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93870144,"statfs":{"total":96636764160,"available":96123027456,"internally_reserved":0,"allocated":208723968,"data_stored":208564990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,3,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":9.5199999999999996},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":2.2519999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":5.3070000000000004},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":6.5229999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512668,"kb_used_data":206996,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93859172,"statfs":{"total":96636764160,"available":96111792128,"internally_reserved":0,"allocated":211963904,"data_stored":211804926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0410000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":3.9660000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378540,"kb_used_data":72420,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93993300,"statfs":{"total":96636764160,"available":96249139200,"internally_reserved":0,"allocated":74158080,"data_stored":73999102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.532}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.223}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.258}]}]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508972,"kb_used_data":207012,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93862868,"statfs":{"total":96636764160,"available":96115576832,"internally_reserved":0,"allocated":211980288,"data_stored":211821310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,1,2,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.14399999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":7.867}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":1.4630000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211943424,"data_stored":211943424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75296768,"data_stored":75296768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211771392,"data_stored":211771392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209498112,"data_stored":209498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:35.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:35.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3296", 'reported_seq': 3337, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.341668+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:29.341668+0000', 'last_peered': '2024-09-18T11:45:29.341668+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:29.341668+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3296, 'log_dups_size': 0, 'ondisk_log_size': 3296, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27000832, 'num_objects': 3296, 'num_object_clones': 0, 'num_object_copies': 13184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3296, 'num_write_kb': 26368, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1166}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:35.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:35.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3247", 'reported_seq': 3309, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:31.017674+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:31.017674+0000', 'last_peered': '2024-09-18T11:45:31.017674+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:31.017674+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3247, 'log_dups_size': 0, 'ondisk_log_size': 3247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26563830, 'num_objects': 3243, 'num_object_clones': 0, 'num_object_copies': 12972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3243, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3245, 'num_write_kb': 25956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:35.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:35.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3225", 'reported_seq': 3305, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:31.021373+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:31.021373+0000', 'last_peered': '2024-09-18T11:45:31.021373+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:31.021373+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3225, 'log_dups_size': 0, 'ondisk_log_size': 3225, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26393211, 'num_objects': 3222, 'num_object_clones': 0, 'num_object_copies': 12888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3224, 'num_write_kb': 25782, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1168}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:35.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:35.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3230", 'reported_seq': 3292, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:31.023055+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:31.023055+0000', 'last_peered': '2024-09-18T11:45:31.023055+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:31.023055+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3230, 'log_dups_size': 0, 'ondisk_log_size': 3230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26424566, 'num_objects': 3226, 'num_object_clones': 0, 'num_object_copies': 12904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3228, 'num_write_kb': 25813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1112}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:35.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:35.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3205", 'reported_seq': 3260, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.226820+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:29.226820+0000', 'last_peered': '2024-09-18T11:45:29.226820+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:29.226820+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3205, 'log_dups_size': 0, 'ondisk_log_size': 3205, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26227958, 'num_objects': 3202, 'num_object_clones': 0, 'num_object_copies': 12808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3203, 'num_write_kb': 25621, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1120}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:35.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:35.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3220", 'reported_seq': 3263, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.263969+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:29.263969+0000', 'last_peered': '2024-09-18T11:45:29.263969+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:29.263969+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3220, 'log_dups_size': 0, 'ondisk_log_size': 3220, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26378240, 'num_objects': 3220, 'num_object_clones': 0, 'num_object_copies': 12880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3220, 'num_write_kb': 25760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1066}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:35.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:35.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3307", 'reported_seq': 3372, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:30.943000+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:30.943000+0000', 'last_peered': '2024-09-18T11:45:30.943000+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:30.943000+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3307, 'log_dups_size': 0, 'ondisk_log_size': 3307, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27038966, 'num_objects': 3301, 'num_object_clones': 0, 'num_object_copies': 13204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3305, 'num_write_kb': 26420, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1167}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:35.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:35.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3157", 'reported_seq': 3220, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:31.048921+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:31.048921+0000', 'last_peered': '2024-09-18T11:45:31.048921+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:31.048921+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3157, 'log_dups_size': 0, 'ondisk_log_size': 3157, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25818358, 'num_objects': 3152, 'num_object_clones': 0, 'num_object_copies': 12608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3155, 'num_write_kb': 25228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1118}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:35.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:35.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3298", 'reported_seq': 3356, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.318234+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:29.318234+0000', 'last_peered': '2024-09-18T11:45:29.318234+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:29.318234+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3298, 'log_dups_size': 0, 'ondisk_log_size': 3298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26946028, 'num_objects': 3290, 'num_object_clones': 0, 'num_object_copies': 13160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3295, 'num_write_kb': 26337, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1155}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:35.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:35.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3115", 'reported_seq': 3199, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:30.904926+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:30.904926+0000', 'last_peered': '2024-09-18T11:45:30.904926+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:30.904926+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3115, 'log_dups_size': 0, 'ondisk_log_size': 3115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25498870, 'num_objects': 3113, 'num_object_clones': 0, 'num_object_copies': 12452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3113, 'num_write_kb': 24902, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1138}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:35.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:35.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3156", 'reported_seq': 3212, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.156216+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:29.156216+0000', 'last_peered': '2024-09-18T11:45:29.156216+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:29.156216+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3156, 'log_dups_size': 0, 'ondisk_log_size': 3156, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25808753, 'num_objects': 3151, 'num_object_clones': 0, 'num_object_copies': 12604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3153, 'num_write_kb': 25212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1088}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:35.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:35.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3235", 'reported_seq': 3272, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.274841+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:29.274841+0000', 'last_peered': '2024-09-18T11:45:29.274841+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:29.274841+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3235, 'log_dups_size': 0, 'ondisk_log_size': 3235, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26491515, 'num_objects': 3234, 'num_object_clones': 0, 'num_object_copies': 12936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3234, 'num_write_kb': 25871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:35.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:35.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3214", 'reported_seq': 3290, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:31.022480+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:31.022480+0000', 'last_peered': '2024-09-18T11:45:31.022480+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:31.022480+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3214, 'log_dups_size': 0, 'ondisk_log_size': 3214, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26319483, 'num_objects': 3213, 'num_object_clones': 0, 'num_object_copies': 12852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3213, 'num_write_kb': 25703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:35.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:35.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3267", 'reported_seq': 3316, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:29.360949+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:29.360949+0000', 'last_peered': '2024-09-18T11:45:29.360949+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:29.360949+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3267, 'log_dups_size': 0, 'ondisk_log_size': 3267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26727670, 'num_objects': 3263, 'num_object_clones': 0, 'num_object_copies': 13052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3265, 'num_write_kb': 26116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:35.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:35.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3248", 'reported_seq': 4590, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:30.976802+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:30.976802+0000', 'last_peered': '2024-09-18T11:45:30.976802+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:30.976802+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3248, 'log_dups_size': 0, 'ondisk_log_size': 3248, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26580214, 'num_objects': 3245, 'num_object_clones': 0, 'num_object_copies': 12980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1461, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3246, 'num_write_kb': 25965, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 632, 'num_bytes_recovered': 5160960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1784}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:35.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:35.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3139", 'reported_seq': 3201, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:31.112561+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:31.112561+0000', 'last_peered': '2024-09-18T11:45:31.112561+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:31.112561+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3139, 'log_dups_size': 0, 'ondisk_log_size': 3139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25670902, 'num_objects': 3134, 'num_object_clones': 0, 'num_object_copies': 12536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3137, 'num_write_kb': 25084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:35.396 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:36.037 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:36.037 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32797/209332 objects degraded (15.667%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":134,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52333,"data_bytes":428672372,"bytes_used":1971593216,"bytes_avail":384575463424,"bytes_total":386547056640,"degraded_objects":32797,"degraded_total":209332,"degraded_ratio":0.15667456480614525,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":94867,"recovering_keys_per_sec":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"write_bytes_sec":3279405,"read_op_per_sec":0,"write_op_per_sec":400},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T11:45:05.837774+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:45:36.067 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:36.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:36.219 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 79 20 53678 53658 5.30541 2.72656 0.0619637 0.0294359 2024-09-18T11:45:36.612 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:36.612 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:36.632 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-18T11:45:35.846804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":428672372,"num_objects":52333,"num_object_clones":0,"num_object_copies":209332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52360,"num_write_kb":418762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52387,"ondisk_log_size":52387,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1925384,"kb_used_data":705448,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375561976,"statfs":{"total":386547056640,"available":384575463424,"internally_reserved":0,"allocated":722378752,"data_stored":721742840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,15,22,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39362560,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4805,"num_write_kb":38440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002954"},"pg_stats":[{"pgid":"1.f","version":"31'3409","reported_seq":3450,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:34.364981+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:34.364981+0000","last_peered":"2024-09-18T11:45:34.364981+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:34.364981+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3409,"log_dups_size":0,"ondisk_log_size":3409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27926528,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3409,"num_write_kb":27272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3247","reported_seq":3309,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.017674+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:31.017674+0000","last_peered":"2024-09-18T11:45:31.017674+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:31.017674+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26563830,"num_objects":3243,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3245,"num_write_kb":25956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3225","reported_seq":3305,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.021373+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:31.021373+0000","last_peered":"2024-09-18T11:45:31.021373+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:31.021373+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26393211,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3230","reported_seq":3292,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.023055+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:31.023055+0000","last_peered":"2024-09-18T11:45:31.023055+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:31.023055+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3230,"log_dups_size":0,"ondisk_log_size":3230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26424566,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":25813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1112}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3326","reported_seq":3381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:34.360006+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:34.360006+0000","last_peered":"2024-09-18T11:45:34.360006+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:34.360006+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3326,"log_dups_size":0,"ondisk_log_size":3326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27219190,"num_objects":3323,"num_object_clones":0,"num_object_copies":13292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3324,"num_write_kb":26589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1241}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3324","reported_seq":3367,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:34.215898+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:34.215898+0000","last_peered":"2024-09-18T11:45:34.215898+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:34.215898+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3324,"log_dups_size":0,"ondisk_log_size":3324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27230208,"num_objects":3324,"num_object_clones":0,"num_object_copies":13296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3324,"num_write_kb":26592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3307","reported_seq":3372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:30.943000+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:30.943000+0000","last_peered":"2024-09-18T11:45:30.943000+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:30.943000+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038966,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3305,"num_write_kb":26420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1167}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3157","reported_seq":3220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.048921+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:31.048921+0000","last_peered":"2024-09-18T11:45:31.048921+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:31.048921+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3157,"log_dups_size":0,"ondisk_log_size":3157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25818358,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3155,"num_write_kb":25228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3415","reported_seq":3473,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:34.349079+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:34.349079+0000","last_peered":"2024-09-18T11:45:34.349079+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:34.349079+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27904492,"num_objects":3407,"num_object_clones":0,"num_object_copies":13628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3412,"num_write_kb":27273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1272}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3115","reported_seq":3199,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:30.904926+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:30.904926+0000","last_peered":"2024-09-18T11:45:30.904926+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:30.904926+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498870,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1138}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3271","reported_seq":3328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:34.362689+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:34.362689+0000","last_peered":"2024-09-18T11:45:34.362689+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:34.362689+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3271,"log_dups_size":0,"ondisk_log_size":3271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26750833,"num_objects":3266,"num_object_clones":0,"num_object_copies":13064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3268,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1203}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3383","reported_seq":3419,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:34.374168+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:34.374168+0000","last_peered":"2024-09-18T11:45:34.374168+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:34.374168+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3383,"log_dups_size":0,"ondisk_log_size":3383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27703931,"num_objects":3382,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":27055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3214","reported_seq":3290,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.022480+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:31.022480+0000","last_peered":"2024-09-18T11:45:31.022480+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:31.022480+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26319483,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":25703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1159}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3377","reported_seq":3426,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:34.388753+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:34.388753+0000","last_peered":"2024-09-18T11:45:34.388753+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:34.388753+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3377,"log_dups_size":0,"ondisk_log_size":3377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27628790,"num_objects":3373,"num_object_clones":0,"num_object_copies":13492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3375,"num_write_kb":26996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1215}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3248","reported_seq":4590,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:30.976802+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:30.976802+0000","last_peered":"2024-09-18T11:45:30.976802+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:30.976802+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3248,"log_dups_size":0,"ondisk_log_size":3248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26580214,"num_objects":3245,"num_object_clones":0,"num_object_copies":12980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3246,"num_write_kb":25965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3139","reported_seq":3201,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:31.112561+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:31.112561+0000","last_peered":"2024-09-18T11:45:31.112561+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:31.112561+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25670902,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":25084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":428672372,"num_objects":52333,"num_object_clones":0,"num_object_copies":209332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52360,"num_write_kb":418762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":724340736,"data_stored":724340736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52387,"ondisk_log_size":52387,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513236,"kb_used_data":211212,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93858604,"statfs":{"total":96636764160,"available":96111210496,"internally_reserved":0,"allocated":216281088,"data_stored":216122110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,10,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":4.8099999999999996},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":9.5890000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6229999999999993}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":1.55},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":0.20200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512668,"kb_used_data":206996,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93859172,"statfs":{"total":96636764160,"available":96111792128,"internally_reserved":0,"allocated":211963904,"data_stored":211804926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0410000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":3.9660000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":390508,"kb_used_data":80228,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93981332,"statfs":{"total":96636764160,"available":96236883968,"internally_reserved":0,"allocated":82153472,"data_stored":81994494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8819999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4619999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508972,"kb_used_data":207012,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93862868,"statfs":{"total":96636764160,"available":96115576832,"internally_reserved":0,"allocated":211980288,"data_stored":211821310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,1,2,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.14399999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":7.867}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":1.4630000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211943424,"data_stored":211943424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83374080,"data_stored":83374080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211771392,"data_stored":211771392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217251840,"data_stored":217251840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:37.219 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:45:37.218866+0000 min lat: 0.00272854 max lat: 2.42146 avg lat: 0.0295987 2024-09-18T11:45:37.219 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:45:37.219 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 80 20 54055 54035 5.27591 2.94531 0.0358172 0.0295987 2024-09-18T11:45:38.219 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 81 20 54452 54432 5.24906 3.10156 0.0294738 0.0297521 2024-09-18T11:45:39.219 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 82 20 54841 54821 5.22211 3.03906 0.073154 0.029903 2024-09-18T11:45:39.634 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:40.195 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:40.195 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:40.220 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-18T11:45:39.847991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":443843956,"num_objects":54185,"num_object_clones":0,"num_object_copies":216740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54212,"num_write_kb":433578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":5799936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54239,"ondisk_log_size":54239,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1954824,"kb_used_data":734876,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375532536,"statfs":{"total":386547056640,"available":384545316864,"internally_reserved":0,"allocated":752513024,"data_stored":751877112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,11,13,26,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38256640,"num_objects":4670,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4670,"num_write_kb":37360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":1171456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003092"},"pg_stats":[{"pgid":"1.f","version":"31'3508","reported_seq":3549,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.389376+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:39.389376+0000","last_peered":"2024-09-18T11:45:39.389376+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:39.389376+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3508,"log_dups_size":0,"ondisk_log_size":3508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28737536,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3364","reported_seq":3427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:35.965114+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:35.965114+0000","last_peered":"2024-09-18T11:45:35.965114+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:35.965114+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3364,"log_dups_size":0,"ondisk_log_size":3364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27522294,"num_objects":3360,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3362,"num_write_kb":26892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3352","reported_seq":3434,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:35.887485+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:35.887485+0000","last_peered":"2024-09-18T11:45:35.887485+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:35.887485+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3352,"log_dups_size":0,"ondisk_log_size":3352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27433595,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":26798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1295}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3351","reported_seq":3413,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:36.078786+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:36.078786+0000","last_peered":"2024-09-18T11:45:36.078786+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:36.078786+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27415798,"num_objects":3347,"num_object_clones":0,"num_object_copies":13388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3349,"num_write_kb":26781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1233}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3423","reported_seq":3478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.396579+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:39.396579+0000","last_peered":"2024-09-18T11:45:39.396579+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:39.396579+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3423,"log_dups_size":0,"ondisk_log_size":3423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013814,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3438","reported_seq":3481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.371735+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:39.371735+0000","last_peered":"2024-09-18T11:45:39.371735+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:39.371735+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28164096,"num_objects":3438,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3438,"num_write_kb":27504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1284}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3425","reported_seq":3490,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:36.114381+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:36.114381+0000","last_peered":"2024-09-18T11:45:36.114381+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:36.114381+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28005622,"num_objects":3419,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3423,"num_write_kb":27364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1285}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3278","reported_seq":3341,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:36.013837+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:36.013837+0000","last_peered":"2024-09-18T11:45:36.013837+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:36.013837+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3278,"log_dups_size":0,"ondisk_log_size":3278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26809590,"num_objects":3273,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":26196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3530","reported_seq":3588,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.308057+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:39.308057+0000","last_peered":"2024-09-18T11:45:39.308057+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:39.308057+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28846572,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3243","reported_seq":3327,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:35.922974+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:35.922974+0000","last_peered":"2024-09-18T11:45:35.922974+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:35.922974+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26547446,"num_objects":3241,"num_object_clones":0,"num_object_copies":12964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3241,"num_write_kb":25926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1266}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3390","reported_seq":3447,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.347487+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:39.347487+0000","last_peered":"2024-09-18T11:45:39.347487+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:39.347487+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27725681,"num_objects":3385,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3387,"num_write_kb":27084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1322}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3505","reported_seq":3540,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.371518+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:39.371518+0000","last_peered":"2024-09-18T11:45:39.371518+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:39.371518+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703355,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":28031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1395}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3321","reported_seq":3397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:35.868191+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:35.868191+0000","last_peered":"2024-09-18T11:45:35.868191+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:35.868191+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27196027,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":26559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1266}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3496","reported_seq":3545,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.260488+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:39.260488+0000","last_peered":"2024-09-18T11:45:39.260488+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:39.260488+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3496,"log_dups_size":0,"ondisk_log_size":3496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28603638,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3494,"num_write_kb":27948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1334}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3359","reported_seq":4857,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:36.018346+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:36.018346+0000","last_peered":"2024-09-18T11:45:36.018346+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:36.018346+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3359,"log_dups_size":0,"ondisk_log_size":3359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27489526,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3357,"num_write_kb":26853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":5799936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3256","reported_seq":3318,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:36.011149+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:36.011149+0000","last_peered":"2024-09-18T11:45:36.011149+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:36.011149+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3256,"log_dups_size":0,"ondisk_log_size":3256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26629366,"num_objects":3251,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3254,"num_write_kb":26020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1238}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":443843956,"num_objects":54185,"num_object_clones":0,"num_object_copies":216740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54212,"num_write_kb":433578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":5799936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":755752960,"data_stored":755752960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54239,"ondisk_log_size":54239,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524964,"kb_used_data":218772,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93846876,"statfs":{"total":96636764160,"available":96099201024,"internally_reserved":0,"allocated":224022528,"data_stored":223863550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,9,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":1.895},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":2.202}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":3.7080000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523228,"kb_used_data":213392,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93848612,"statfs":{"total":96636764160,"available":96100978688,"internally_reserved":0,"allocated":218513408,"data_stored":218354430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":2.3769999999999998},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.32800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":2.278},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":1.875}]}]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385860,"kb_used_data":88060,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93985980,"statfs":{"total":96636764160,"available":96241643520,"internally_reserved":0,"allocated":90173440,"data_stored":90014462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,2,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.226}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5220000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":520772,"kb_used_data":214652,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93851068,"statfs":{"total":96636764160,"available":96103493632,"internally_reserved":0,"allocated":219803648,"data_stored":219644670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":4.96},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":5.0019999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":1.925},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":6.2249999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":219783168,"data_stored":219783168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91398144,"data_stored":91398144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219578368,"data_stored":219578368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224993280,"data_stored":224993280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:40.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:40.221 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 83 20 55209 55189 5.19384 2.875 0.115065 0.0300649 2024-09-18T11:45:40.735 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:40.735 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:40.757 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-18T11:45:39.847991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":443843956,"num_objects":54185,"num_object_clones":0,"num_object_copies":216740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54212,"num_write_kb":433578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":5799936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54239,"ondisk_log_size":54239,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1954824,"kb_used_data":734876,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375532536,"statfs":{"total":386547056640,"available":384545316864,"internally_reserved":0,"allocated":752513024,"data_stored":751877112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,11,13,26,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38256640,"num_objects":4670,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4670,"num_write_kb":37360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":1171456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003092"},"pg_stats":[{"pgid":"1.f","version":"31'3508","reported_seq":3549,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.389376+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:39.389376+0000","last_peered":"2024-09-18T11:45:39.389376+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:39.389376+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3508,"log_dups_size":0,"ondisk_log_size":3508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28737536,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3364","reported_seq":3427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:35.965114+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:35.965114+0000","last_peered":"2024-09-18T11:45:35.965114+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:35.965114+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3364,"log_dups_size":0,"ondisk_log_size":3364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27522294,"num_objects":3360,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3362,"num_write_kb":26892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3352","reported_seq":3434,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:35.887485+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:35.887485+0000","last_peered":"2024-09-18T11:45:35.887485+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:35.887485+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3352,"log_dups_size":0,"ondisk_log_size":3352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27433595,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":26798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1295}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3351","reported_seq":3413,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:36.078786+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:36.078786+0000","last_peered":"2024-09-18T11:45:36.078786+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:36.078786+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27415798,"num_objects":3347,"num_object_clones":0,"num_object_copies":13388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3349,"num_write_kb":26781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1233}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3423","reported_seq":3478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.396579+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:39.396579+0000","last_peered":"2024-09-18T11:45:39.396579+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:39.396579+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3423,"log_dups_size":0,"ondisk_log_size":3423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013814,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3438","reported_seq":3481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.371735+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:39.371735+0000","last_peered":"2024-09-18T11:45:39.371735+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:39.371735+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28164096,"num_objects":3438,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3438,"num_write_kb":27504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1284}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3425","reported_seq":3490,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:36.114381+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:36.114381+0000","last_peered":"2024-09-18T11:45:36.114381+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:36.114381+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28005622,"num_objects":3419,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3423,"num_write_kb":27364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1285}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3278","reported_seq":3341,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:36.013837+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:36.013837+0000","last_peered":"2024-09-18T11:45:36.013837+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:36.013837+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3278,"log_dups_size":0,"ondisk_log_size":3278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26809590,"num_objects":3273,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":26196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3530","reported_seq":3588,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.308057+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:39.308057+0000","last_peered":"2024-09-18T11:45:39.308057+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:39.308057+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28846572,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3243","reported_seq":3327,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:35.922974+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:35.922974+0000","last_peered":"2024-09-18T11:45:35.922974+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:35.922974+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26547446,"num_objects":3241,"num_object_clones":0,"num_object_copies":12964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3241,"num_write_kb":25926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1266}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3390","reported_seq":3447,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.347487+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:39.347487+0000","last_peered":"2024-09-18T11:45:39.347487+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:39.347487+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27725681,"num_objects":3385,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3387,"num_write_kb":27084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1322}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3505","reported_seq":3540,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.371518+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:39.371518+0000","last_peered":"2024-09-18T11:45:39.371518+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:39.371518+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703355,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":28031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1395}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3321","reported_seq":3397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:35.868191+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:35.868191+0000","last_peered":"2024-09-18T11:45:35.868191+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:35.868191+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27196027,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":26559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1266}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3496","reported_seq":3545,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.260488+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:39.260488+0000","last_peered":"2024-09-18T11:45:39.260488+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:39.260488+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3496,"log_dups_size":0,"ondisk_log_size":3496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28603638,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3494,"num_write_kb":27948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1334}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3359","reported_seq":4857,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:36.018346+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:36.018346+0000","last_peered":"2024-09-18T11:45:36.018346+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:36.018346+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3359,"log_dups_size":0,"ondisk_log_size":3359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27489526,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3357,"num_write_kb":26853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":5799936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3256","reported_seq":3318,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:36.011149+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:36.011149+0000","last_peered":"2024-09-18T11:45:36.011149+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:36.011149+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3256,"log_dups_size":0,"ondisk_log_size":3256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26629366,"num_objects":3251,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3254,"num_write_kb":26020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1238}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":443843956,"num_objects":54185,"num_object_clones":0,"num_object_copies":216740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54212,"num_write_kb":433578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":5799936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":755752960,"data_stored":755752960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54239,"ondisk_log_size":54239,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524964,"kb_used_data":218772,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93846876,"statfs":{"total":96636764160,"available":96099201024,"internally_reserved":0,"allocated":224022528,"data_stored":223863550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,9,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":1.895},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":2.202}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":3.7080000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523228,"kb_used_data":213392,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93848612,"statfs":{"total":96636764160,"available":96100978688,"internally_reserved":0,"allocated":218513408,"data_stored":218354430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":2.3769999999999998},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.32800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":2.278},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":1.875}]}]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385860,"kb_used_data":88060,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93985980,"statfs":{"total":96636764160,"available":96241643520,"internally_reserved":0,"allocated":90173440,"data_stored":90014462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,2,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.226}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5220000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":520772,"kb_used_data":214652,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93851068,"statfs":{"total":96636764160,"available":96103493632,"internally_reserved":0,"allocated":219803648,"data_stored":219644670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":4.96},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":5.0019999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":1.925},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":6.2249999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":219783168,"data_stored":219783168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91398144,"data_stored":91398144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219578368,"data_stored":219578368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224993280,"data_stored":224993280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:40.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:40.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3508", 'reported_seq': 3549, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:39.389376+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:39.389376+0000', 'last_peered': '2024-09-18T11:45:39.389376+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:39.389376+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3508, 'log_dups_size': 0, 'ondisk_log_size': 3508, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28737536, 'num_objects': 3508, 'num_object_clones': 0, 'num_object_copies': 14032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3508, 'num_write_kb': 28064, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1378}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:40.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:40.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3364", 'reported_seq': 3427, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:35.965114+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:35.965114+0000', 'last_peered': '2024-09-18T11:45:35.965114+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:35.965114+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3364, 'log_dups_size': 0, 'ondisk_log_size': 3364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27522294, 'num_objects': 3360, 'num_object_clones': 0, 'num_object_copies': 13440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3362, 'num_write_kb': 26892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1258}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:40.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:40.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3352", 'reported_seq': 3434, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:35.887485+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:35.887485+0000', 'last_peered': '2024-09-18T11:45:35.887485+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:35.887485+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3352, 'log_dups_size': 0, 'ondisk_log_size': 3352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27433595, 'num_objects': 3349, 'num_object_clones': 0, 'num_object_copies': 13396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3351, 'num_write_kb': 26798, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1295}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:40.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:40.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3351", 'reported_seq': 3413, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:36.078786+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:36.078786+0000', 'last_peered': '2024-09-18T11:45:36.078786+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:36.078786+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3351, 'log_dups_size': 0, 'ondisk_log_size': 3351, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27415798, 'num_objects': 3347, 'num_object_clones': 0, 'num_object_copies': 13388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3347, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3349, 'num_write_kb': 26781, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1233}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:40.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:40.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3423", 'reported_seq': 3478, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:39.396579+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:39.396579+0000', 'last_peered': '2024-09-18T11:45:39.396579+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:39.396579+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3423, 'log_dups_size': 0, 'ondisk_log_size': 3423, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28013814, 'num_objects': 3420, 'num_object_clones': 0, 'num_object_copies': 13680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3421, 'num_write_kb': 27365, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1338}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:40.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:40.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3438", 'reported_seq': 3481, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:39.371735+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:39.371735+0000', 'last_peered': '2024-09-18T11:45:39.371735+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:39.371735+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3438, 'log_dups_size': 0, 'ondisk_log_size': 3438, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28164096, 'num_objects': 3438, 'num_object_clones': 0, 'num_object_copies': 13752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3438, 'num_write_kb': 27504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1284}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:40.760 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:40.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3425", 'reported_seq': 3490, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:36.114381+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:36.114381+0000', 'last_peered': '2024-09-18T11:45:36.114381+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:36.114381+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3425, 'log_dups_size': 0, 'ondisk_log_size': 3425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28005622, 'num_objects': 3419, 'num_object_clones': 0, 'num_object_copies': 13676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3423, 'num_write_kb': 27364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1285}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:40.760 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:40.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3278", 'reported_seq': 3341, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:36.013837+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:36.013837+0000', 'last_peered': '2024-09-18T11:45:36.013837+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:36.013837+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3278, 'log_dups_size': 0, 'ondisk_log_size': 3278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26809590, 'num_objects': 3273, 'num_object_clones': 0, 'num_object_copies': 13092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3276, 'num_write_kb': 26196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1239}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:40.760 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:40.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3530", 'reported_seq': 3588, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:39.308057+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:39.308057+0000', 'last_peered': '2024-09-18T11:45:39.308057+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:39.308057+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3530, 'log_dups_size': 0, 'ondisk_log_size': 3530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28846572, 'num_objects': 3522, 'num_object_clones': 0, 'num_object_copies': 14088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3527, 'num_write_kb': 28193, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1387}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:40.761 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:40.761 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3243", 'reported_seq': 3327, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:35.922974+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:35.922974+0000', 'last_peered': '2024-09-18T11:45:35.922974+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:35.922974+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3243, 'log_dups_size': 0, 'ondisk_log_size': 3243, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26547446, 'num_objects': 3241, 'num_object_clones': 0, 'num_object_copies': 12964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3241, 'num_write_kb': 25926, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1266}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:40.761 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:40.761 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3390", 'reported_seq': 3447, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:39.347487+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:39.347487+0000', 'last_peered': '2024-09-18T11:45:39.347487+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:39.347487+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3390, 'log_dups_size': 0, 'ondisk_log_size': 3390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27725681, 'num_objects': 3385, 'num_object_clones': 0, 'num_object_copies': 13540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3387, 'num_write_kb': 27084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1322}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:40.761 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:40.761 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3505", 'reported_seq': 3540, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:39.371518+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:39.371518+0000', 'last_peered': '2024-09-18T11:45:39.371518+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:39.371518+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3505, 'log_dups_size': 0, 'ondisk_log_size': 3505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28703355, 'num_objects': 3504, 'num_object_clones': 0, 'num_object_copies': 14016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3504, 'num_write_kb': 28031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:40.761 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:40.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3321", 'reported_seq': 3397, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:35.868191+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:35.868191+0000', 'last_peered': '2024-09-18T11:45:35.868191+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:35.868191+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3321, 'log_dups_size': 0, 'ondisk_log_size': 3321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27196027, 'num_objects': 3320, 'num_object_clones': 0, 'num_object_copies': 13280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3320, 'num_write_kb': 26559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1266}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:40.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:40.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3496", 'reported_seq': 3545, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:39.260488+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:39.260488+0000', 'last_peered': '2024-09-18T11:45:39.260488+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:39.260488+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3496, 'log_dups_size': 0, 'ondisk_log_size': 3496, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28603638, 'num_objects': 3492, 'num_object_clones': 0, 'num_object_copies': 13968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3494, 'num_write_kb': 27948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:40.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:40.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3359", 'reported_seq': 4857, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:36.018346+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:36.018346+0000', 'last_peered': '2024-09-18T11:45:36.018346+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:36.018346+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3359, 'log_dups_size': 0, 'ondisk_log_size': 3359, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27489526, 'num_objects': 3356, 'num_object_clones': 0, 'num_object_copies': 13424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1383, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3357, 'num_write_kb': 26853, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 710, 'num_bytes_recovered': 5799936, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1973}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:40.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:40.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3256", 'reported_seq': 3318, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:36.011149+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:36.011149+0000', 'last_peered': '2024-09-18T11:45:36.011149+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:36.011149+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3256, 'log_dups_size': 0, 'ondisk_log_size': 3256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26629366, 'num_objects': 3251, 'num_object_clones': 0, 'num_object_copies': 13004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3254, 'num_write_kb': 26020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:40.763 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:41.219 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 84 20 55545 55525 5.16326 2.625 0.0497815 0.0302446 2024-09-18T11:45:41.417 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:41.417 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32719/216740 objects degraded (15.096%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":139,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54185,"data_bytes":443843956,"bytes_used":2001739776,"bytes_avail":384545316864,"bytes_total":386547056640,"degraded_objects":32719,"degraded_total":216740,"degraded_ratio":0.15095967518685982,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":97596,"recovering_keys_per_sec":0,"num_objects_recovered":143,"num_bytes_recovered":1171456,"num_keys_recovered":0,"write_bytes_sec":3187232,"read_op_per_sec":0,"write_op_per_sec":389},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T11:45:05.837774+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:45:41.437 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:41.437 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:41.969 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:41.990 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-18T11:45:41.848429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":452691316,"num_objects":55265,"num_object_clones":0,"num_object_copies":221060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55292,"num_write_kb":442218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55319,"ondisk_log_size":55319,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1965672,"kb_used_data":749888,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":375521688,"statfs":{"total":386547056640,"available":384534208512,"internally_reserved":0,"allocated":767885312,"data_stored":767249400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,7,15,23,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39731200,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4850,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003077"},"pg_stats":[{"pgid":"1.f","version":"31'3508","reported_seq":3549,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.389376+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:39.389376+0000","last_peered":"2024-09-18T11:45:39.389376+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:39.389376+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3508,"log_dups_size":0,"ondisk_log_size":3508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28737536,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3485","reported_seq":3548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.098322+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:41.098322+0000","last_peered":"2024-09-18T11:45:41.098322+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:41.098322+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28513526,"num_objects":3481,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3456","reported_seq":3538,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.930923+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:40.930923+0000","last_peered":"2024-09-18T11:45:40.930923+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:40.930923+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28285563,"num_objects":3453,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":27630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1399}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3484","reported_seq":3546,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.185482+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:41.185482+0000","last_peered":"2024-09-18T11:45:41.185482+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:41.185482+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3484,"log_dups_size":0,"ondisk_log_size":3484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505334,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3423","reported_seq":3478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.396579+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:39.396579+0000","last_peered":"2024-09-18T11:45:39.396579+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:39.396579+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3423,"log_dups_size":0,"ondisk_log_size":3423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013814,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3438","reported_seq":3481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.371735+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:39.371735+0000","last_peered":"2024-09-18T11:45:39.371735+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:39.371735+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28164096,"num_objects":3438,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3438,"num_write_kb":27504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1284}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3547","reported_seq":3612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.182717+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:41.182717+0000","last_peered":"2024-09-18T11:45:41.182717+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:41.182717+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29005046,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1407}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3391","reported_seq":3454,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.102991+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:41.102991+0000","last_peered":"2024-09-18T11:45:41.102991+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:41.102991+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27735286,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3389,"num_write_kb":27100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3530","reported_seq":3588,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.308057+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:39.308057+0000","last_peered":"2024-09-18T11:45:39.308057+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:39.308057+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28846572,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3368","reported_seq":3452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.861212+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:40.861212+0000","last_peered":"2024-09-18T11:45:40.861212+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:40.861212+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3368,"log_dups_size":0,"ondisk_log_size":3368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27571446,"num_objects":3366,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3366,"num_write_kb":26926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3390","reported_seq":3447,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.347487+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:39.347487+0000","last_peered":"2024-09-18T11:45:39.347487+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:39.347487+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27725681,"num_objects":3385,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3387,"num_write_kb":27084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1322}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3505","reported_seq":3540,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.371518+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:39.371518+0000","last_peered":"2024-09-18T11:45:39.371518+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:39.371518+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703355,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":28031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1395}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3453","reported_seq":3529,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.973288+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:40.973288+0000","last_peered":"2024-09-18T11:45:40.973288+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:40.973288+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3453,"log_dups_size":0,"ondisk_log_size":3453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28277371,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3452,"num_write_kb":27615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1398}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3496","reported_seq":3545,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.260488+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:39.260488+0000","last_peered":"2024-09-18T11:45:39.260488+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:39.260488+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3496,"log_dups_size":0,"ondisk_log_size":3496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28603638,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3494,"num_write_kb":27948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1334}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3465","reported_seq":5116,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.990207+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:40.990207+0000","last_peered":"2024-09-18T11:45:40.990207+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:40.990207+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3465,"log_dups_size":0,"ondisk_log_size":3465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28357878,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3463,"num_write_kb":27701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2155}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3380","reported_seq":3442,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.081197+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:41.081197+0000","last_peered":"2024-09-18T11:45:41.081197+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:41.081197+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3380,"log_dups_size":0,"ondisk_log_size":3380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27645174,"num_objects":3375,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3378,"num_write_kb":27012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":452691316,"num_objects":55265,"num_object_clones":0,"num_object_copies":221060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55292,"num_write_kb":442218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":771108864,"data_stored":771108864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55319,"ondisk_log_size":55319,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524964,"kb_used_data":218772,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93846876,"statfs":{"total":96636764160,"available":96099201024,"internally_reserved":0,"allocated":224022528,"data_stored":223863550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,9,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":1.895},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":2.202}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":3.7080000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518400,"kb_used_data":221048,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93853440,"statfs":{"total":96636764160,"available":96105922560,"internally_reserved":0,"allocated":226353152,"data_stored":226194174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":5.952},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":5.8899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0619999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":8.8659999999999997},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":8.8249999999999993}]}]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385860,"kb_used_data":88060,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93985980,"statfs":{"total":96636764160,"available":96241643520,"internally_reserved":0,"allocated":90173440,"data_stored":90014462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,2,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.226}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5220000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536448,"kb_used_data":222008,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93835392,"statfs":{"total":96636764160,"available":96087441408,"internally_reserved":0,"allocated":227336192,"data_stored":227177214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1139999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":2.7610000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":4.9630000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.0860000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":10.057}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227479552,"data_stored":227479552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91398144,"data_stored":91398144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227237888,"data_stored":227237888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224993280,"data_stored":224993280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:42.221 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 85 20 55891 55871 5.13431 2.70312 0.0510729 0.0304188 2024-09-18T11:45:43.220 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 86 20 56287 56267 5.11052 3.09375 0.05574 0.0305594 2024-09-18T11:45:44.221 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 87 20 56679 56659 5.08698 3.0625 0.0221661 0.0307009 2024-09-18T11:45:44.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:45.221 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 88 20 57054 57034 5.06247 2.92969 0.0397621 0.0308459 2024-09-18T11:45:45.498 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:45.499 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:45.523 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-18T11:45:43.848864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":452691316,"num_objects":55265,"num_object_clones":0,"num_object_copies":221060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55292,"num_write_kb":442218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55319,"ondisk_log_size":55319,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1965672,"kb_used_data":749888,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":375521688,"statfs":{"total":386547056640,"available":384534208512,"internally_reserved":0,"allocated":767885312,"data_stored":767249400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,7,15,23,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3760,"num_write_kb":30080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003063"},"pg_stats":[{"pgid":"1.f","version":"31'3508","reported_seq":3549,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.389376+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:39.389376+0000","last_peered":"2024-09-18T11:45:39.389376+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:39.389376+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3508,"log_dups_size":0,"ondisk_log_size":3508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28737536,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3485","reported_seq":3548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.098322+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:41.098322+0000","last_peered":"2024-09-18T11:45:41.098322+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:41.098322+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28513526,"num_objects":3481,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3456","reported_seq":3538,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.930923+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:40.930923+0000","last_peered":"2024-09-18T11:45:40.930923+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:40.930923+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28285563,"num_objects":3453,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":27630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1399}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3484","reported_seq":3546,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.185482+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:41.185482+0000","last_peered":"2024-09-18T11:45:41.185482+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:41.185482+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3484,"log_dups_size":0,"ondisk_log_size":3484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505334,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3423","reported_seq":3478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.396579+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:39.396579+0000","last_peered":"2024-09-18T11:45:39.396579+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:39.396579+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3423,"log_dups_size":0,"ondisk_log_size":3423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013814,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3438","reported_seq":3481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.371735+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:39.371735+0000","last_peered":"2024-09-18T11:45:39.371735+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:39.371735+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28164096,"num_objects":3438,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3438,"num_write_kb":27504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1284}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3547","reported_seq":3612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.182717+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:41.182717+0000","last_peered":"2024-09-18T11:45:41.182717+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:41.182717+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29005046,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1407}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3391","reported_seq":3454,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.102991+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:41.102991+0000","last_peered":"2024-09-18T11:45:41.102991+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:41.102991+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27735286,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3389,"num_write_kb":27100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3530","reported_seq":3588,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.308057+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:39.308057+0000","last_peered":"2024-09-18T11:45:39.308057+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:39.308057+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28846572,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3368","reported_seq":3452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.861212+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:40.861212+0000","last_peered":"2024-09-18T11:45:40.861212+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:40.861212+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3368,"log_dups_size":0,"ondisk_log_size":3368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27571446,"num_objects":3366,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3366,"num_write_kb":26926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3390","reported_seq":3447,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.347487+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:39.347487+0000","last_peered":"2024-09-18T11:45:39.347487+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:39.347487+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27725681,"num_objects":3385,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3387,"num_write_kb":27084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1322}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3505","reported_seq":3540,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.371518+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:39.371518+0000","last_peered":"2024-09-18T11:45:39.371518+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:39.371518+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703355,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":28031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1395}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3453","reported_seq":3529,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.973288+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:40.973288+0000","last_peered":"2024-09-18T11:45:40.973288+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:40.973288+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3453,"log_dups_size":0,"ondisk_log_size":3453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28277371,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3452,"num_write_kb":27615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1398}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3496","reported_seq":3545,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:39.260488+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:39.260488+0000","last_peered":"2024-09-18T11:45:39.260488+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:39.260488+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3496,"log_dups_size":0,"ondisk_log_size":3496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28603638,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3494,"num_write_kb":27948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1334}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3465","reported_seq":5116,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.990207+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:40.990207+0000","last_peered":"2024-09-18T11:45:40.990207+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:40.990207+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3465,"log_dups_size":0,"ondisk_log_size":3465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28357878,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3463,"num_write_kb":27701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2155}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3380","reported_seq":3442,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.081197+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:41.081197+0000","last_peered":"2024-09-18T11:45:41.081197+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:41.081197+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3380,"log_dups_size":0,"ondisk_log_size":3380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27645174,"num_objects":3375,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3378,"num_write_kb":27012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":452691316,"num_objects":55265,"num_object_clones":0,"num_object_copies":221060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55292,"num_write_kb":442218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":771108864,"data_stored":771108864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55319,"ondisk_log_size":55319,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524964,"kb_used_data":218772,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93846876,"statfs":{"total":96636764160,"available":96099201024,"internally_reserved":0,"allocated":224022528,"data_stored":223863550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,9,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":1.895},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":2.202}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":3.7080000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518400,"kb_used_data":221048,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93853440,"statfs":{"total":96636764160,"available":96105922560,"internally_reserved":0,"allocated":226353152,"data_stored":226194174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":5.952},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":5.8899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0619999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":8.8659999999999997},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":8.8249999999999993}]}]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385860,"kb_used_data":88060,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93985980,"statfs":{"total":96636764160,"available":96241643520,"internally_reserved":0,"allocated":90173440,"data_stored":90014462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,2,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.226}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5220000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536448,"kb_used_data":222008,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93835392,"statfs":{"total":96636764160,"available":96087441408,"internally_reserved":0,"allocated":227336192,"data_stored":227177214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1139999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":2.7610000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":4.9630000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.0860000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":10.057}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227479552,"data_stored":227479552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91398144,"data_stored":91398144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227237888,"data_stored":227237888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224993280,"data_stored":224993280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:45.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:46.041 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:46.041 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:46.071 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-18T11:45:45.849256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459269492,"num_objects":56068,"num_object_clones":0,"num_object_copies":224272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56095,"num_write_kb":448642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56122,"ondisk_log_size":56122,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1993380,"kb_used_data":765124,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375493980,"statfs":{"total":386547056640,"available":384505835520,"internally_reserved":0,"allocated":783486976,"data_stored":782851064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[22,4,3,9,13,12,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37380096,"num_objects":4563,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4563,"num_write_kb":36504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003002"},"pg_stats":[{"pgid":"1.f","version":"31'3626","reported_seq":3667,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.406418+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:44.406418+0000","last_peered":"2024-09-18T11:45:44.406418+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:44.406418+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3626,"log_dups_size":0,"ondisk_log_size":3626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29704192,"num_objects":3626,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":29008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3485","reported_seq":3548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.098322+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:41.098322+0000","last_peered":"2024-09-18T11:45:41.098322+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:41.098322+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28513526,"num_objects":3481,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3456","reported_seq":3538,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.930923+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:40.930923+0000","last_peered":"2024-09-18T11:45:40.930923+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:40.930923+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28285563,"num_objects":3453,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":27630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1399}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3484","reported_seq":3546,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.185482+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:41.185482+0000","last_peered":"2024-09-18T11:45:41.185482+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:41.185482+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3484,"log_dups_size":0,"ondisk_log_size":3484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505334,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3519","reported_seq":3574,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.368570+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:44.368570+0000","last_peered":"2024-09-18T11:45:44.368570+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:44.368570+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3519,"log_dups_size":0,"ondisk_log_size":3519,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28800246,"num_objects":3516,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3517,"num_write_kb":28133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3551","reported_seq":3594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.343245+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:44.343245+0000","last_peered":"2024-09-18T11:45:44.343245+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:44.343245+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3551,"log_dups_size":0,"ondisk_log_size":3551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29089792,"num_objects":3551,"num_object_clones":0,"num_object_copies":14204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3547","reported_seq":3612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.182717+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:41.182717+0000","last_peered":"2024-09-18T11:45:41.182717+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:41.182717+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29005046,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1407}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3391","reported_seq":3454,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.102991+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:41.102991+0000","last_peered":"2024-09-18T11:45:41.102991+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:41.102991+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27735286,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3389,"num_write_kb":27100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3633","reported_seq":3691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.389582+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:44.389582+0000","last_peered":"2024-09-18T11:45:44.389582+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:44.389582+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29690348,"num_objects":3625,"num_object_clones":0,"num_object_copies":14500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3630,"num_write_kb":29017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3368","reported_seq":3452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.861212+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:40.861212+0000","last_peered":"2024-09-18T11:45:40.861212+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:40.861212+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3368,"log_dups_size":0,"ondisk_log_size":3368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27571446,"num_objects":3366,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3366,"num_write_kb":26926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3536","reported_seq":3593,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.458738+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:44.458738+0000","last_peered":"2024-09-18T11:45:44.458738+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:44.458738+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28921713,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":28252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1468}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3615","reported_seq":3652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.441082+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:44.441082+0000","last_peered":"2024-09-18T11:45:44.441082+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:44.441082+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3615,"log_dups_size":0,"ondisk_log_size":3615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29604475,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":28911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3453","reported_seq":3529,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.973288+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:40.973288+0000","last_peered":"2024-09-18T11:45:40.973288+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:40.973288+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3453,"log_dups_size":0,"ondisk_log_size":3453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28277371,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3452,"num_write_kb":27615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1398}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3613","reported_seq":3662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.395073+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:44.395073+0000","last_peered":"2024-09-18T11:45:44.395073+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:44.395073+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29562102,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":28884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1451}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3465","reported_seq":5116,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.990207+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:40.990207+0000","last_peered":"2024-09-18T11:45:40.990207+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:40.990207+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3465,"log_dups_size":0,"ondisk_log_size":3465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28357878,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3463,"num_write_kb":27701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2155}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3380","reported_seq":3442,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.081197+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:41.081197+0000","last_peered":"2024-09-18T11:45:41.081197+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:41.081197+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3380,"log_dups_size":0,"ondisk_log_size":3380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27645174,"num_objects":3375,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3378,"num_write_kb":27012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":459269492,"num_objects":56068,"num_object_clones":0,"num_object_copies":224272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56095,"num_write_kb":448642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":786567168,"data_stored":786567168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56122,"ondisk_log_size":56122,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540748,"kb_used_data":226244,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93831092,"statfs":{"total":96636764160,"available":96083038208,"internally_reserved":0,"allocated":231673856,"data_stored":231514878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[22,0,0,0,2,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":4.0229999999999997},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":4.6120000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6959999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":7.8179999999999996},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":16.271000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518400,"kb_used_data":221048,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93853440,"statfs":{"total":96636764160,"available":96105922560,"internally_reserved":0,"allocated":226353152,"data_stored":226194174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":5.952},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":5.8899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0619999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":8.8659999999999997},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":8.8249999999999993}]}]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397784,"kb_used_data":95824,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93974056,"statfs":{"total":96636764160,"available":96229433344,"internally_reserved":0,"allocated":98123776,"data_stored":97964798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,7,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.802}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3179999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2960000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536448,"kb_used_data":222008,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93835392,"statfs":{"total":96636764160,"available":96087441408,"internally_reserved":0,"allocated":227336192,"data_stored":227177214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1139999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":2.7610000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":4.9630000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.0860000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":10.057}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227479552,"data_stored":227479552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99295232,"data_stored":99295232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227237888,"data_stored":227237888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":232554496,"data_stored":232554496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:46.072 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:46.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3626", 'reported_seq': 3667, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:44.406418+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:44.406418+0000', 'last_peered': '2024-09-18T11:45:44.406418+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:44.406418+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3626, 'log_dups_size': 0, 'ondisk_log_size': 3626, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29704192, 'num_objects': 3626, 'num_object_clones': 0, 'num_object_copies': 14504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3626, 'num_write_kb': 29008, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1496}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:46.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:46.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3485", 'reported_seq': 3548, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:41.098322+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:41.098322+0000', 'last_peered': '2024-09-18T11:45:41.098322+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:41.098322+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3485, 'log_dups_size': 0, 'ondisk_log_size': 3485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28513526, 'num_objects': 3481, 'num_object_clones': 0, 'num_object_copies': 13924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3483, 'num_write_kb': 27860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:46.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:46.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3456", 'reported_seq': 3538, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:40.930923+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:40.930923+0000', 'last_peered': '2024-09-18T11:45:40.930923+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:40.930923+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3456, 'log_dups_size': 0, 'ondisk_log_size': 3456, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28285563, 'num_objects': 3453, 'num_object_clones': 0, 'num_object_copies': 13812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3455, 'num_write_kb': 27630, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1399}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:46.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:46.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3484", 'reported_seq': 3546, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:41.185482+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:41.185482+0000', 'last_peered': '2024-09-18T11:45:41.185482+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:41.185482+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3484, 'log_dups_size': 0, 'ondisk_log_size': 3484, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28505334, 'num_objects': 3480, 'num_object_clones': 0, 'num_object_copies': 13920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3482, 'num_write_kb': 27845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1366}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:46.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:46.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3519", 'reported_seq': 3574, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:44.368570+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:44.368570+0000', 'last_peered': '2024-09-18T11:45:44.368570+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:44.368570+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3519, 'log_dups_size': 0, 'ondisk_log_size': 3519, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28800246, 'num_objects': 3516, 'num_object_clones': 0, 'num_object_copies': 14064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3517, 'num_write_kb': 28133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1434}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3551", 'reported_seq': 3594, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:44.343245+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:44.343245+0000', 'last_peered': '2024-09-18T11:45:44.343245+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:44.343245+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3551, 'log_dups_size': 0, 'ondisk_log_size': 3551, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29089792, 'num_objects': 3551, 'num_object_clones': 0, 'num_object_copies': 14204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3551, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3551, 'num_write_kb': 28408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1397}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3547", 'reported_seq': 3612, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:41.182717+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:41.182717+0000', 'last_peered': '2024-09-18T11:45:41.182717+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:41.182717+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3547, 'log_dups_size': 0, 'ondisk_log_size': 3547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29005046, 'num_objects': 3541, 'num_object_clones': 0, 'num_object_copies': 14164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3545, 'num_write_kb': 28340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1407}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3391", 'reported_seq': 3454, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:41.102991+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:41.102991+0000', 'last_peered': '2024-09-18T11:45:41.102991+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:41.102991+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3391, 'log_dups_size': 0, 'ondisk_log_size': 3391, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27735286, 'num_objects': 3386, 'num_object_clones': 0, 'num_object_copies': 13544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3386, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3389, 'num_write_kb': 27100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1352}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3633", 'reported_seq': 3691, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:44.389582+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:44.389582+0000', 'last_peered': '2024-09-18T11:45:44.389582+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:44.389582+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3633, 'log_dups_size': 0, 'ondisk_log_size': 3633, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29690348, 'num_objects': 3625, 'num_object_clones': 0, 'num_object_copies': 14500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3625, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3630, 'num_write_kb': 29017, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1490}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3368", 'reported_seq': 3452, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:40.861212+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:40.861212+0000', 'last_peered': '2024-09-18T11:45:40.861212+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:40.861212+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3368, 'log_dups_size': 0, 'ondisk_log_size': 3368, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27571446, 'num_objects': 3366, 'num_object_clones': 0, 'num_object_copies': 13464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3366, 'num_write_kb': 26926, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1391}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:46.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3536", 'reported_seq': 3593, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:44.458738+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:44.458738+0000', 'last_peered': '2024-09-18T11:45:44.458738+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:44.458738+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3536, 'log_dups_size': 0, 'ondisk_log_size': 3536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28921713, 'num_objects': 3531, 'num_object_clones': 0, 'num_object_copies': 14124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3533, 'num_write_kb': 28252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1468}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3615", 'reported_seq': 3652, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:44.441082+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:44.441082+0000', 'last_peered': '2024-09-18T11:45:44.441082+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:44.441082+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3615, 'log_dups_size': 0, 'ondisk_log_size': 3615, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29604475, 'num_objects': 3614, 'num_object_clones': 0, 'num_object_copies': 14456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3614, 'num_write_kb': 28911, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3453", 'reported_seq': 3529, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:40.973288+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:40.973288+0000', 'last_peered': '2024-09-18T11:45:40.973288+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:40.973288+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3453, 'log_dups_size': 0, 'ondisk_log_size': 3453, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28277371, 'num_objects': 3452, 'num_object_clones': 0, 'num_object_copies': 13808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3452, 'num_write_kb': 27615, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1398}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3613", 'reported_seq': 3662, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:44.395073+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:44.395073+0000', 'last_peered': '2024-09-18T11:45:44.395073+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:44.395073+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3613, 'log_dups_size': 0, 'ondisk_log_size': 3613, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29562102, 'num_objects': 3609, 'num_object_clones': 0, 'num_object_copies': 14436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3611, 'num_write_kb': 28884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1451}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3465", 'reported_seq': 5116, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:40.990207+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:40.990207+0000', 'last_peered': '2024-09-18T11:45:40.990207+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:40.990207+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3465, 'log_dups_size': 0, 'ondisk_log_size': 3465, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28357878, 'num_objects': 3462, 'num_object_clones': 0, 'num_object_copies': 13848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3463, 'num_write_kb': 27701, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 786, 'num_bytes_recovered': 6422528, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:46.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3380", 'reported_seq': 3442, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:41.081197+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:41.081197+0000', 'last_peered': '2024-09-18T11:45:41.081197+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:41.081197+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3380, 'log_dups_size': 0, 'ondisk_log_size': 3380, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27645174, 'num_objects': 3375, 'num_object_clones': 0, 'num_object_copies': 13500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3375, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3378, 'num_write_kb': 27012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1362}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:46.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:46.221 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 89 20 57385 57365 5.03464 2.58594 0.0387286 0.0310136 2024-09-18T11:45:46.725 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:46.726 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32643/224272 objects degraded (14.555%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":144,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56068,"data_bytes":459269492,"bytes_used":2041221120,"bytes_avail":384505835520,"bytes_total":386547056640,"degraded_objects":32643,"degraded_total":224272,"degraded_ratio":0.14555093814653636,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105104,"recovering_keys_per_sec":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"write_bytes_sec":3114228,"read_op_per_sec":0,"write_op_per_sec":380},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:45:46.748 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:46.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:47.221 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 90 20 57723 57703 5.00804 2.64062 0.0723107 0.0311857 2024-09-18T11:45:47.295 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:47.295 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:47.323 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-18T11:45:45.849256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459269492,"num_objects":56068,"num_object_clones":0,"num_object_copies":224272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56095,"num_write_kb":448642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56122,"ondisk_log_size":56122,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1993380,"kb_used_data":765124,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375493980,"statfs":{"total":386547056640,"available":384505835520,"internally_reserved":0,"allocated":783486976,"data_stored":782851064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[22,4,3,9,13,12,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37380096,"num_objects":4563,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4563,"num_write_kb":36504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003002"},"pg_stats":[{"pgid":"1.f","version":"31'3626","reported_seq":3667,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.406418+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:44.406418+0000","last_peered":"2024-09-18T11:45:44.406418+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:44.406418+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3626,"log_dups_size":0,"ondisk_log_size":3626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29704192,"num_objects":3626,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":29008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3485","reported_seq":3548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.098322+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:41.098322+0000","last_peered":"2024-09-18T11:45:41.098322+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:41.098322+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28513526,"num_objects":3481,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":27860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3456","reported_seq":3538,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.930923+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:40.930923+0000","last_peered":"2024-09-18T11:45:40.930923+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:40.930923+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28285563,"num_objects":3453,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":27630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1399}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3484","reported_seq":3546,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.185482+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:41.185482+0000","last_peered":"2024-09-18T11:45:41.185482+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:41.185482+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3484,"log_dups_size":0,"ondisk_log_size":3484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505334,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3519","reported_seq":3574,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.368570+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:44.368570+0000","last_peered":"2024-09-18T11:45:44.368570+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:44.368570+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3519,"log_dups_size":0,"ondisk_log_size":3519,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28800246,"num_objects":3516,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3517,"num_write_kb":28133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3551","reported_seq":3594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.343245+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:44.343245+0000","last_peered":"2024-09-18T11:45:44.343245+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:44.343245+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3551,"log_dups_size":0,"ondisk_log_size":3551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29089792,"num_objects":3551,"num_object_clones":0,"num_object_copies":14204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3547","reported_seq":3612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.182717+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:41.182717+0000","last_peered":"2024-09-18T11:45:41.182717+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:41.182717+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29005046,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1407}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3391","reported_seq":3454,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.102991+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:41.102991+0000","last_peered":"2024-09-18T11:45:41.102991+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:41.102991+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27735286,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3389,"num_write_kb":27100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3633","reported_seq":3691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.389582+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:44.389582+0000","last_peered":"2024-09-18T11:45:44.389582+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:44.389582+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29690348,"num_objects":3625,"num_object_clones":0,"num_object_copies":14500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3630,"num_write_kb":29017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3368","reported_seq":3452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.861212+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:40.861212+0000","last_peered":"2024-09-18T11:45:40.861212+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:40.861212+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3368,"log_dups_size":0,"ondisk_log_size":3368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27571446,"num_objects":3366,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3366,"num_write_kb":26926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3536","reported_seq":3593,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.458738+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:44.458738+0000","last_peered":"2024-09-18T11:45:44.458738+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:44.458738+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28921713,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":28252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1468}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3615","reported_seq":3652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.441082+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:44.441082+0000","last_peered":"2024-09-18T11:45:44.441082+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:44.441082+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3615,"log_dups_size":0,"ondisk_log_size":3615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29604475,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":28911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3453","reported_seq":3529,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.973288+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:40.973288+0000","last_peered":"2024-09-18T11:45:40.973288+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:40.973288+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3453,"log_dups_size":0,"ondisk_log_size":3453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28277371,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3452,"num_write_kb":27615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1398}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3613","reported_seq":3662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:44.395073+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:44.395073+0000","last_peered":"2024-09-18T11:45:44.395073+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:44.395073+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29562102,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":28884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1451}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3465","reported_seq":5116,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:40.990207+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:40.990207+0000","last_peered":"2024-09-18T11:45:40.990207+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:40.990207+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3465,"log_dups_size":0,"ondisk_log_size":3465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28357878,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3463,"num_write_kb":27701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2155}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3380","reported_seq":3442,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:41.081197+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:41.081197+0000","last_peered":"2024-09-18T11:45:41.081197+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:41.081197+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3380,"log_dups_size":0,"ondisk_log_size":3380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27645174,"num_objects":3375,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3378,"num_write_kb":27012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":459269492,"num_objects":56068,"num_object_clones":0,"num_object_copies":224272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56095,"num_write_kb":448642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":786567168,"data_stored":786567168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56122,"ondisk_log_size":56122,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540748,"kb_used_data":226244,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93831092,"statfs":{"total":96636764160,"available":96083038208,"internally_reserved":0,"allocated":231673856,"data_stored":231514878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[22,0,0,0,2,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":4.0229999999999997},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":4.6120000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6959999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":7.8179999999999996},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":16.271000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518400,"kb_used_data":221048,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93853440,"statfs":{"total":96636764160,"available":96105922560,"internally_reserved":0,"allocated":226353152,"data_stored":226194174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":5.952},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":5.8899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0619999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":8.8659999999999997},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":8.8249999999999993}]}]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397784,"kb_used_data":95824,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93974056,"statfs":{"total":96636764160,"available":96229433344,"internally_reserved":0,"allocated":98123776,"data_stored":97964798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,7,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.802}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3179999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2960000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536448,"kb_used_data":222008,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93835392,"statfs":{"total":96636764160,"available":96087441408,"internally_reserved":0,"allocated":227336192,"data_stored":227177214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1139999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":2.7610000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":4.9630000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.0860000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":10.057}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227479552,"data_stored":227479552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99295232,"data_stored":99295232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227237888,"data_stored":227237888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":232554496,"data_stored":232554496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:48.221 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 91 20 58112 58092 4.9864 3.03906 0.0752893 0.0313224 2024-09-18T11:45:49.221 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 92 20 58501 58481 4.96523 3.03906 0.0486433 0.0314598 2024-09-18T11:45:50.221 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 93 20 58889 58869 4.94444 3.03125 0.0428691 0.0315904 2024-09-18T11:45:50.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:50.892 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:50.896 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:50.929 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-18T11:45:49.850279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":474686836,"num_objects":57950,"num_object_clones":0,"num_object_copies":231800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32562,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57977,"num_write_kb":463698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":892,"num_bytes_recovered":7086080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58004,"ondisk_log_size":58004,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2006420,"kb_used_data":794804,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375480940,"statfs":{"total":386547056640,"available":384492482560,"internally_reserved":0,"allocated":813879296,"data_stored":813243384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,4,9,20,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37273600,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4550,"num_write_kb":36400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002813"},"pg_stats":[{"pgid":"1.f","version":"31'3760","reported_seq":3800,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.508646+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:49.508646+0000","last_peered":"2024-09-18T11:45:49.508646+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:49.508646+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3760,"num_write_kb":30080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3592","reported_seq":3655,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.048209+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:46.048209+0000","last_peered":"2024-09-18T11:45:46.048209+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:46.048209+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3592,"log_dups_size":0,"ondisk_log_size":3592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29390070,"num_objects":3588,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3590,"num_write_kb":28716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3564","reported_seq":3645,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:45.998567+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:45.998567+0000","last_peered":"2024-09-18T11:45:45.998567+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:45.998567+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29170299,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3563,"num_write_kb":28494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1507}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3607","reported_seq":3669,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.150324+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:46.150324+0000","last_peered":"2024-09-18T11:45:46.150324+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:46.150324+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3607,"log_dups_size":0,"ondisk_log_size":3607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29512950,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":28829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1489}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3633","reported_seq":3688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.489123+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:49.489123+0000","last_peered":"2024-09-18T11:45:49.489123+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:49.489123+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734134,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3631,"num_write_kb":29045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3668","reported_seq":3711,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.488167+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:49.488167+0000","last_peered":"2024-09-18T11:45:49.488167+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:49.488167+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3668,"log_dups_size":0,"ondisk_log_size":3668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30048256,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":29344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3652","reported_seq":3716,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.093560+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:46.093560+0000","last_peered":"2024-09-18T11:45:46.093560+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:46.093560+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3652,"log_dups_size":0,"ondisk_log_size":3652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29865206,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":29180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1512}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3520","reported_seq":3583,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.101716+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:46.101716+0000","last_peered":"2024-09-18T11:45:46.101716+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:46.101716+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3520,"log_dups_size":0,"ondisk_log_size":3520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28792054,"num_objects":3515,"num_object_clones":0,"num_object_copies":14060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3518,"num_write_kb":28132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1481}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3747","reported_seq":3805,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.424845+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:49.424845+0000","last_peered":"2024-09-18T11:45:49.424845+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:49.424845+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3747,"log_dups_size":0,"ondisk_log_size":3747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30624236,"num_objects":3739,"num_object_clones":0,"num_object_copies":14956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":29929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3486","reported_seq":3570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:45.949262+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:45.949262+0000","last_peered":"2024-09-18T11:45:45.949262+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:45.949262+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28538102,"num_objects":3484,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":27870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3672","reported_seq":3729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.483998+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:49.483998+0000","last_peered":"2024-09-18T11:45:49.483998+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:49.483998+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3672,"log_dups_size":0,"ondisk_log_size":3672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30035825,"num_objects":3667,"num_object_clones":0,"num_object_copies":14668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3669,"num_write_kb":29340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3728","reported_seq":3765,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.436972+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:49.436972+0000","last_peered":"2024-09-18T11:45:49.436972+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:49.436972+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3728,"log_dups_size":0,"ondisk_log_size":3728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30530171,"num_objects":3727,"num_object_clones":0,"num_object_copies":14908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3727,"num_write_kb":29815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1618}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3570","reported_seq":3646,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:45.946793+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:45.946793+0000","last_peered":"2024-09-18T11:45:45.946793+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:45.946793+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3570,"log_dups_size":0,"ondisk_log_size":3570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29235835,"num_objects":3569,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":28551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1515}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3719","reported_seq":3768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.491513+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:49.491513+0000","last_peered":"2024-09-18T11:45:49.491513+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:49.491513+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30430454,"num_objects":3715,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1557}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3572","reported_seq":5385,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.031596+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:46.031596+0000","last_peered":"2024-09-18T11:45:46.031596+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:46.031596+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29234422,"num_objects":3569,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":28557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":7086080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2343}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3514","reported_seq":3576,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.204224+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:46.204224+0000","last_peered":"2024-09-18T11:45:46.204224+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:46.204224+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28742902,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3512,"num_write_kb":28084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":474686836,"num_objects":57950,"num_object_clones":0,"num_object_copies":231800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32562,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57977,"num_write_kb":463698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":892,"num_bytes_recovered":7086080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":817045504,"data_stored":817045504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58004,"ondisk_log_size":58004,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535564,"kb_used_data":233540,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93836276,"statfs":{"total":96636764160,"available":96088346624,"internally_reserved":0,"allocated":239144960,"data_stored":238985982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":4.0229999999999997},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":4.6120000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.976},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":2.5430000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":2.3359999999999999},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":0.501}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530076,"kb_used_data":228564,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93841764,"statfs":{"total":96636764160,"available":96093966336,"internally_reserved":0,"allocated":234049536,"data_stored":233890558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":3.9460000000000002},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.40699999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":16.527000000000001},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":3.9140000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":3.9750000000000001},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":16.673999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409508,"kb_used_data":103388,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93962332,"statfs":{"total":96636764160,"available":96217427968,"internally_reserved":0,"allocated":105869312,"data_stored":105710334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":5.9729999999999999},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":7.3570000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":4.4800000000000004},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":19.876000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":1.6759999999999999},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":14.195}]}]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531272,"kb_used_data":229312,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93840568,"statfs":{"total":96636764160,"available":96092741632,"internally_reserved":0,"allocated":234815488,"data_stored":234656510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,0,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3689999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":8.3789999999999996},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":5.4269999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":7.4059999999999997},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":3.1200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235040768,"data_stored":235040768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107053056,"data_stored":107053056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":234852352,"data_stored":234852352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":240099328,"data_stored":240099328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:50.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:51.224 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 94 20 59235 59215 4.92047 2.70312 0.0851618 0.0317416 2024-09-18T11:45:51.505 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:51.506 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:51.528 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-18T11:45:49.850279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":474686836,"num_objects":57950,"num_object_clones":0,"num_object_copies":231800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32562,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57977,"num_write_kb":463698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":892,"num_bytes_recovered":7086080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58004,"ondisk_log_size":58004,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2006420,"kb_used_data":794804,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375480940,"statfs":{"total":386547056640,"available":384492482560,"internally_reserved":0,"allocated":813879296,"data_stored":813243384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,4,9,20,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37273600,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4550,"num_write_kb":36400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002813"},"pg_stats":[{"pgid":"1.f","version":"31'3760","reported_seq":3800,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.508646+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:49.508646+0000","last_peered":"2024-09-18T11:45:49.508646+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:49.508646+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3760,"num_write_kb":30080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3592","reported_seq":3655,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.048209+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:46.048209+0000","last_peered":"2024-09-18T11:45:46.048209+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:46.048209+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3592,"log_dups_size":0,"ondisk_log_size":3592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29390070,"num_objects":3588,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3590,"num_write_kb":28716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3564","reported_seq":3645,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:45.998567+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:45.998567+0000","last_peered":"2024-09-18T11:45:45.998567+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:45.998567+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29170299,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3563,"num_write_kb":28494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1507}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3607","reported_seq":3669,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.150324+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:46.150324+0000","last_peered":"2024-09-18T11:45:46.150324+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:46.150324+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3607,"log_dups_size":0,"ondisk_log_size":3607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29512950,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":28829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1489}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3633","reported_seq":3688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.489123+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:49.489123+0000","last_peered":"2024-09-18T11:45:49.489123+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:49.489123+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734134,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3631,"num_write_kb":29045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3668","reported_seq":3711,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.488167+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:49.488167+0000","last_peered":"2024-09-18T11:45:49.488167+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:49.488167+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3668,"log_dups_size":0,"ondisk_log_size":3668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30048256,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":29344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3652","reported_seq":3716,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.093560+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:46.093560+0000","last_peered":"2024-09-18T11:45:46.093560+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:46.093560+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3652,"log_dups_size":0,"ondisk_log_size":3652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29865206,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":29180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1512}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3520","reported_seq":3583,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.101716+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:46.101716+0000","last_peered":"2024-09-18T11:45:46.101716+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:46.101716+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3520,"log_dups_size":0,"ondisk_log_size":3520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28792054,"num_objects":3515,"num_object_clones":0,"num_object_copies":14060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3518,"num_write_kb":28132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1481}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3747","reported_seq":3805,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.424845+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:49.424845+0000","last_peered":"2024-09-18T11:45:49.424845+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:49.424845+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3747,"log_dups_size":0,"ondisk_log_size":3747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30624236,"num_objects":3739,"num_object_clones":0,"num_object_copies":14956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":29929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3486","reported_seq":3570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:45.949262+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:45.949262+0000","last_peered":"2024-09-18T11:45:45.949262+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:45.949262+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28538102,"num_objects":3484,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":27870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3672","reported_seq":3729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.483998+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:49.483998+0000","last_peered":"2024-09-18T11:45:49.483998+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:49.483998+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3672,"log_dups_size":0,"ondisk_log_size":3672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30035825,"num_objects":3667,"num_object_clones":0,"num_object_copies":14668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3669,"num_write_kb":29340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3728","reported_seq":3765,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.436972+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:49.436972+0000","last_peered":"2024-09-18T11:45:49.436972+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:49.436972+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3728,"log_dups_size":0,"ondisk_log_size":3728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30530171,"num_objects":3727,"num_object_clones":0,"num_object_copies":14908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3727,"num_write_kb":29815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1618}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3570","reported_seq":3646,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:45.946793+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:45.946793+0000","last_peered":"2024-09-18T11:45:45.946793+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:45.946793+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3570,"log_dups_size":0,"ondisk_log_size":3570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29235835,"num_objects":3569,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":28551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1515}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3719","reported_seq":3768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.491513+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:49.491513+0000","last_peered":"2024-09-18T11:45:49.491513+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:49.491513+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30430454,"num_objects":3715,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1557}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3572","reported_seq":5385,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.031596+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:46.031596+0000","last_peered":"2024-09-18T11:45:46.031596+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:46.031596+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29234422,"num_objects":3569,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":28557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":7086080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2343}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3514","reported_seq":3576,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:46.204224+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:46.204224+0000","last_peered":"2024-09-18T11:45:46.204224+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:46.204224+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28742902,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3512,"num_write_kb":28084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":474686836,"num_objects":57950,"num_object_clones":0,"num_object_copies":231800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32562,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57977,"num_write_kb":463698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":892,"num_bytes_recovered":7086080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":817045504,"data_stored":817045504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58004,"ondisk_log_size":58004,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535564,"kb_used_data":233540,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93836276,"statfs":{"total":96636764160,"available":96088346624,"internally_reserved":0,"allocated":239144960,"data_stored":238985982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":4.0229999999999997},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":4.6120000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.976},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":2.5430000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":2.3359999999999999},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":0.501}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530076,"kb_used_data":228564,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93841764,"statfs":{"total":96636764160,"available":96093966336,"internally_reserved":0,"allocated":234049536,"data_stored":233890558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":3.9460000000000002},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.40699999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":16.527000000000001},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":3.9140000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":3.9750000000000001},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":16.673999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409508,"kb_used_data":103388,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93962332,"statfs":{"total":96636764160,"available":96217427968,"internally_reserved":0,"allocated":105869312,"data_stored":105710334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":5.9729999999999999},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":7.3570000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":4.4800000000000004},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":19.876000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":1.6759999999999999},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":14.195}]}]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531272,"kb_used_data":229312,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93840568,"statfs":{"total":96636764160,"available":96092741632,"internally_reserved":0,"allocated":234815488,"data_stored":234656510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,0,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3689999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":8.3789999999999996},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":5.4269999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":7.4059999999999997},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":3.1200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235040768,"data_stored":235040768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107053056,"data_stored":107053056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":234852352,"data_stored":234852352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":240099328,"data_stored":240099328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:51.529 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:51.530 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3760", 'reported_seq': 3800, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:49.508646+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:49.508646+0000', 'last_peered': '2024-09-18T11:45:49.508646+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:49.508646+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3760, 'log_dups_size': 0, 'ondisk_log_size': 3760, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30801920, 'num_objects': 3760, 'num_object_clones': 0, 'num_object_copies': 15040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3760, 'num_write_kb': 30080, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1630}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:51.530 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:51.530 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3592", 'reported_seq': 3655, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:46.048209+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:46.048209+0000', 'last_peered': '2024-09-18T11:45:46.048209+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:46.048209+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3592, 'log_dups_size': 0, 'ondisk_log_size': 3592, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29390070, 'num_objects': 3588, 'num_object_clones': 0, 'num_object_copies': 14352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3590, 'num_write_kb': 28716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1486}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:51.530 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:51.530 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3564", 'reported_seq': 3645, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:45.998567+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:45.998567+0000', 'last_peered': '2024-09-18T11:45:45.998567+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:45.998567+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3564, 'log_dups_size': 0, 'ondisk_log_size': 3564, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29170299, 'num_objects': 3561, 'num_object_clones': 0, 'num_object_copies': 14244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3563, 'num_write_kb': 28494, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1507}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:51.530 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:51.530 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3607", 'reported_seq': 3669, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:46.150324+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:46.150324+0000', 'last_peered': '2024-09-18T11:45:46.150324+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:46.150324+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3607, 'log_dups_size': 0, 'ondisk_log_size': 3607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29512950, 'num_objects': 3603, 'num_object_clones': 0, 'num_object_copies': 14412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3605, 'num_write_kb': 28829, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:51.530 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:51.530 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3633", 'reported_seq': 3688, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:49.489123+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:49.489123+0000', 'last_peered': '2024-09-18T11:45:49.489123+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:49.489123+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3633, 'log_dups_size': 0, 'ondisk_log_size': 3633, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29734134, 'num_objects': 3630, 'num_object_clones': 0, 'num_object_copies': 14520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3631, 'num_write_kb': 29045, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1548}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3668", 'reported_seq': 3711, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:49.488167+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:49.488167+0000', 'last_peered': '2024-09-18T11:45:49.488167+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:49.488167+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3668, 'log_dups_size': 0, 'ondisk_log_size': 3668, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30048256, 'num_objects': 3668, 'num_object_clones': 0, 'num_object_copies': 14672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3668, 'num_write_kb': 29344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1514}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3652", 'reported_seq': 3716, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:46.093560+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:46.093560+0000', 'last_peered': '2024-09-18T11:45:46.093560+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:46.093560+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3652, 'log_dups_size': 0, 'ondisk_log_size': 3652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29865206, 'num_objects': 3646, 'num_object_clones': 0, 'num_object_copies': 14584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3650, 'num_write_kb': 29180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1512}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3520", 'reported_seq': 3583, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:46.101716+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:46.101716+0000', 'last_peered': '2024-09-18T11:45:46.101716+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:46.101716+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3520, 'log_dups_size': 0, 'ondisk_log_size': 3520, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28792054, 'num_objects': 3515, 'num_object_clones': 0, 'num_object_copies': 14060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3518, 'num_write_kb': 28132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1481}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3747", 'reported_seq': 3805, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:49.424845+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:49.424845+0000', 'last_peered': '2024-09-18T11:45:49.424845+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:49.424845+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3747, 'log_dups_size': 0, 'ondisk_log_size': 3747, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30624236, 'num_objects': 3739, 'num_object_clones': 0, 'num_object_copies': 14956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3739, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3744, 'num_write_kb': 29929, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1604}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3486", 'reported_seq': 3570, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:45.949262+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:45.949262+0000', 'last_peered': '2024-09-18T11:45:45.949262+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:45.949262+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3486, 'log_dups_size': 0, 'ondisk_log_size': 3486, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28538102, 'num_objects': 3484, 'num_object_clones': 0, 'num_object_copies': 13936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3484, 'num_write_kb': 27870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1509}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:51.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3672", 'reported_seq': 3729, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:49.483998+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:49.483998+0000', 'last_peered': '2024-09-18T11:45:49.483998+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:49.483998+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3672, 'log_dups_size': 0, 'ondisk_log_size': 3672, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30035825, 'num_objects': 3667, 'num_object_clones': 0, 'num_object_copies': 14668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3669, 'num_write_kb': 29340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1604}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3728", 'reported_seq': 3765, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:49.436972+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:49.436972+0000', 'last_peered': '2024-09-18T11:45:49.436972+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:49.436972+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3728, 'log_dups_size': 0, 'ondisk_log_size': 3728, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30530171, 'num_objects': 3727, 'num_object_clones': 0, 'num_object_copies': 14908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3727, 'num_write_kb': 29815, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1618}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3570", 'reported_seq': 3646, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:45.946793+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:45.946793+0000', 'last_peered': '2024-09-18T11:45:45.946793+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:45.946793+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3570, 'log_dups_size': 0, 'ondisk_log_size': 3570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29235835, 'num_objects': 3569, 'num_object_clones': 0, 'num_object_copies': 14276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3569, 'num_write_kb': 28551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1515}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3719", 'reported_seq': 3768, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:49.491513+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:49.491513+0000', 'last_peered': '2024-09-18T11:45:49.491513+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:49.491513+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3719, 'log_dups_size': 0, 'ondisk_log_size': 3719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30430454, 'num_objects': 3715, 'num_object_clones': 0, 'num_object_copies': 14860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3717, 'num_write_kb': 29732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3572", 'reported_seq': 5385, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:46.031596+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:46.031596+0000', 'last_peered': '2024-09-18T11:45:46.031596+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:46.031596+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3572, 'log_dups_size': 0, 'ondisk_log_size': 3572, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29234422, 'num_objects': 3569, 'num_object_clones': 0, 'num_object_copies': 14276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3570, 'num_write_kb': 28557, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 867, 'num_bytes_recovered': 7086080, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2343}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3514", 'reported_seq': 3576, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:46.204224+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:46.204224+0000', 'last_peered': '2024-09-18T11:45:46.204224+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:46.204224+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3514, 'log_dups_size': 0, 'ondisk_log_size': 3514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28742902, 'num_objects': 3509, 'num_object_clones': 0, 'num_object_copies': 14036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3512, 'num_write_kb': 28084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1496}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:51.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:52.159 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:52.159 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32490/235896 objects degraded (13.773%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":150,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58974,"data_bytes":483075444,"bytes_used":2078236672,"bytes_avail":384468819968,"bytes_total":386547056640,"degraded_objects":32490,"degraded_total":235896,"degraded_ratio":0.137730186183742,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":156290,"recovering_keys_per_sec":0,"num_objects_recovered":229,"num_bytes_recovered":1875968,"num_keys_recovered":0,"write_bytes_sec":3268444,"read_op_per_sec":0,"write_op_per_sec":398},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:45:52.182 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:52.183 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:52.224 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 95 20 59573 59553 4.89647 2.64062 0.053276 0.0318953 2024-09-18T11:45:52.770 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:52.770 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:52.798 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-18T11:45:51.851098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":483075444,"num_objects":58974,"num_object_clones":0,"num_object_copies":235896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59001,"num_write_kb":471890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":7675904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59028,"ondisk_log_size":59028,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2029528,"kb_used_data":809592,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375457832,"statfs":{"total":386547056640,"available":384468819968,"internally_reserved":0,"allocated":829022208,"data_stored":828386296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,8,2,8,8,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39231488,"num_objects":4789,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":1875968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003106"},"pg_stats":[{"pgid":"1.f","version":"31'3760","reported_seq":3800,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.508646+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:49.508646+0000","last_peered":"2024-09-18T11:45:49.508646+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:49.508646+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3760,"num_write_kb":30080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3709","reported_seq":3772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.220894+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:51.220894+0000","last_peered":"2024-09-18T11:45:51.220894+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:51.220894+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30348534,"num_objects":3705,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3707,"num_write_kb":29652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3667","reported_seq":3749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.041695+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:51.041695+0000","last_peered":"2024-09-18T11:45:51.041695+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:51.041695+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3667,"log_dups_size":0,"ondisk_log_size":3667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30014075,"num_objects":3664,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3666,"num_write_kb":29318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3714","reported_seq":3774,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.186901+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:51.186901+0000","last_peered":"2024-09-18T11:45:51.186901+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:51.186901+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30389494,"num_objects":3710,"num_object_clones":0,"num_object_copies":14840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3712,"num_write_kb":29685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3633","reported_seq":3688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.489123+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:49.489123+0000","last_peered":"2024-09-18T11:45:49.489123+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:49.489123+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734134,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3631,"num_write_kb":29045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3668","reported_seq":3711,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.488167+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:49.488167+0000","last_peered":"2024-09-18T11:45:49.488167+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:49.488167+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3668,"log_dups_size":0,"ondisk_log_size":3668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30048256,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":29344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3767","reported_seq":3832,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.034436+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:51.034436+0000","last_peered":"2024-09-18T11:45:51.034436+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:51.034436+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30807286,"num_objects":3761,"num_object_clones":0,"num_object_copies":15044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":30100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1627}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3630","reported_seq":3692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.196128+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:51.196128+0000","last_peered":"2024-09-18T11:45:51.196128+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:51.196128+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29693174,"num_objects":3625,"num_object_clones":0,"num_object_copies":14500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3628,"num_write_kb":29012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3747","reported_seq":3805,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.424845+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:49.424845+0000","last_peered":"2024-09-18T11:45:49.424845+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:49.424845+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3747,"log_dups_size":0,"ondisk_log_size":3747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30624236,"num_objects":3739,"num_object_clones":0,"num_object_copies":14956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":29929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3605","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:50.991390+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:50.991390+0000","last_peered":"2024-09-18T11:45:50.991390+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:50.991390+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29512950,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3603,"num_write_kb":28822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1628}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3672","reported_seq":3729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.483998+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:49.483998+0000","last_peered":"2024-09-18T11:45:49.483998+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:49.483998+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3672,"log_dups_size":0,"ondisk_log_size":3672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30035825,"num_objects":3667,"num_object_clones":0,"num_object_copies":14668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3669,"num_write_kb":29340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3728","reported_seq":3765,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.436972+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:49.436972+0000","last_peered":"2024-09-18T11:45:49.436972+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:49.436972+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3728,"log_dups_size":0,"ondisk_log_size":3728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30530171,"num_objects":3727,"num_object_clones":0,"num_object_copies":14908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3727,"num_write_kb":29815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1618}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3702","reported_seq":3778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.013932+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:51.013932+0000","last_peered":"2024-09-18T11:45:51.013932+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:51.013932+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3702,"log_dups_size":0,"ondisk_log_size":3702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30317179,"num_objects":3701,"num_object_clones":0,"num_object_copies":14804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":29607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1647}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3719","reported_seq":3768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:49.491513+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:49.491513+0000","last_peered":"2024-09-18T11:45:49.491513+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:49.491513+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30430454,"num_objects":3715,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1557}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3680","reported_seq":5636,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.093781+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:51.093781+0000","last_peered":"2024-09-18T11:45:51.093781+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:51.093781+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3680,"log_dups_size":0,"ondisk_log_size":3680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30119158,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3678,"num_write_kb":29421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":7675904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3627","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.150196+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:51.150196+0000","last_peered":"2024-09-18T11:45:51.150196+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:51.150196+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29668598,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3625,"num_write_kb":28988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1609}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":483075444,"num_objects":58974,"num_object_clones":0,"num_object_copies":235896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59001,"num_write_kb":471890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":7675904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":832229376,"data_stored":832229376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59028,"ondisk_log_size":59028,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535564,"kb_used_data":233540,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93836276,"statfs":{"total":96636764160,"available":96088346624,"internally_reserved":0,"allocated":239144960,"data_stored":238985982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":4.0229999999999997},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":4.6120000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.976},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":2.5430000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":2.3359999999999999},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":0.501}]}]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541604,"kb_used_data":235932,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93830236,"statfs":{"total":96636764160,"available":96082161664,"internally_reserved":0,"allocated":241594368,"data_stored":241435390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,1,2,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":10.742000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.32400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":5.2130000000000001},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":4.3959999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":0.38100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409508,"kb_used_data":103388,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93962332,"statfs":{"total":96636764160,"available":96217427968,"internally_reserved":0,"allocated":105869312,"data_stored":105710334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":5.9729999999999999},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":7.3570000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":4.4800000000000004},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":19.876000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":1.6759999999999999},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":14.195}]}]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542852,"kb_used_data":236732,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93828988,"statfs":{"total":96636764160,"available":96080883712,"internally_reserved":0,"allocated":242413568,"data_stored":242254590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":3.532},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.35099999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.75},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.79300000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":2.3780000000000001},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":3.5739999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":242626560,"data_stored":242626560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107053056,"data_stored":107053056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242450432,"data_stored":242450432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":240099328,"data_stored":240099328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:53.224 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 96 20 59926 59906 4.87419 2.75781 0.0657876 0.0320425 2024-09-18T11:45:54.226 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 97 20 60309 60289 4.85479 2.99219 0.0576337 0.0321698 2024-09-18T11:45:55.224 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 98 20 60689 60669 4.83554 2.96875 0.053616 0.0322981 2024-09-18T11:45:55.801 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:56.224 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 99 20 61043 61023 4.81463 2.76562 0.0665143 0.0324399 2024-09-18T11:45:56.314 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:56.315 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:56.338 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-18T11:45:55.852803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":489629044,"num_objects":59774,"num_object_clones":0,"num_object_copies":239096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59801,"num_write_kb":478290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":7675904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59828,"ondisk_log_size":59828,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2052808,"kb_used_data":824548,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375434552,"statfs":{"total":386547056640,"available":384444981248,"internally_reserved":0,"allocated":844337152,"data_stored":843701240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,11,5,7,14,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36937728,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4509,"num_write_kb":36072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003939"},"pg_stats":[{"pgid":"1.f","version":"31'3875","reported_seq":3916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.514963+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:54.514963+0000","last_peered":"2024-09-18T11:45:54.514963+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:54.514963+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3875,"log_dups_size":0,"ondisk_log_size":3875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31744000,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3875,"num_write_kb":31000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3709","reported_seq":3772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.220894+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:51.220894+0000","last_peered":"2024-09-18T11:45:51.220894+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:51.220894+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30348534,"num_objects":3705,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3707,"num_write_kb":29652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3667","reported_seq":3749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.041695+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:51.041695+0000","last_peered":"2024-09-18T11:45:51.041695+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:51.041695+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3667,"log_dups_size":0,"ondisk_log_size":3667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30014075,"num_objects":3664,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3666,"num_write_kb":29318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3714","reported_seq":3774,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.186901+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:51.186901+0000","last_peered":"2024-09-18T11:45:51.186901+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:51.186901+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30389494,"num_objects":3710,"num_object_clones":0,"num_object_copies":14840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3712,"num_write_kb":29685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3742","reported_seq":3795,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.532749+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:54.532749+0000","last_peered":"2024-09-18T11:45:54.532749+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:54.532749+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30627062,"num_objects":3739,"num_object_clones":0,"num_object_copies":14956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3740,"num_write_kb":29917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3784","reported_seq":3827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.487068+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:54.487068+0000","last_peered":"2024-09-18T11:45:54.487068+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:54.487068+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30998528,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":30272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3767","reported_seq":3832,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.034436+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:51.034436+0000","last_peered":"2024-09-18T11:45:51.034436+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:51.034436+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30807286,"num_objects":3761,"num_object_clones":0,"num_object_copies":15044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":30100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1627}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3630","reported_seq":3692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.196128+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:51.196128+0000","last_peered":"2024-09-18T11:45:51.196128+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:51.196128+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29693174,"num_objects":3625,"num_object_clones":0,"num_object_copies":14500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3628,"num_write_kb":29012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3867","reported_seq":3925,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.541891+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:54.541891+0000","last_peered":"2024-09-18T11:45:54.541891+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:54.541891+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31607276,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3864,"num_write_kb":30889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1724}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3605","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:50.991390+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:50.991390+0000","last_peered":"2024-09-18T11:45:50.991390+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:50.991390+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29512950,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3603,"num_write_kb":28822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1628}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3784","reported_seq":3841,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.490662+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:54.490662+0000","last_peered":"2024-09-18T11:45:54.490662+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:54.490662+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30953329,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3781,"num_write_kb":30236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3849","reported_seq":3886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.431819+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:54.431819+0000","last_peered":"2024-09-18T11:45:54.431819+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:54.431819+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3849,"log_dups_size":0,"ondisk_log_size":3849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31521403,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":30783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1739}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3702","reported_seq":3778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.013932+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:51.013932+0000","last_peered":"2024-09-18T11:45:51.013932+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:51.013932+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3702,"log_dups_size":0,"ondisk_log_size":3702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30317179,"num_objects":3701,"num_object_clones":0,"num_object_copies":14804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":29607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1647}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3826","reported_seq":3875,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.407868+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:54.407868+0000","last_peered":"2024-09-18T11:45:54.407868+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:54.407868+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3826,"log_dups_size":0,"ondisk_log_size":3826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31306998,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3824,"num_write_kb":30588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1664}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3680","reported_seq":5636,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.093781+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:51.093781+0000","last_peered":"2024-09-18T11:45:51.093781+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:51.093781+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3680,"log_dups_size":0,"ondisk_log_size":3680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30119158,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3678,"num_write_kb":29421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":7675904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3627","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.150196+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:51.150196+0000","last_peered":"2024-09-18T11:45:51.150196+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:51.150196+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29668598,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3625,"num_write_kb":28988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1609}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":489629044,"num_objects":59774,"num_object_clones":0,"num_object_copies":239096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59801,"num_write_kb":478290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":7675904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":847396864,"data_stored":847396864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59828,"ondisk_log_size":59828,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546916,"kb_used_data":240732,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93824924,"statfs":{"total":96636764160,"available":96076722176,"internally_reserved":0,"allocated":246509568,"data_stored":246350590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":3.3370000000000002},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":3.2930000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":4.7859999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541604,"kb_used_data":235932,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93830236,"statfs":{"total":96636764160,"available":96082161664,"internally_reserved":0,"allocated":241594368,"data_stored":241435390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,1,2,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":10.742000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.32400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":5.2130000000000001},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":4.3959999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":0.38100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421436,"kb_used_data":111152,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93950404,"statfs":{"total":96636764160,"available":96205213696,"internally_reserved":0,"allocated":113819648,"data_stored":113660670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,1,3,10,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":5.1689999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":3.298},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":0.98899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":1.117},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":3.23}]}]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542852,"kb_used_data":236732,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93828988,"statfs":{"total":96636764160,"available":96080883712,"internally_reserved":0,"allocated":242413568,"data_stored":242254590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":3.532},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.35099999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.75},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.79300000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":2.3780000000000001},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":3.5739999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":242626560,"data_stored":242626560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114786304,"data_stored":114786304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242450432,"data_stored":242450432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247533568,"data_stored":247533568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:56.340 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:56.905 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:56.916 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:56.936 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-18T11:45:55.852803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":489629044,"num_objects":59774,"num_object_clones":0,"num_object_copies":239096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59801,"num_write_kb":478290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":7675904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59828,"ondisk_log_size":59828,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2052808,"kb_used_data":824548,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375434552,"statfs":{"total":386547056640,"available":384444981248,"internally_reserved":0,"allocated":844337152,"data_stored":843701240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,11,5,7,14,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36937728,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4509,"num_write_kb":36072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003939"},"pg_stats":[{"pgid":"1.f","version":"31'3875","reported_seq":3916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.514963+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:54.514963+0000","last_peered":"2024-09-18T11:45:54.514963+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:54.514963+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3875,"log_dups_size":0,"ondisk_log_size":3875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31744000,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3875,"num_write_kb":31000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3709","reported_seq":3772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.220894+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:51.220894+0000","last_peered":"2024-09-18T11:45:51.220894+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:51.220894+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30348534,"num_objects":3705,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3707,"num_write_kb":29652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3667","reported_seq":3749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.041695+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:51.041695+0000","last_peered":"2024-09-18T11:45:51.041695+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:51.041695+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3667,"log_dups_size":0,"ondisk_log_size":3667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30014075,"num_objects":3664,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3666,"num_write_kb":29318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3714","reported_seq":3774,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.186901+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:51.186901+0000","last_peered":"2024-09-18T11:45:51.186901+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:51.186901+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30389494,"num_objects":3710,"num_object_clones":0,"num_object_copies":14840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3712,"num_write_kb":29685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3742","reported_seq":3795,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.532749+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:54.532749+0000","last_peered":"2024-09-18T11:45:54.532749+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:54.532749+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30627062,"num_objects":3739,"num_object_clones":0,"num_object_copies":14956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3740,"num_write_kb":29917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3784","reported_seq":3827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.487068+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:54.487068+0000","last_peered":"2024-09-18T11:45:54.487068+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:54.487068+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30998528,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":30272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3767","reported_seq":3832,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.034436+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:51.034436+0000","last_peered":"2024-09-18T11:45:51.034436+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:51.034436+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30807286,"num_objects":3761,"num_object_clones":0,"num_object_copies":15044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":30100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1627}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3630","reported_seq":3692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.196128+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:51.196128+0000","last_peered":"2024-09-18T11:45:51.196128+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:51.196128+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29693174,"num_objects":3625,"num_object_clones":0,"num_object_copies":14500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3628,"num_write_kb":29012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3867","reported_seq":3925,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.541891+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:54.541891+0000","last_peered":"2024-09-18T11:45:54.541891+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:54.541891+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31607276,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3864,"num_write_kb":30889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1724}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3605","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:50.991390+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:50.991390+0000","last_peered":"2024-09-18T11:45:50.991390+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:50.991390+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29512950,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3603,"num_write_kb":28822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1628}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3784","reported_seq":3841,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.490662+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:54.490662+0000","last_peered":"2024-09-18T11:45:54.490662+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:54.490662+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30953329,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3781,"num_write_kb":30236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3849","reported_seq":3886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.431819+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:54.431819+0000","last_peered":"2024-09-18T11:45:54.431819+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:54.431819+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3849,"log_dups_size":0,"ondisk_log_size":3849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31521403,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":30783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1739}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3702","reported_seq":3778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.013932+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:51.013932+0000","last_peered":"2024-09-18T11:45:51.013932+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:51.013932+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3702,"log_dups_size":0,"ondisk_log_size":3702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30317179,"num_objects":3701,"num_object_clones":0,"num_object_copies":14804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":29607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1647}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3826","reported_seq":3875,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.407868+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:54.407868+0000","last_peered":"2024-09-18T11:45:54.407868+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:54.407868+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3826,"log_dups_size":0,"ondisk_log_size":3826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31306998,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3824,"num_write_kb":30588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1664}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3680","reported_seq":5636,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.093781+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:51.093781+0000","last_peered":"2024-09-18T11:45:51.093781+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:51.093781+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3680,"log_dups_size":0,"ondisk_log_size":3680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30119158,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3678,"num_write_kb":29421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":7675904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3627","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:51.150196+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:51.150196+0000","last_peered":"2024-09-18T11:45:51.150196+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:51.150196+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29668598,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3625,"num_write_kb":28988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1609}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":489629044,"num_objects":59774,"num_object_clones":0,"num_object_copies":239096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59801,"num_write_kb":478290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":7675904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":847396864,"data_stored":847396864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59828,"ondisk_log_size":59828,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546916,"kb_used_data":240732,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93824924,"statfs":{"total":96636764160,"available":96076722176,"internally_reserved":0,"allocated":246509568,"data_stored":246350590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":3.3370000000000002},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":3.2930000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":4.7859999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541604,"kb_used_data":235932,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93830236,"statfs":{"total":96636764160,"available":96082161664,"internally_reserved":0,"allocated":241594368,"data_stored":241435390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,1,2,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":10.742000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.32400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":5.2130000000000001},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":4.3959999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":0.38100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421436,"kb_used_data":111152,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93950404,"statfs":{"total":96636764160,"available":96205213696,"internally_reserved":0,"allocated":113819648,"data_stored":113660670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,1,3,10,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":5.1689999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":3.298},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":0.98899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":1.117},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":3.23}]}]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542852,"kb_used_data":236732,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93828988,"statfs":{"total":96636764160,"available":96080883712,"internally_reserved":0,"allocated":242413568,"data_stored":242254590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":3.532},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.35099999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":0.75},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.79300000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":2.3780000000000001},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":3.5739999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":242626560,"data_stored":242626560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114786304,"data_stored":114786304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242450432,"data_stored":242450432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247533568,"data_stored":247533568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:56.938 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:45:56.938 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3875", 'reported_seq': 3916, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:54.514963+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:54.514963+0000', 'last_peered': '2024-09-18T11:45:54.514963+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:54.514963+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3875, 'log_dups_size': 0, 'ondisk_log_size': 3875, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31744000, 'num_objects': 3875, 'num_object_clones': 0, 'num_object_copies': 15500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3875, 'num_write_kb': 31000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1745}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:56.938 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:45:56.938 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3709", 'reported_seq': 3772, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:51.220894+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:45:51.220894+0000', 'last_peered': '2024-09-18T11:45:51.220894+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:45:51.220894+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3709, 'log_dups_size': 0, 'ondisk_log_size': 3709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30348534, 'num_objects': 3705, 'num_object_clones': 0, 'num_object_copies': 14820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3707, 'num_write_kb': 29652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1603}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:56.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:45:56.939 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3667", 'reported_seq': 3749, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:51.041695+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:45:51.041695+0000', 'last_peered': '2024-09-18T11:45:51.041695+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:45:51.041695+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3667, 'log_dups_size': 0, 'ondisk_log_size': 3667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30014075, 'num_objects': 3664, 'num_object_clones': 0, 'num_object_copies': 14656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3666, 'num_write_kb': 29318, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1610}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:56.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:45:56.939 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3714", 'reported_seq': 3774, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:51.186901+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:45:51.186901+0000', 'last_peered': '2024-09-18T11:45:51.186901+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:45:51.186901+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3714, 'log_dups_size': 0, 'ondisk_log_size': 3714, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30389494, 'num_objects': 3710, 'num_object_clones': 0, 'num_object_copies': 14840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3712, 'num_write_kb': 29685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1596}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:56.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:45:56.939 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3742", 'reported_seq': 3795, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:54.532749+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:54.532749+0000', 'last_peered': '2024-09-18T11:45:54.532749+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:54.532749+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3742, 'log_dups_size': 0, 'ondisk_log_size': 3742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30627062, 'num_objects': 3739, 'num_object_clones': 0, 'num_object_copies': 14956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3739, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3740, 'num_write_kb': 29917, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1657}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:56.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:45:56.939 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3784", 'reported_seq': 3827, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:54.487068+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:54.487068+0000', 'last_peered': '2024-09-18T11:45:54.487068+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:54.487068+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3784, 'log_dups_size': 0, 'ondisk_log_size': 3784, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30998528, 'num_objects': 3784, 'num_object_clones': 0, 'num_object_copies': 15136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3784, 'num_write_kb': 30272, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1630}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:56.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:45:56.939 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3767", 'reported_seq': 3832, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:51.034436+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:45:51.034436+0000', 'last_peered': '2024-09-18T11:45:51.034436+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:45:51.034436+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3767, 'log_dups_size': 0, 'ondisk_log_size': 3767, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30807286, 'num_objects': 3761, 'num_object_clones': 0, 'num_object_copies': 15044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3765, 'num_write_kb': 30100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1627}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:56.940 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:45:56.940 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3630", 'reported_seq': 3692, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:51.196128+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:45:51.196128+0000', 'last_peered': '2024-09-18T11:45:51.196128+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:45:51.196128+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3630, 'log_dups_size': 0, 'ondisk_log_size': 3630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29693174, 'num_objects': 3625, 'num_object_clones': 0, 'num_object_copies': 14500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3625, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3628, 'num_write_kb': 29012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:56.940 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:45:56.940 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3867", 'reported_seq': 3925, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:54.541891+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:54.541891+0000', 'last_peered': '2024-09-18T11:45:54.541891+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:54.541891+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3867, 'log_dups_size': 0, 'ondisk_log_size': 3867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31607276, 'num_objects': 3859, 'num_object_clones': 0, 'num_object_copies': 15436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3864, 'num_write_kb': 30889, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1724}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:56.940 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:45:56.940 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3605", 'reported_seq': 3689, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:50.991390+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:45:50.991390+0000', 'last_peered': '2024-09-18T11:45:50.991390+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:45:50.991390+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3605, 'log_dups_size': 0, 'ondisk_log_size': 3605, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29512950, 'num_objects': 3603, 'num_object_clones': 0, 'num_object_copies': 14412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3603, 'num_write_kb': 28822, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1628}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:56.940 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:45:56.940 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3784", 'reported_seq': 3841, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:54.490662+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:54.490662+0000', 'last_peered': '2024-09-18T11:45:54.490662+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:54.490662+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3784, 'log_dups_size': 0, 'ondisk_log_size': 3784, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30953329, 'num_objects': 3779, 'num_object_clones': 0, 'num_object_copies': 15116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3779, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3781, 'num_write_kb': 30236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1716}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:56.940 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:45:56.940 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3849", 'reported_seq': 3886, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:54.431819+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:54.431819+0000', 'last_peered': '2024-09-18T11:45:54.431819+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:54.431819+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3849, 'log_dups_size': 0, 'ondisk_log_size': 3849, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31521403, 'num_objects': 3848, 'num_object_clones': 0, 'num_object_copies': 15392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3848, 'num_write_kb': 30783, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1739}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:56.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:45:56.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3702", 'reported_seq': 3778, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:51.013932+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:45:51.013932+0000', 'last_peered': '2024-09-18T11:45:51.013932+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:45:51.013932+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3702, 'log_dups_size': 0, 'ondisk_log_size': 3702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30317179, 'num_objects': 3701, 'num_object_clones': 0, 'num_object_copies': 14804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3701, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3701, 'num_write_kb': 29607, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1647}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:56.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:45:56.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3826", 'reported_seq': 3875, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:54.407868+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:54.407868+0000', 'last_peered': '2024-09-18T11:45:54.407868+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:54.407868+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3826, 'log_dups_size': 0, 'ondisk_log_size': 3826, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31306998, 'num_objects': 3822, 'num_object_clones': 0, 'num_object_copies': 15288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3824, 'num_write_kb': 30588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1664}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:45:56.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:45:56.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3680", 'reported_seq': 5636, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:51.093781+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:45:51.093781+0000', 'last_peered': '2024-09-18T11:45:51.093781+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:45:51.093781+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3680, 'log_dups_size': 0, 'ondisk_log_size': 3680, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30119158, 'num_objects': 3677, 'num_object_clones': 0, 'num_object_copies': 14708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3678, 'num_write_kb': 29421, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 939, 'num_bytes_recovered': 7675904, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2523}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:45:56.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:45:56.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3627", 'reported_seq': 3689, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:51.150196+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:45:51.150196+0000', 'last_peered': '2024-09-18T11:45:51.150196+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:45:51.150196+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3627, 'log_dups_size': 0, 'ondisk_log_size': 3627, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29668598, 'num_objects': 3622, 'num_object_clones': 0, 'num_object_copies': 14488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3625, 'num_write_kb': 28988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1609}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:45:56.942 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:45:57.224 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:45:57.224037+0000 min lat: 0.00272854 max lat: 2.42146 avg lat: 0.0325888 2024-09-18T11:45:57.224 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:45:57.224 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 100 20 61379 61359 4.79274 2.625 0.0639824 0.0325888 2024-09-18T11:45:57.617 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:57.617 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32490/239096 objects degraded (13.589%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":155,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59774,"data_bytes":489629044,"bytes_used":2102075392,"bytes_avail":384444981248,"bytes_total":386547056640,"degraded_objects":32490,"degraded_total":239096,"degraded_ratio":0.13588684043229499,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":104413,"recovering_keys_per_sec":0,"num_objects_recovered":153,"num_bytes_recovered":1253376,"num_keys_recovered":0,"write_bytes_sec":3077133,"read_op_per_sec":0,"write_op_per_sec":375},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:45:57.642 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:45:57.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:45:58.206 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:45:58.207 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:45:58.224 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 101 20 61706 61686 4.77058 2.55469 0.0791696 0.0327396 2024-09-18T11:45:58.228 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-18T11:45:57.853443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":497870196,"num_objects":60780,"num_object_clones":0,"num_object_copies":243120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60807,"num_write_kb":486338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60834,"ondisk_log_size":60834,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2079840,"kb_used_data":839100,"kb_used_omap":14,"kb_used_meta":1240689,"kb_avail":375407520,"statfs":{"total":386547056640,"available":384417300480,"internally_reserved":0,"allocated":859238400,"data_stored":858602488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1270465563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,4,4,10,10,15,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38600704,"num_objects":4712,"num_object_clones":0,"num_object_copies":18848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4712,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1892352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004187"},"pg_stats":[{"pgid":"1.f","version":"31'3875","reported_seq":3916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.514963+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:54.514963+0000","last_peered":"2024-09-18T11:45:54.514963+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:54.514963+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3875,"log_dups_size":0,"ondisk_log_size":3875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31744000,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3875,"num_write_kb":31000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3811","reported_seq":3874,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.187534+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:56.187534+0000","last_peered":"2024-09-18T11:45:56.187534+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:56.187534+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3811,"log_dups_size":0,"ondisk_log_size":3811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31184118,"num_objects":3807,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":30468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3791","reported_seq":3873,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.001434+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:56.001434+0000","last_peered":"2024-09-18T11:45:56.001434+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:56.001434+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3791,"log_dups_size":0,"ondisk_log_size":3791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31029883,"num_objects":3788,"num_object_clones":0,"num_object_copies":15152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3790,"num_write_kb":30310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1734}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3817","reported_seq":3879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.134196+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:56.134196+0000","last_peered":"2024-09-18T11:45:56.134196+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:56.134196+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3817,"log_dups_size":0,"ondisk_log_size":3817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31233270,"num_objects":3813,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":30509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3742","reported_seq":3795,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.532749+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:54.532749+0000","last_peered":"2024-09-18T11:45:54.532749+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:54.532749+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30627062,"num_objects":3739,"num_object_clones":0,"num_object_copies":14956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3740,"num_write_kb":29917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3784","reported_seq":3827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.487068+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:54.487068+0000","last_peered":"2024-09-18T11:45:54.487068+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:54.487068+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30998528,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":30272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3873","reported_seq":3938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:55.919952+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:55.919952+0000","last_peered":"2024-09-18T11:45:55.919952+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:55.919952+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31675638,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3871,"num_write_kb":30948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1733}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3742","reported_seq":3805,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.170810+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:56.170810+0000","last_peered":"2024-09-18T11:45:56.170810+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:56.170810+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30610678,"num_objects":3737,"num_object_clones":0,"num_object_copies":14948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3740,"num_write_kb":29908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1703}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3867","reported_seq":3925,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.541891+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:54.541891+0000","last_peered":"2024-09-18T11:45:54.541891+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:54.541891+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31607276,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3864,"num_write_kb":30889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1724}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3704","reported_seq":3785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:55.864397+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:55.864397+0000","last_peered":"2024-09-18T11:45:55.864397+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:55.864397+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30323958,"num_objects":3702,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3702,"num_write_kb":29614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1727}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3784","reported_seq":3841,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.490662+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:54.490662+0000","last_peered":"2024-09-18T11:45:54.490662+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:54.490662+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30953329,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3781,"num_write_kb":30236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3849","reported_seq":3886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.431819+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:54.431819+0000","last_peered":"2024-09-18T11:45:54.431819+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:54.431819+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3849,"log_dups_size":0,"ondisk_log_size":3849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31521403,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":30783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1739}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3822","reported_seq":3898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.080726+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:56.080726+0000","last_peered":"2024-09-18T11:45:56.080726+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:56.080726+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3822,"log_dups_size":0,"ondisk_log_size":3822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31300219,"num_objects":3821,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3821,"num_write_kb":30567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3826","reported_seq":3875,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:54.407868+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:54.407868+0000","last_peered":"2024-09-18T11:45:54.407868+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:54.407868+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3826,"log_dups_size":0,"ondisk_log_size":3826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31306998,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3824,"num_write_kb":30588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1664}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3794","reported_seq":5907,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.092810+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:56.092810+0000","last_peered":"2024-09-18T11:45:56.092810+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:56.092810+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31053046,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":30333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2715}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3753","reported_seq":3814,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.191230+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:56.191230+0000","last_peered":"2024-09-18T11:45:56.191230+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:56.191230+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30700790,"num_objects":3748,"num_object_clones":0,"num_object_copies":14992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":29996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1735}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":497870196,"num_objects":60780,"num_object_clones":0,"num_object_copies":243120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60807,"num_write_kb":486338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":862248960,"data_stored":862248960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60834,"ondisk_log_size":60834,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546916,"kb_used_data":240732,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93824924,"statfs":{"total":96636764160,"available":96076722176,"internally_reserved":0,"allocated":246509568,"data_stored":246350590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":3.3370000000000002},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":3.2930000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":4.7859999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557208,"kb_used_data":243216,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93814632,"statfs":{"total":96636764160,"available":96066183168,"internally_reserved":0,"allocated":249053184,"data_stored":248894206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.17699999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":2.843},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":4.1879999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":4.1429999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":2.5840000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421436,"kb_used_data":111152,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93950404,"statfs":{"total":96636764160,"available":96205213696,"internally_reserved":0,"allocated":113819648,"data_stored":113660670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,1,3,10,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":5.1689999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":3.298},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":0.98899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":1.117},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":3.23}]}]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554280,"kb_used_data":244000,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93817560,"statfs":{"total":96636764160,"available":96069181440,"internally_reserved":0,"allocated":249856000,"data_stored":249697022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,2,4,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":5.0049999999999999},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.438}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":11.936999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":4.1360000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":250023936,"data_stored":250023936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114786304,"data_stored":114786304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249905152,"data_stored":249905152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247533568,"data_stored":247533568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:45:59.229 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 102 20 62086 62066 4.7527 2.96875 0.0431066 0.0328637 2024-09-18T11:46:00.229 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 103 20 62461 62441 4.735 2.92969 0.0444587 0.0329831 2024-09-18T11:46:01.229 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 104 20 62847 62827 4.71846 3.01562 0.103188 0.0330992 2024-09-18T11:46:01.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:01.748 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:01.750 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:01.767 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-18T11:45:59.854076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":504251764,"num_objects":61559,"num_object_clones":0,"num_object_copies":246236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61586,"num_write_kb":492570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61613,"ondisk_log_size":61613,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2090080,"kb_used_data":853504,"kb_used_omap":14,"kb_used_meta":1236529,"kb_avail":375397280,"statfs":{"total":386547056640,"available":384406814720,"internally_reserved":0,"allocated":873988096,"data_stored":873352184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1266205723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,1,2,9,14,6,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36397056,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":35544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004479"},"pg_stats":[{"pgid":"1.f","version":"31'3975","reported_seq":4015,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.559628+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:59.559628+0000","last_peered":"2024-09-18T11:45:59.559628+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:59.559628+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3975,"log_dups_size":0,"ondisk_log_size":3975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32563200,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":31800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3811","reported_seq":3874,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.187534+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:45:56.187534+0000","last_peered":"2024-09-18T11:45:56.187534+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:45:56.187534+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3811,"log_dups_size":0,"ondisk_log_size":3811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31184118,"num_objects":3807,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":30468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3791","reported_seq":3873,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.001434+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:45:56.001434+0000","last_peered":"2024-09-18T11:45:56.001434+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:45:56.001434+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3791,"log_dups_size":0,"ondisk_log_size":3791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31029883,"num_objects":3788,"num_object_clones":0,"num_object_copies":15152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3790,"num_write_kb":30310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1734}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3817","reported_seq":3879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.134196+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:45:56.134196+0000","last_peered":"2024-09-18T11:45:56.134196+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:45:56.134196+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3817,"log_dups_size":0,"ondisk_log_size":3817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31233270,"num_objects":3813,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":30509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3860","reported_seq":3915,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.307861+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:59.307861+0000","last_peered":"2024-09-18T11:45:59.307861+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:59.307861+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31593718,"num_objects":3857,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3858,"num_write_kb":30861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3889","reported_seq":3932,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.570789+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:59.570789+0000","last_peered":"2024-09-18T11:45:59.570789+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:59.570789+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3889,"log_dups_size":0,"ondisk_log_size":3889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31858688,"num_objects":3889,"num_object_clones":0,"num_object_copies":15556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3889,"num_write_kb":31112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3873","reported_seq":3938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:55.919952+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:45:55.919952+0000","last_peered":"2024-09-18T11:45:55.919952+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:45:55.919952+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31675638,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3871,"num_write_kb":30948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1733}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3742","reported_seq":3805,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.170810+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:45:56.170810+0000","last_peered":"2024-09-18T11:45:56.170810+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:45:56.170810+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30610678,"num_objects":3737,"num_object_clones":0,"num_object_copies":14948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3740,"num_write_kb":29908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1703}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3984","reported_seq":4042,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.523924+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:59.523924+0000","last_peered":"2024-09-18T11:45:59.523924+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:59.523924+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32565740,"num_objects":3976,"num_object_clones":0,"num_object_copies":15904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":31825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3704","reported_seq":3785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:55.864397+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:45:55.864397+0000","last_peered":"2024-09-18T11:45:55.864397+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:45:55.864397+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30323958,"num_objects":3702,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3702,"num_write_kb":29614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1727}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3894","reported_seq":3950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.487383+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:59.487383+0000","last_peered":"2024-09-18T11:45:59.487383+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:59.487383+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31854449,"num_objects":3889,"num_object_clones":0,"num_object_copies":15556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3891,"num_write_kb":31116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3966","reported_seq":4003,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.293938+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:59.293938+0000","last_peered":"2024-09-18T11:45:59.293938+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:59.293938+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32479867,"num_objects":3965,"num_object_clones":0,"num_object_copies":15860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3965,"num_write_kb":31719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1856}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3822","reported_seq":3898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.080726+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:45:56.080726+0000","last_peered":"2024-09-18T11:45:56.080726+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:45:56.080726+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3822,"log_dups_size":0,"ondisk_log_size":3822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31300219,"num_objects":3821,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3821,"num_write_kb":30567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3938","reported_seq":3985,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.531535+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:59.531535+0000","last_peered":"2024-09-18T11:45:59.531535+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:59.531535+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3938,"log_dups_size":0,"ondisk_log_size":3938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32224502,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3794","reported_seq":5907,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.092810+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:45:56.092810+0000","last_peered":"2024-09-18T11:45:56.092810+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:45:56.092810+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31053046,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":30333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2715}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3753","reported_seq":3814,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:56.191230+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:45:56.191230+0000","last_peered":"2024-09-18T11:45:56.191230+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:45:56.191230+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30700790,"num_objects":3748,"num_object_clones":0,"num_object_copies":14992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":29996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1735}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":504251764,"num_objects":61559,"num_object_clones":0,"num_object_copies":246236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61586,"num_write_kb":492570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":877166592,"data_stored":877166592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61613,"ondisk_log_size":61613,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558272,"kb_used_data":247928,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93813568,"statfs":{"total":96636764160,"available":96065093632,"internally_reserved":0,"allocated":253878272,"data_stored":253719294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,7,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":0.59399999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":7.5069999999999997},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.57499999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":4.9320000000000004},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":3.9350000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557208,"kb_used_data":243216,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93814632,"statfs":{"total":96636764160,"available":96066183168,"internally_reserved":0,"allocated":249053184,"data_stored":248894206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.17699999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":2.843},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":4.1879999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":4.1429999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":2.5840000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420320,"kb_used_data":118360,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93951520,"statfs":{"total":96636764160,"available":96206356480,"internally_reserved":0,"allocated":121200640,"data_stored":121041662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":2.9660000000000002},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":0.318}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":6.1950000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554280,"kb_used_data":244000,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93817560,"statfs":{"total":96636764160,"available":96069181440,"internally_reserved":0,"allocated":249856000,"data_stored":249697022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,2,4,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":5.0049999999999999},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.438}]},{"osd":2,"last update":"Wed Sep 18 11:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":11.936999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":4.1360000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":250023936,"data_stored":250023936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122425344,"data_stored":122425344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249905152,"data_stored":249905152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":254812160,"data_stored":254812160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:01.769 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:02.229 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 105 20 63171 63151 4.69763 2.53125 0.0558512 0.0332475 2024-09-18T11:46:02.327 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:02.327 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:02.352 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-18T11:46:01.854883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":512853364,"num_objects":62609,"num_object_clones":0,"num_object_copies":250436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62636,"num_write_kb":500970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62663,"ondisk_log_size":62663,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2079356,"kb_used_data":867736,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375408004,"statfs":{"total":386547056640,"available":384417796096,"internally_reserved":0,"allocated":888561664,"data_stored":887925752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,6,16,10,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38166528,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4659,"num_write_kb":37272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004604"},"pg_stats":[{"pgid":"1.f","version":"31'3975","reported_seq":4015,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.559628+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:59.559628+0000","last_peered":"2024-09-18T11:45:59.559628+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:59.559628+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3975,"log_dups_size":0,"ondisk_log_size":3975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32563200,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":31800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3941","reported_seq":4004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.244318+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:01.244318+0000","last_peered":"2024-09-18T11:46:01.244318+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:01.244318+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32249078,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3939,"num_write_kb":31508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3923","reported_seq":4005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.091697+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:01.091697+0000","last_peered":"2024-09-18T11:46:01.091697+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:01.091697+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32111227,"num_objects":3920,"num_object_clones":0,"num_object_copies":15680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3922,"num_write_kb":31366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3933","reported_seq":3995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.213784+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:01.213784+0000","last_peered":"2024-09-18T11:46:01.213784+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:01.213784+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183542,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3931,"num_write_kb":31437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3860","reported_seq":3915,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.307861+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:59.307861+0000","last_peered":"2024-09-18T11:45:59.307861+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:59.307861+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31593718,"num_objects":3857,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3858,"num_write_kb":30861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3889","reported_seq":3932,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.570789+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:59.570789+0000","last_peered":"2024-09-18T11:45:59.570789+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:59.570789+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3889,"log_dups_size":0,"ondisk_log_size":3889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31858688,"num_objects":3889,"num_object_clones":0,"num_object_copies":15556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3889,"num_write_kb":31112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3978","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.205341+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:01.205341+0000","last_peered":"2024-09-18T11:46:01.205341+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:01.205341+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535798,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3976,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3845","reported_seq":3908,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.161963+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:01.161963+0000","last_peered":"2024-09-18T11:46:01.161963+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:01.161963+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31454454,"num_objects":3840,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3984","reported_seq":4042,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.523924+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:59.523924+0000","last_peered":"2024-09-18T11:45:59.523924+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:59.523924+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32565740,"num_objects":3976,"num_object_clones":0,"num_object_copies":15904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":31825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3817","reported_seq":3901,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.150638+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:01.150638+0000","last_peered":"2024-09-18T11:46:01.150638+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:01.150638+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3817,"log_dups_size":0,"ondisk_log_size":3817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31249654,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":30518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1840}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3894","reported_seq":3950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.487383+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:59.487383+0000","last_peered":"2024-09-18T11:45:59.487383+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:59.487383+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31854449,"num_objects":3889,"num_object_clones":0,"num_object_copies":15556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3891,"num_write_kb":31116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3966","reported_seq":4003,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.293938+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:59.293938+0000","last_peered":"2024-09-18T11:45:59.293938+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:59.293938+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32479867,"num_objects":3965,"num_object_clones":0,"num_object_copies":15860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3965,"num_write_kb":31719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1856}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3935","reported_seq":4011,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.092757+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:01.092757+0000","last_peered":"2024-09-18T11:46:01.092757+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:01.092757+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3935,"log_dups_size":0,"ondisk_log_size":3935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32225915,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3934,"num_write_kb":31471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3938","reported_seq":3985,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.531535+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:59.531535+0000","last_peered":"2024-09-18T11:45:59.531535+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:59.531535+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3938,"log_dups_size":0,"ondisk_log_size":3938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32224502,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3917","reported_seq":6172,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.188302+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:01.188302+0000","last_peered":"2024-09-18T11:46:01.188302+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:01.188302+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3917,"log_dups_size":0,"ondisk_log_size":3917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32060662,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3915,"num_write_kb":31317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1089,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3868","reported_seq":3928,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.208164+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:01.208164+0000","last_peered":"2024-09-18T11:46:01.208164+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:01.208164+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3868,"log_dups_size":0,"ondisk_log_size":3868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642870,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":30916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1850}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":512853364,"num_objects":62609,"num_object_clones":0,"num_object_copies":250436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62636,"num_write_kb":500970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":892039168,"data_stored":892039168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62663,"ondisk_log_size":62663,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558272,"kb_used_data":247928,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93813568,"statfs":{"total":96636764160,"available":96065093632,"internally_reserved":0,"allocated":253878272,"data_stored":253719294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,7,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":0.59399999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":7.5069999999999997},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.57499999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":4.9320000000000004},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":3.9350000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551816,"kb_used_data":250300,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93820024,"statfs":{"total":96636764160,"available":96071704576,"internally_reserved":0,"allocated":256307200,"data_stored":256148222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.462},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":0.63400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":6.6029999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420320,"kb_used_data":118360,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93951520,"statfs":{"total":96636764160,"available":96206356480,"internally_reserved":0,"allocated":121200640,"data_stored":121041662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":2.9660000000000002},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":0.318}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":6.1950000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548948,"kb_used_data":251148,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93822892,"statfs":{"total":96636764160,"available":96074641408,"internally_reserved":0,"allocated":257175552,"data_stored":257016574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.248},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":1.994}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.378}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.016},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257445888,"data_stored":257445888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122425344,"data_stored":122425344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":257355776,"data_stored":257355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":254812160,"data_stored":254812160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:02.353 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:02.353 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3975", 'reported_seq': 4015, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:59.559628+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:45:59.559628+0000', 'last_peered': '2024-09-18T11:45:59.559628+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:45:59.559628+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3975, 'log_dups_size': 0, 'ondisk_log_size': 3975, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32563200, 'num_objects': 3975, 'num_object_clones': 0, 'num_object_copies': 15900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3975, 'num_write_kb': 31800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1845}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:02.353 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:02.353 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3941", 'reported_seq': 4004, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.244318+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:01.244318+0000', 'last_peered': '2024-09-18T11:46:01.244318+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:01.244318+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3941, 'log_dups_size': 0, 'ondisk_log_size': 3941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32249078, 'num_objects': 3937, 'num_object_clones': 0, 'num_object_copies': 15748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3939, 'num_write_kb': 31508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1835}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3923", 'reported_seq': 4005, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.091697+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:01.091697+0000', 'last_peered': '2024-09-18T11:46:01.091697+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:01.091697+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3923, 'log_dups_size': 0, 'ondisk_log_size': 3923, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32111227, 'num_objects': 3920, 'num_object_clones': 0, 'num_object_copies': 15680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3920, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3922, 'num_write_kb': 31366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1866}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3933", 'reported_seq': 3995, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.213784+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:01.213784+0000', 'last_peered': '2024-09-18T11:46:01.213784+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:01.213784+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3933, 'log_dups_size': 0, 'ondisk_log_size': 3933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32183542, 'num_objects': 3929, 'num_object_clones': 0, 'num_object_copies': 15716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3931, 'num_write_kb': 31437, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3860", 'reported_seq': 3915, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:59.307861+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:45:59.307861+0000', 'last_peered': '2024-09-18T11:45:59.307861+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:45:59.307861+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3860, 'log_dups_size': 0, 'ondisk_log_size': 3860, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31593718, 'num_objects': 3857, 'num_object_clones': 0, 'num_object_copies': 15428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3858, 'num_write_kb': 30861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1775}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3889", 'reported_seq': 3932, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:59.570789+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:45:59.570789+0000', 'last_peered': '2024-09-18T11:45:59.570789+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:45:59.570789+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3889, 'log_dups_size': 0, 'ondisk_log_size': 3889, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31858688, 'num_objects': 3889, 'num_object_clones': 0, 'num_object_copies': 15556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3889, 'num_write_kb': 31112, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1735}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3978", 'reported_seq': 4040, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.205341+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:01.205341+0000', 'last_peered': '2024-09-18T11:46:01.205341+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:01.205341+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3978, 'log_dups_size': 0, 'ondisk_log_size': 3978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32535798, 'num_objects': 3972, 'num_object_clones': 0, 'num_object_copies': 15888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3976, 'num_write_kb': 31788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1838}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:02.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3845", 'reported_seq': 3908, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.161963+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:01.161963+0000', 'last_peered': '2024-09-18T11:46:01.161963+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:01.161963+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3845, 'log_dups_size': 0, 'ondisk_log_size': 3845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31454454, 'num_objects': 3840, 'num_object_clones': 0, 'num_object_copies': 15360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3840, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3843, 'num_write_kb': 30732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1806}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3984", 'reported_seq': 4042, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:59.523924+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:45:59.523924+0000', 'last_peered': '2024-09-18T11:45:59.523924+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:45:59.523924+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3984, 'log_dups_size': 0, 'ondisk_log_size': 3984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32565740, 'num_objects': 3976, 'num_object_clones': 0, 'num_object_copies': 15904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3981, 'num_write_kb': 31825, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1841}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3817", 'reported_seq': 3901, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.150638+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:01.150638+0000', 'last_peered': '2024-09-18T11:46:01.150638+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:01.150638+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3817, 'log_dups_size': 0, 'ondisk_log_size': 3817, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31249654, 'num_objects': 3815, 'num_object_clones': 0, 'num_object_copies': 15260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3815, 'num_write_kb': 30518, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1840}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3894", 'reported_seq': 3950, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:59.487383+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:45:59.487383+0000', 'last_peered': '2024-09-18T11:45:59.487383+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:45:59.487383+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3894, 'log_dups_size': 0, 'ondisk_log_size': 3894, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31854449, 'num_objects': 3889, 'num_object_clones': 0, 'num_object_copies': 15556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3891, 'num_write_kb': 31116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1826}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3966", 'reported_seq': 4003, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:59.293938+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:45:59.293938+0000', 'last_peered': '2024-09-18T11:45:59.293938+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:45:59.293938+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3966, 'log_dups_size': 0, 'ondisk_log_size': 3966, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32479867, 'num_objects': 3965, 'num_object_clones': 0, 'num_object_copies': 15860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3965, 'num_write_kb': 31719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1856}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3935", 'reported_seq': 4011, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.092757+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:01.092757+0000', 'last_peered': '2024-09-18T11:46:01.092757+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:01.092757+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3935, 'log_dups_size': 0, 'ondisk_log_size': 3935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32225915, 'num_objects': 3934, 'num_object_clones': 0, 'num_object_copies': 15736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3934, 'num_write_kb': 31471, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1880}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3938", 'reported_seq': 3985, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:45:59.531535+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:45:59.531535+0000', 'last_peered': '2024-09-18T11:45:59.531535+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:45:59.531535+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3938, 'log_dups_size': 0, 'ondisk_log_size': 3938, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32224502, 'num_objects': 3934, 'num_object_clones': 0, 'num_object_copies': 15736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3936, 'num_write_kb': 31484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1776}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3917", 'reported_seq': 6172, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.188302+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:01.188302+0000', 'last_peered': '2024-09-18T11:46:01.188302+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:01.188302+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3917, 'log_dups_size': 0, 'ondisk_log_size': 3917, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32060662, 'num_objects': 3914, 'num_object_clones': 0, 'num_object_copies': 15656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3914, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3915, 'num_write_kb': 31317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1089, 'num_bytes_recovered': 8904704, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3868", 'reported_seq': 3928, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.208164+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:01.208164+0000', 'last_peered': '2024-09-18T11:46:01.208164+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:01.208164+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3868, 'log_dups_size': 0, 'ondisk_log_size': 3868, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31642870, 'num_objects': 3863, 'num_object_clones': 0, 'num_object_copies': 15452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3866, 'num_write_kb': 30916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1850}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:02.356 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:03.023 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:03.023 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32340/250436 objects degraded (12.913%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":161,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62609,"data_bytes":512853364,"bytes_used":2129260544,"bytes_avail":384417796096,"bytes_total":386547056640,"degraded_objects":32340,"degraded_total":250436,"degraded_ratio":0.12913478892810937,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":151493,"recovering_keys_per_sec":0,"num_objects_recovered":222,"num_bytes_recovered":1818624,"num_keys_recovered":0,"write_bytes_sec":3179324,"read_op_per_sec":0,"write_op_per_sec":388},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:03.047 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:03.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:03.231 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 106 20 63501 63481 4.67758 2.57812 0.105319 0.0333869 2024-09-18T11:46:03.583 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:03.583 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:03.617 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-18T11:46:01.854883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":512853364,"num_objects":62609,"num_object_clones":0,"num_object_copies":250436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62636,"num_write_kb":500970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62663,"ondisk_log_size":62663,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2079356,"kb_used_data":867736,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375408004,"statfs":{"total":386547056640,"available":384417796096,"internally_reserved":0,"allocated":888561664,"data_stored":887925752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,6,16,10,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38166528,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4659,"num_write_kb":37272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004604"},"pg_stats":[{"pgid":"1.f","version":"31'3975","reported_seq":4015,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.559628+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:45:59.559628+0000","last_peered":"2024-09-18T11:45:59.559628+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:45:59.559628+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3975,"log_dups_size":0,"ondisk_log_size":3975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32563200,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":31800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3941","reported_seq":4004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.244318+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:01.244318+0000","last_peered":"2024-09-18T11:46:01.244318+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:01.244318+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32249078,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3939,"num_write_kb":31508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3923","reported_seq":4005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.091697+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:01.091697+0000","last_peered":"2024-09-18T11:46:01.091697+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:01.091697+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32111227,"num_objects":3920,"num_object_clones":0,"num_object_copies":15680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3922,"num_write_kb":31366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3933","reported_seq":3995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.213784+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:01.213784+0000","last_peered":"2024-09-18T11:46:01.213784+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:01.213784+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183542,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3931,"num_write_kb":31437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3860","reported_seq":3915,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.307861+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:45:59.307861+0000","last_peered":"2024-09-18T11:45:59.307861+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:45:59.307861+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31593718,"num_objects":3857,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3858,"num_write_kb":30861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3889","reported_seq":3932,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.570789+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:45:59.570789+0000","last_peered":"2024-09-18T11:45:59.570789+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:45:59.570789+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3889,"log_dups_size":0,"ondisk_log_size":3889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31858688,"num_objects":3889,"num_object_clones":0,"num_object_copies":15556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3889,"num_write_kb":31112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3978","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.205341+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:01.205341+0000","last_peered":"2024-09-18T11:46:01.205341+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:01.205341+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535798,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3976,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3845","reported_seq":3908,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.161963+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:01.161963+0000","last_peered":"2024-09-18T11:46:01.161963+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:01.161963+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31454454,"num_objects":3840,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3984","reported_seq":4042,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.523924+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:45:59.523924+0000","last_peered":"2024-09-18T11:45:59.523924+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:45:59.523924+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32565740,"num_objects":3976,"num_object_clones":0,"num_object_copies":15904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":31825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3817","reported_seq":3901,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.150638+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:01.150638+0000","last_peered":"2024-09-18T11:46:01.150638+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:01.150638+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3817,"log_dups_size":0,"ondisk_log_size":3817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31249654,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":30518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1840}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3894","reported_seq":3950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.487383+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:45:59.487383+0000","last_peered":"2024-09-18T11:45:59.487383+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:45:59.487383+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31854449,"num_objects":3889,"num_object_clones":0,"num_object_copies":15556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3891,"num_write_kb":31116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3966","reported_seq":4003,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.293938+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:45:59.293938+0000","last_peered":"2024-09-18T11:45:59.293938+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:45:59.293938+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32479867,"num_objects":3965,"num_object_clones":0,"num_object_copies":15860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3965,"num_write_kb":31719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1856}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3935","reported_seq":4011,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.092757+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:01.092757+0000","last_peered":"2024-09-18T11:46:01.092757+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:01.092757+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3935,"log_dups_size":0,"ondisk_log_size":3935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32225915,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3934,"num_write_kb":31471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3938","reported_seq":3985,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:45:59.531535+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:45:59.531535+0000","last_peered":"2024-09-18T11:45:59.531535+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:45:59.531535+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3938,"log_dups_size":0,"ondisk_log_size":3938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32224502,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3917","reported_seq":6172,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.188302+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:01.188302+0000","last_peered":"2024-09-18T11:46:01.188302+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:01.188302+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3917,"log_dups_size":0,"ondisk_log_size":3917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32060662,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3915,"num_write_kb":31317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1089,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3868","reported_seq":3928,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.208164+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:01.208164+0000","last_peered":"2024-09-18T11:46:01.208164+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:01.208164+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3868,"log_dups_size":0,"ondisk_log_size":3868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642870,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":30916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1850}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":512853364,"num_objects":62609,"num_object_clones":0,"num_object_copies":250436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62636,"num_write_kb":500970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":892039168,"data_stored":892039168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62663,"ondisk_log_size":62663,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558272,"kb_used_data":247928,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93813568,"statfs":{"total":96636764160,"available":96065093632,"internally_reserved":0,"allocated":253878272,"data_stored":253719294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,7,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":0.59399999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":7.5069999999999997},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.57499999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":4.9320000000000004},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":3.9350000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551816,"kb_used_data":250300,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93820024,"statfs":{"total":96636764160,"available":96071704576,"internally_reserved":0,"allocated":256307200,"data_stored":256148222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.462},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":0.63400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":6.6029999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420320,"kb_used_data":118360,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93951520,"statfs":{"total":96636764160,"available":96206356480,"internally_reserved":0,"allocated":121200640,"data_stored":121041662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":2.9660000000000002},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":0.318}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":6.1950000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548948,"kb_used_data":251148,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93822892,"statfs":{"total":96636764160,"available":96074641408,"internally_reserved":0,"allocated":257175552,"data_stored":257016574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.248},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":1.994}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.378}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.016},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257445888,"data_stored":257445888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122425344,"data_stored":122425344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":257355776,"data_stored":257355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":254812160,"data_stored":254812160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:04.231 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 107 20 63858 63838 4.65994 2.78906 0.0378118 0.0335157 2024-09-18T11:46:05.231 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 108 20 64231 64211 4.64377 2.91406 0.0562288 0.0336353 2024-09-18T11:46:06.231 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 109 20 64597 64577 4.6274 2.85938 0.0678572 0.0337532 2024-09-18T11:46:06.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:07.154 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:07.155 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:07.183 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-18T11:46:05.856868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":519136628,"num_objects":63376,"num_object_clones":0,"num_object_copies":253504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63403,"num_write_kb":507106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63430,"ondisk_log_size":63430,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2085532,"kb_used_data":882232,"kb_used_omap":14,"kb_used_meta":1203249,"kb_avail":375401828,"statfs":{"total":386547056640,"available":384411471872,"internally_reserved":0,"allocated":903405568,"data_stored":902769656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232127003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,3,7,12,14,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36061184,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4402,"num_write_kb":35216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004929"},"pg_stats":[{"pgid":"1.f","version":"31'4077","reported_seq":4118,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.470432+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:04.470432+0000","last_peered":"2024-09-18T11:46:04.470432+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:04.470432+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33398784,"num_objects":4077,"num_object_clones":0,"num_object_copies":16308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":32616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3941","reported_seq":4004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.244318+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:01.244318+0000","last_peered":"2024-09-18T11:46:01.244318+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:01.244318+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32249078,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3939,"num_write_kb":31508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3923","reported_seq":4005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.091697+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:01.091697+0000","last_peered":"2024-09-18T11:46:01.091697+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:01.091697+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32111227,"num_objects":3920,"num_object_clones":0,"num_object_copies":15680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3922,"num_write_kb":31366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3933","reported_seq":3995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.213784+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:01.213784+0000","last_peered":"2024-09-18T11:46:01.213784+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:01.213784+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183542,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3931,"num_write_kb":31437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3976","reported_seq":4031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.517452+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:04.517452+0000","last_peered":"2024-09-18T11:46:04.517452+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:04.517452+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3976,"log_dups_size":0,"ondisk_log_size":3976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32543990,"num_objects":3973,"num_object_clones":0,"num_object_copies":15892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":31789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3989","reported_seq":4032,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.447552+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:04.447552+0000","last_peered":"2024-09-18T11:46:04.447552+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:04.447552+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32677888,"num_objects":3989,"num_object_clones":0,"num_object_copies":15956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":31912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3978","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.205341+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:01.205341+0000","last_peered":"2024-09-18T11:46:01.205341+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:01.205341+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535798,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3976,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3845","reported_seq":3908,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.161963+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:01.161963+0000","last_peered":"2024-09-18T11:46:01.161963+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:01.161963+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31454454,"num_objects":3840,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4099","reported_seq":4157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.569997+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:04.569997+0000","last_peered":"2024-09-18T11:46:04.569997+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:04.569997+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33507820,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4096,"num_write_kb":32745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3817","reported_seq":3901,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.150638+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:01.150638+0000","last_peered":"2024-09-18T11:46:01.150638+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:01.150638+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3817,"log_dups_size":0,"ondisk_log_size":3817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31249654,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":30518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1840}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4000","reported_seq":4057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.526459+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:04.526459+0000","last_peered":"2024-09-18T11:46:04.526459+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:04.526459+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4000,"log_dups_size":0,"ondisk_log_size":4000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32722801,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3997,"num_write_kb":31964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4087","reported_seq":4121,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.570400+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:04.570400+0000","last_peered":"2024-09-18T11:46:04.570400+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:04.570400+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33471099,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":32687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3935","reported_seq":4011,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.092757+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:01.092757+0000","last_peered":"2024-09-18T11:46:01.092757+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:01.092757+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3935,"log_dups_size":0,"ondisk_log_size":3935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32225915,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3934,"num_write_kb":31471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4045","reported_seq":4093,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.509040+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:04.509040+0000","last_peered":"2024-09-18T11:46:04.509040+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:04.509040+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4045,"log_dups_size":0,"ondisk_log_size":4045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33101046,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":32340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1883}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3917","reported_seq":6172,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.188302+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:01.188302+0000","last_peered":"2024-09-18T11:46:01.188302+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:01.188302+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3917,"log_dups_size":0,"ondisk_log_size":3917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32060662,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3915,"num_write_kb":31317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1089,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3868","reported_seq":3928,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.208164+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:01.208164+0000","last_peered":"2024-09-18T11:46:01.208164+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:01.208164+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3868,"log_dups_size":0,"ondisk_log_size":3868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642870,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":30916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1850}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":519136628,"num_objects":63376,"num_object_clones":0,"num_object_copies":253504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63403,"num_write_kb":507106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":906911744,"data_stored":906911744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63430,"ondisk_log_size":63430,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552960,"kb_used_data":255096,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93818880,"statfs":{"total":96636764160,"available":96070533120,"internally_reserved":0,"allocated":261218304,"data_stored":261059326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":1.595},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":1.629}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":7.6980000000000004},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.6479999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":12.781000000000001},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":15.292}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551816,"kb_used_data":250300,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93820024,"statfs":{"total":96636764160,"available":96071704576,"internally_reserved":0,"allocated":256307200,"data_stored":256148222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.462},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":0.63400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":6.6029999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431808,"kb_used_data":125688,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93940032,"statfs":{"total":96636764160,"available":96194592768,"internally_reserved":0,"allocated":128704512,"data_stored":128545534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":2.9239999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":1.9359999999999999},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":7.0629999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":2.96},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":5.5819999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548948,"kb_used_data":251148,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93822892,"statfs":{"total":96636764160,"available":96074641408,"internally_reserved":0,"allocated":257175552,"data_stored":257016574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.248},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":1.994}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.378}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.016},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257445888,"data_stored":257445888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130002944,"data_stored":130002944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":257355776,"data_stored":257355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262107136,"data_stored":262107136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:07.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:07.231 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 110 20 64934 64914 4.60927 2.63281 0.0609102 0.0338825 2024-09-18T11:46:07.741 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:07.742 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:07.767 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-18T11:46:05.856868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":519136628,"num_objects":63376,"num_object_clones":0,"num_object_copies":253504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63403,"num_write_kb":507106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63430,"ondisk_log_size":63430,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2085532,"kb_used_data":882232,"kb_used_omap":14,"kb_used_meta":1203249,"kb_avail":375401828,"statfs":{"total":386547056640,"available":384411471872,"internally_reserved":0,"allocated":903405568,"data_stored":902769656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232127003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,3,7,12,14,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36061184,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4402,"num_write_kb":35216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004929"},"pg_stats":[{"pgid":"1.f","version":"31'4077","reported_seq":4118,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.470432+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:04.470432+0000","last_peered":"2024-09-18T11:46:04.470432+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:04.470432+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33398784,"num_objects":4077,"num_object_clones":0,"num_object_copies":16308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":32616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3941","reported_seq":4004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.244318+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:01.244318+0000","last_peered":"2024-09-18T11:46:01.244318+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:01.244318+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32249078,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3939,"num_write_kb":31508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3923","reported_seq":4005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.091697+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:01.091697+0000","last_peered":"2024-09-18T11:46:01.091697+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:01.091697+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32111227,"num_objects":3920,"num_object_clones":0,"num_object_copies":15680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3922,"num_write_kb":31366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3933","reported_seq":3995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.213784+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:01.213784+0000","last_peered":"2024-09-18T11:46:01.213784+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:01.213784+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183542,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3931,"num_write_kb":31437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3976","reported_seq":4031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.517452+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:04.517452+0000","last_peered":"2024-09-18T11:46:04.517452+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:04.517452+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3976,"log_dups_size":0,"ondisk_log_size":3976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32543990,"num_objects":3973,"num_object_clones":0,"num_object_copies":15892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":31789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3989","reported_seq":4032,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.447552+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:04.447552+0000","last_peered":"2024-09-18T11:46:04.447552+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:04.447552+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32677888,"num_objects":3989,"num_object_clones":0,"num_object_copies":15956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":31912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3978","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.205341+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:01.205341+0000","last_peered":"2024-09-18T11:46:01.205341+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:01.205341+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535798,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3976,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3845","reported_seq":3908,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.161963+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:01.161963+0000","last_peered":"2024-09-18T11:46:01.161963+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:01.161963+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31454454,"num_objects":3840,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4099","reported_seq":4157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.569997+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:04.569997+0000","last_peered":"2024-09-18T11:46:04.569997+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:04.569997+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33507820,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4096,"num_write_kb":32745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3817","reported_seq":3901,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.150638+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:01.150638+0000","last_peered":"2024-09-18T11:46:01.150638+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:01.150638+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3817,"log_dups_size":0,"ondisk_log_size":3817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31249654,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":30518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1840}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4000","reported_seq":4057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.526459+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:04.526459+0000","last_peered":"2024-09-18T11:46:04.526459+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:04.526459+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4000,"log_dups_size":0,"ondisk_log_size":4000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32722801,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3997,"num_write_kb":31964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4087","reported_seq":4121,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.570400+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:04.570400+0000","last_peered":"2024-09-18T11:46:04.570400+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:04.570400+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33471099,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":32687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3935","reported_seq":4011,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.092757+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:01.092757+0000","last_peered":"2024-09-18T11:46:01.092757+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:01.092757+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3935,"log_dups_size":0,"ondisk_log_size":3935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32225915,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3934,"num_write_kb":31471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4045","reported_seq":4093,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.509040+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:04.509040+0000","last_peered":"2024-09-18T11:46:04.509040+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:04.509040+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4045,"log_dups_size":0,"ondisk_log_size":4045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33101046,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":32340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1883}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3917","reported_seq":6172,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.188302+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:01.188302+0000","last_peered":"2024-09-18T11:46:01.188302+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:01.188302+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3917,"log_dups_size":0,"ondisk_log_size":3917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32060662,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3915,"num_write_kb":31317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1089,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3868","reported_seq":3928,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:01.208164+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:01.208164+0000","last_peered":"2024-09-18T11:46:01.208164+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:01.208164+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3868,"log_dups_size":0,"ondisk_log_size":3868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642870,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":30916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1850}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":519136628,"num_objects":63376,"num_object_clones":0,"num_object_copies":253504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63403,"num_write_kb":507106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":8904704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":906911744,"data_stored":906911744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63430,"ondisk_log_size":63430,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552960,"kb_used_data":255096,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93818880,"statfs":{"total":96636764160,"available":96070533120,"internally_reserved":0,"allocated":261218304,"data_stored":261059326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":1.595},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":1.629}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":7.6980000000000004},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.6479999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":12.781000000000001},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":15.292}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551816,"kb_used_data":250300,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93820024,"statfs":{"total":96636764160,"available":96071704576,"internally_reserved":0,"allocated":256307200,"data_stored":256148222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.462},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":0.63400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":6.6029999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431808,"kb_used_data":125688,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93940032,"statfs":{"total":96636764160,"available":96194592768,"internally_reserved":0,"allocated":128704512,"data_stored":128545534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":2.9239999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":1.9359999999999999},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":7.0629999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":2.96},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":5.5819999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548948,"kb_used_data":251148,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93822892,"statfs":{"total":96636764160,"available":96074641408,"internally_reserved":0,"allocated":257175552,"data_stored":257016574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.248},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":1.994}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.378}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":6.016},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257445888,"data_stored":257445888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130002944,"data_stored":130002944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":257355776,"data_stored":257355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262107136,"data_stored":262107136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:07.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:07.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4077", 'reported_seq': 4118, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:04.470432+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:04.470432+0000', 'last_peered': '2024-09-18T11:46:04.470432+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:04.470432+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4077, 'log_dups_size': 0, 'ondisk_log_size': 4077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33398784, 'num_objects': 4077, 'num_object_clones': 0, 'num_object_copies': 16308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4077, 'num_write_kb': 32616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1947}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:07.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:07.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3941", 'reported_seq': 4004, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.244318+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:01.244318+0000', 'last_peered': '2024-09-18T11:46:01.244318+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:01.244318+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3941, 'log_dups_size': 0, 'ondisk_log_size': 3941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32249078, 'num_objects': 3937, 'num_object_clones': 0, 'num_object_copies': 15748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3939, 'num_write_kb': 31508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1835}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:07.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:07.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3923", 'reported_seq': 4005, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.091697+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:01.091697+0000', 'last_peered': '2024-09-18T11:46:01.091697+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:01.091697+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3923, 'log_dups_size': 0, 'ondisk_log_size': 3923, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32111227, 'num_objects': 3920, 'num_object_clones': 0, 'num_object_copies': 15680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3920, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3922, 'num_write_kb': 31366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1866}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:07.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:07.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3933", 'reported_seq': 3995, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.213784+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:01.213784+0000', 'last_peered': '2024-09-18T11:46:01.213784+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:01.213784+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3933, 'log_dups_size': 0, 'ondisk_log_size': 3933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32183542, 'num_objects': 3929, 'num_object_clones': 0, 'num_object_copies': 15716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3931, 'num_write_kb': 31437, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:07.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:07.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3976", 'reported_seq': 4031, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:04.517452+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:04.517452+0000', 'last_peered': '2024-09-18T11:46:04.517452+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:04.517452+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3976, 'log_dups_size': 0, 'ondisk_log_size': 3976, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32543990, 'num_objects': 3973, 'num_object_clones': 0, 'num_object_copies': 15892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3974, 'num_write_kb': 31789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1891}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:07.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:07.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3989", 'reported_seq': 4032, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:04.447552+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:04.447552+0000', 'last_peered': '2024-09-18T11:46:04.447552+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:04.447552+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3989, 'log_dups_size': 0, 'ondisk_log_size': 3989, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32677888, 'num_objects': 3989, 'num_object_clones': 0, 'num_object_copies': 15956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3989, 'num_write_kb': 31912, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1835}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:07.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:07.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3978", 'reported_seq': 4040, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.205341+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:01.205341+0000', 'last_peered': '2024-09-18T11:46:01.205341+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:01.205341+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3978, 'log_dups_size': 0, 'ondisk_log_size': 3978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32535798, 'num_objects': 3972, 'num_object_clones': 0, 'num_object_copies': 15888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3976, 'num_write_kb': 31788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1838}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:07.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:07.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3845", 'reported_seq': 3908, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.161963+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:01.161963+0000', 'last_peered': '2024-09-18T11:46:01.161963+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:01.161963+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3845, 'log_dups_size': 0, 'ondisk_log_size': 3845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31454454, 'num_objects': 3840, 'num_object_clones': 0, 'num_object_copies': 15360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3840, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3843, 'num_write_kb': 30732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1806}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:07.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:07.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4099", 'reported_seq': 4157, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:04.569997+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:04.569997+0000', 'last_peered': '2024-09-18T11:46:04.569997+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:04.569997+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4099, 'log_dups_size': 0, 'ondisk_log_size': 4099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33507820, 'num_objects': 4091, 'num_object_clones': 0, 'num_object_copies': 16364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4096, 'num_write_kb': 32745, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1956}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:07.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:07.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3817", 'reported_seq': 3901, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.150638+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:01.150638+0000', 'last_peered': '2024-09-18T11:46:01.150638+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:01.150638+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3817, 'log_dups_size': 0, 'ondisk_log_size': 3817, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31249654, 'num_objects': 3815, 'num_object_clones': 0, 'num_object_copies': 15260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3815, 'num_write_kb': 30518, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1840}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:07.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:07.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4000", 'reported_seq': 4057, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:04.526459+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:04.526459+0000', 'last_peered': '2024-09-18T11:46:04.526459+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:04.526459+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4000, 'log_dups_size': 0, 'ondisk_log_size': 4000, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32722801, 'num_objects': 3995, 'num_object_clones': 0, 'num_object_copies': 15980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3997, 'num_write_kb': 31964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1932}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:07.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:07.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4087", 'reported_seq': 4121, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:04.570400+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:04.570400+0000', 'last_peered': '2024-09-18T11:46:04.570400+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:04.570400+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4087, 'log_dups_size': 0, 'ondisk_log_size': 4087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33471099, 'num_objects': 4086, 'num_object_clones': 0, 'num_object_copies': 16344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4086, 'num_write_kb': 32687, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:07.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:07.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3935", 'reported_seq': 4011, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.092757+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:01.092757+0000', 'last_peered': '2024-09-18T11:46:01.092757+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:01.092757+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3935, 'log_dups_size': 0, 'ondisk_log_size': 3935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32225915, 'num_objects': 3934, 'num_object_clones': 0, 'num_object_copies': 15736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3934, 'num_write_kb': 31471, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1880}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:07.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:07.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4045", 'reported_seq': 4093, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:04.509040+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:04.509040+0000', 'last_peered': '2024-09-18T11:46:04.509040+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:04.509040+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4045, 'log_dups_size': 0, 'ondisk_log_size': 4045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33101046, 'num_objects': 4041, 'num_object_clones': 0, 'num_object_copies': 16164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4043, 'num_write_kb': 32340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1883}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:07.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:07.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3917", 'reported_seq': 6172, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.188302+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:01.188302+0000', 'last_peered': '2024-09-18T11:46:01.188302+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:01.188302+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3917, 'log_dups_size': 0, 'ondisk_log_size': 3917, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32060662, 'num_objects': 3914, 'num_object_clones': 0, 'num_object_copies': 15656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3914, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3915, 'num_write_kb': 31317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1089, 'num_bytes_recovered': 8904704, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:07.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:07.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3868", 'reported_seq': 3928, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:01.208164+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:01.208164+0000', 'last_peered': '2024-09-18T11:46:01.208164+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:01.208164+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 3868, 'log_dups_size': 0, 'ondisk_log_size': 3868, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31642870, 'num_objects': 3863, 'num_object_clones': 0, 'num_object_copies': 15452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3866, 'num_write_kb': 30916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1850}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:07.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:08.231 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 111 20 65254 65234 4.59027 2.5 0.0581169 0.0340254 2024-09-18T11:46:08.413 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:08.413 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32265/257440 objects degraded (12.533%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":166,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":64360,"data_bytes":527197556,"bytes_used":2158546944,"bytes_avail":384388509696,"bytes_total":386547056640,"degraded_objects":32265,"degraded_total":257440,"degraded_ratio":0.12533017402113114,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":153536,"recovering_keys_per_sec":0,"num_objects_recovered":225,"num_bytes_recovered":1843200,"num_keys_recovered":0,"write_bytes_sec":3129420,"read_op_per_sec":0,"write_op_per_sec":382},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:08.437 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:08.438 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:09.016 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:09.016 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:09.049 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-18T11:46:07.857743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":527197556,"num_objects":64360,"num_object_clones":0,"num_object_copies":257440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64387,"num_write_kb":514978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":9519104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64414,"ondisk_log_size":64414,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2107956,"kb_used_data":896340,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375379404,"statfs":{"total":386547056640,"available":384388509696,"internally_reserved":0,"allocated":917852160,"data_stored":917216248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,4,6,8,11,16],"upper_bound":128},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37568512,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":36688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1843200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004940"},"pg_stats":[{"pgid":"1.f","version":"31'4077","reported_seq":4118,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.470432+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:04.470432+0000","last_peered":"2024-09-18T11:46:04.470432+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:04.470432+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33398784,"num_objects":4077,"num_object_clones":0,"num_object_copies":16308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":32616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4060","reported_seq":4123,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:06.210566+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:06.210566+0000","last_peered":"2024-09-18T11:46:06.210566+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:06.210566+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223926,"num_objects":4056,"num_object_clones":0,"num_object_copies":16224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4058,"num_write_kb":32460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1954}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4025","reported_seq":4107,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:06.160618+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:06.160618+0000","last_peered":"2024-09-18T11:46:06.160618+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:06.160618+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4025,"log_dups_size":0,"ondisk_log_size":4025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32946811,"num_objects":4022,"num_object_clones":0,"num_object_copies":16088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4024,"num_write_kb":32182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1968}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4056","reported_seq":4117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:06.203843+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:06.203843+0000","last_peered":"2024-09-18T11:46:06.203843+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:06.203843+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4056,"log_dups_size":0,"ondisk_log_size":4056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33191158,"num_objects":4052,"num_object_clones":0,"num_object_copies":16208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4054,"num_write_kb":32421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3976","reported_seq":4031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.517452+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:04.517452+0000","last_peered":"2024-09-18T11:46:04.517452+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:04.517452+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3976,"log_dups_size":0,"ondisk_log_size":3976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32543990,"num_objects":3973,"num_object_clones":0,"num_object_copies":15892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":31789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3989","reported_seq":4032,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.447552+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:04.447552+0000","last_peered":"2024-09-18T11:46:04.447552+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:04.447552+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32677888,"num_objects":3989,"num_object_clones":0,"num_object_copies":15956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":31912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4086","reported_seq":4151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:06.180048+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:06.180048+0000","last_peered":"2024-09-18T11:46:06.180048+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:06.180048+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4086,"log_dups_size":0,"ondisk_log_size":4086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33420534,"num_objects":4080,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4084,"num_write_kb":32652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1946}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3955","reported_seq":4018,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:06.180532+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:06.180532+0000","last_peered":"2024-09-18T11:46:06.180532+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:06.180532+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3955,"log_dups_size":0,"ondisk_log_size":3955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32355574,"num_objects":3950,"num_object_clones":0,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3953,"num_write_kb":31612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1916}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4099","reported_seq":4157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.569997+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:04.569997+0000","last_peered":"2024-09-18T11:46:04.569997+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:04.569997+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33507820,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4096,"num_write_kb":32745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3923","reported_seq":4005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:06.145519+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:06.145519+0000","last_peered":"2024-09-18T11:46:06.145519+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:06.145519+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32118006,"num_objects":3921,"num_object_clones":0,"num_object_copies":15684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3921,"num_write_kb":31366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1946}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4000","reported_seq":4057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.526459+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:04.526459+0000","last_peered":"2024-09-18T11:46:04.526459+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:04.526459+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4000,"log_dups_size":0,"ondisk_log_size":4000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32722801,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3997,"num_write_kb":31964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4087","reported_seq":4121,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.570400+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:04.570400+0000","last_peered":"2024-09-18T11:46:04.570400+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:04.570400+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33471099,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":32687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4052","reported_seq":4126,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:06.204551+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:06.204551+0000","last_peered":"2024-09-18T11:46:06.204551+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:06.204551+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33184379,"num_objects":4051,"num_object_clones":0,"num_object_copies":16204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4051,"num_write_kb":32407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1997}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4045","reported_seq":4093,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:04.509040+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:04.509040+0000","last_peered":"2024-09-18T11:46:04.509040+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:04.509040+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4045,"log_dups_size":0,"ondisk_log_size":4045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33101046,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":32340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1883}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4015","reported_seq":6422,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:06.195288+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:06.195288+0000","last_peered":"2024-09-18T11:46:06.195288+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:06.195288+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32863478,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4013,"num_write_kb":32101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":9519104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3083}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3969","reported_seq":4031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:06.223797+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:06.223797+0000","last_peered":"2024-09-18T11:46:06.223797+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:06.223797+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":3969,"log_dups_size":0,"ondisk_log_size":3969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32470262,"num_objects":3964,"num_object_clones":0,"num_object_copies":15856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3967,"num_write_kb":31724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1951}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":527197556,"num_objects":64360,"num_object_clones":0,"num_object_copies":257440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64387,"num_write_kb":514978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":9519104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":921325568,"data_stored":921325568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64414,"ondisk_log_size":64414,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552960,"kb_used_data":255096,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93818880,"statfs":{"total":96636764160,"available":96070533120,"internally_reserved":0,"allocated":261218304,"data_stored":261059326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":1.595},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":1.629}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":7.6980000000000004},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.6479999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":12.781000000000001},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":15.292}]}]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563028,"kb_used_data":257356,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93808812,"statfs":{"total":96636764160,"available":96060223488,"internally_reserved":0,"allocated":263532544,"data_stored":263373566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":6.6779999999999999},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.657}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":2.359},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":6.7000000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.273,"5min":11.273,"15min":11.273},"last":6.6210000000000004},{"interface":"front","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1639999999999997,"5min":9.1639999999999997,"15min":9.1639999999999997},"last":5.173}]}]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431808,"kb_used_data":125688,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93940032,"statfs":{"total":96636764160,"available":96194592768,"internally_reserved":0,"allocated":128704512,"data_stored":128545534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":2.9239999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":1.9359999999999999},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":7.0629999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":2.96},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":5.5819999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560160,"kb_used_data":258200,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93811680,"statfs":{"total":96636764160,"available":96063160320,"internally_reserved":0,"allocated":264396800,"data_stored":264237822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":5.5030000000000001},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":5.3499999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":5.3810000000000002},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":2.3069999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.266,"5min":12.266,"15min":12.266},"last":5.5359999999999996},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":6.0289999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":264667136,"data_stored":264667136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130002944,"data_stored":130002944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264548352,"data_stored":264548352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262107136,"data_stored":262107136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:09.234 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 112 20 65582 65562 4.57216 2.5625 0.0480579 0.0341634 2024-09-18T11:46:10.231 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 113 20 65941 65921 4.55652 2.80469 0.0948941 0.0342761 2024-09-18T11:46:11.231 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 114 20 66307 66287 4.54164 2.85938 0.0608105 0.0343918 2024-09-18T11:46:12.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:12.231 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 115 20 66661 66641 4.52619 2.76562 0.0612617 0.034506 2024-09-18T11:46:12.624 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:12.625 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:12.645 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-18T11:46:11.859157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":541164916,"num_objects":66065,"num_object_clones":0,"num_object_copies":264260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66092,"num_write_kb":528618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66119,"ondisk_log_size":66119,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2154072,"kb_used_data":925812,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375333288,"statfs":{"total":386547056640,"available":384341286912,"internally_reserved":0,"allocated":948031488,"data_stored":947395576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,12,7,4,9,13,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36913152,"num_objects":4506,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4506,"num_write_kb":36048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1843200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005080"},"pg_stats":[{"pgid":"1.f","version":"31'4185","reported_seq":4225,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.606629+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:09.606629+0000","last_peered":"2024-09-18T11:46:09.606629+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:09.606629+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34283520,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4172","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.150553+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:11.150553+0000","last_peered":"2024-09-18T11:46:11.150553+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:11.150553+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34141430,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4170,"num_write_kb":33356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4114","reported_seq":4196,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.203833+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:11.203833+0000","last_peered":"2024-09-18T11:46:11.203833+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:11.203833+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33675899,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4113,"num_write_kb":32894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2057}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4177","reported_seq":4239,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.270412+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:11.270412+0000","last_peered":"2024-09-18T11:46:11.270412+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:11.270412+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4177,"log_dups_size":0,"ondisk_log_size":4177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34182390,"num_objects":4173,"num_object_clones":0,"num_object_copies":16692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":33389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4085","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.566959+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:09.566959+0000","last_peered":"2024-09-18T11:46:09.566959+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:09.566959+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33436918,"num_objects":4082,"num_object_clones":0,"num_object_copies":16328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4083,"num_write_kb":32661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4090","reported_seq":4133,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.399036+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:09.399036+0000","last_peered":"2024-09-18T11:46:09.399036+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:09.399036+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":4090,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4090,"num_write_kb":32720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4181","reported_seq":4245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.131419+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:11.131419+0000","last_peered":"2024-09-18T11:46:11.131419+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:11.131419+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34198774,"num_objects":4175,"num_object_clones":0,"num_object_copies":16700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4179,"num_write_kb":33412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4066","reported_seq":4129,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.101752+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:11.101752+0000","last_peered":"2024-09-18T11:46:11.101752+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:11.101752+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4066,"log_dups_size":0,"ondisk_log_size":4066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264886,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":32500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2027}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4217","reported_seq":4275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.580380+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:09.580380+0000","last_peered":"2024-09-18T11:46:09.580380+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:09.580380+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34474476,"num_objects":4209,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4214,"num_write_kb":33689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4030","reported_seq":4114,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.200520+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:11.200520+0000","last_peered":"2024-09-18T11:46:11.200520+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:11.200520+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32994550,"num_objects":4028,"num_object_clones":0,"num_object_copies":16112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4028,"num_write_kb":32222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2053}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4115","reported_seq":4172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.504509+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:09.504509+0000","last_peered":"2024-09-18T11:46:09.504509+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:09.504509+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4115,"log_dups_size":0,"ondisk_log_size":4115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33664881,"num_objects":4110,"num_object_clones":0,"num_object_copies":16440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4185","reported_seq":4222,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.581589+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:09.581589+0000","last_peered":"2024-09-18T11:46:09.581589+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:09.581589+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34273915,"num_objects":4184,"num_object_clones":0,"num_object_copies":16736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4184,"num_write_kb":33471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2075}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4159","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.167075+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:11.167075+0000","last_peered":"2024-09-18T11:46:11.167075+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:11.167075+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060923,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4158","reported_seq":4207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.452054+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:09.452054+0000","last_peered":"2024-09-18T11:46:09.452054+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:09.452054+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34026742,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4117","reported_seq":6680,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.203048+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:11.203048+0000","last_peered":"2024-09-18T11:46:11.203048+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:11.203048+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4117,"log_dups_size":0,"ondisk_log_size":4117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33699062,"num_objects":4114,"num_object_clones":0,"num_object_copies":16456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4115,"num_write_kb":32917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1242,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3263}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4068","reported_seq":4130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.214585+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:11.214585+0000","last_peered":"2024-09-18T11:46:11.214585+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:11.214585+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33281270,"num_objects":4063,"num_object_clones":0,"num_object_copies":16252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4066,"num_write_kb":32516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":541164916,"num_objects":66065,"num_object_clones":0,"num_object_copies":264260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66092,"num_write_kb":528618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":949776384,"data_stored":949776384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66119,"ondisk_log_size":66119,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564012,"kb_used_data":261988,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93807828,"statfs":{"total":96636764160,"available":96059215872,"internally_reserved":0,"allocated":268275712,"data_stored":268116734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,11,3,1,1,2,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":10.08},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":10.111000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":6.8849999999999998},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":11.904}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":2.23},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":4.3360000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574096,"kb_used_data":264260,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93797744,"statfs":{"total":96636764160,"available":96048889856,"internally_reserved":0,"allocated":270602240,"data_stored":270443262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.042},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":14.537000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":14.601000000000001},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":1.0669999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":0.21099999999999999},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":7.4429999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444672,"kb_used_data":134392,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93927168,"statfs":{"total":96636764160,"available":96181420032,"internally_reserved":0,"allocated":137617408,"data_stored":137458430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.34899999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":24.600000000000001},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":0.91300000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":3.847}]}]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571292,"kb_used_data":265172,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93800548,"statfs":{"total":96636764160,"available":96051761152,"internally_reserved":0,"allocated":271536128,"data_stored":271377150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":4.0780000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.117},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":1.125}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.26300000000000001},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":4.1399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271642624,"data_stored":271642624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137355264,"data_stored":137355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":271589376,"data_stored":271589376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":269189120,"data_stored":269189120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:12.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:13.214 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:13.215 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:13.231 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 116 20 66977 66957 4.50846 2.46875 0.0413938 0.0346425 2024-09-18T11:46:13.237 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-18T11:46:11.859157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":541164916,"num_objects":66065,"num_object_clones":0,"num_object_copies":264260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66092,"num_write_kb":528618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66119,"ondisk_log_size":66119,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2154072,"kb_used_data":925812,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375333288,"statfs":{"total":386547056640,"available":384341286912,"internally_reserved":0,"allocated":948031488,"data_stored":947395576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,12,7,4,9,13,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36913152,"num_objects":4506,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4506,"num_write_kb":36048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1843200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005080"},"pg_stats":[{"pgid":"1.f","version":"31'4185","reported_seq":4225,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.606629+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:09.606629+0000","last_peered":"2024-09-18T11:46:09.606629+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:09.606629+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34283520,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4172","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.150553+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:11.150553+0000","last_peered":"2024-09-18T11:46:11.150553+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:11.150553+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34141430,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4170,"num_write_kb":33356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4114","reported_seq":4196,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.203833+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:11.203833+0000","last_peered":"2024-09-18T11:46:11.203833+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:11.203833+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33675899,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4113,"num_write_kb":32894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2057}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4177","reported_seq":4239,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.270412+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:11.270412+0000","last_peered":"2024-09-18T11:46:11.270412+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:11.270412+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4177,"log_dups_size":0,"ondisk_log_size":4177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34182390,"num_objects":4173,"num_object_clones":0,"num_object_copies":16692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":33389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4085","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.566959+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:09.566959+0000","last_peered":"2024-09-18T11:46:09.566959+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:09.566959+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33436918,"num_objects":4082,"num_object_clones":0,"num_object_copies":16328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4083,"num_write_kb":32661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4090","reported_seq":4133,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.399036+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:09.399036+0000","last_peered":"2024-09-18T11:46:09.399036+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:09.399036+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":4090,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4090,"num_write_kb":32720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4181","reported_seq":4245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.131419+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:11.131419+0000","last_peered":"2024-09-18T11:46:11.131419+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:11.131419+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34198774,"num_objects":4175,"num_object_clones":0,"num_object_copies":16700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4179,"num_write_kb":33412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4066","reported_seq":4129,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.101752+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:11.101752+0000","last_peered":"2024-09-18T11:46:11.101752+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:11.101752+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4066,"log_dups_size":0,"ondisk_log_size":4066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264886,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":32500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2027}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4217","reported_seq":4275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.580380+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:09.580380+0000","last_peered":"2024-09-18T11:46:09.580380+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:09.580380+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34474476,"num_objects":4209,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4214,"num_write_kb":33689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4030","reported_seq":4114,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.200520+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:11.200520+0000","last_peered":"2024-09-18T11:46:11.200520+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:11.200520+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32994550,"num_objects":4028,"num_object_clones":0,"num_object_copies":16112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4028,"num_write_kb":32222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2053}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4115","reported_seq":4172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.504509+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:09.504509+0000","last_peered":"2024-09-18T11:46:09.504509+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:09.504509+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4115,"log_dups_size":0,"ondisk_log_size":4115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33664881,"num_objects":4110,"num_object_clones":0,"num_object_copies":16440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4185","reported_seq":4222,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.581589+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:09.581589+0000","last_peered":"2024-09-18T11:46:09.581589+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:09.581589+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34273915,"num_objects":4184,"num_object_clones":0,"num_object_copies":16736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4184,"num_write_kb":33471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2075}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4159","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.167075+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:11.167075+0000","last_peered":"2024-09-18T11:46:11.167075+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:11.167075+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060923,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4158","reported_seq":4207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.452054+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:09.452054+0000","last_peered":"2024-09-18T11:46:09.452054+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:09.452054+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34026742,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4117","reported_seq":6680,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.203048+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:11.203048+0000","last_peered":"2024-09-18T11:46:11.203048+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:11.203048+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4117,"log_dups_size":0,"ondisk_log_size":4117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33699062,"num_objects":4114,"num_object_clones":0,"num_object_copies":16456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4115,"num_write_kb":32917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1242,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3263}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4068","reported_seq":4130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.214585+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:11.214585+0000","last_peered":"2024-09-18T11:46:11.214585+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:11.214585+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33281270,"num_objects":4063,"num_object_clones":0,"num_object_copies":16252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4066,"num_write_kb":32516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":541164916,"num_objects":66065,"num_object_clones":0,"num_object_copies":264260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66092,"num_write_kb":528618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":949776384,"data_stored":949776384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66119,"ondisk_log_size":66119,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564012,"kb_used_data":261988,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93807828,"statfs":{"total":96636764160,"available":96059215872,"internally_reserved":0,"allocated":268275712,"data_stored":268116734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,11,3,1,1,2,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":10.08},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":10.111000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":6.8849999999999998},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":11.904}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":2.23},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":4.3360000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574096,"kb_used_data":264260,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93797744,"statfs":{"total":96636764160,"available":96048889856,"internally_reserved":0,"allocated":270602240,"data_stored":270443262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.042},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":14.537000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":14.601000000000001},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":1.0669999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":0.21099999999999999},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":7.4429999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444672,"kb_used_data":134392,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93927168,"statfs":{"total":96636764160,"available":96181420032,"internally_reserved":0,"allocated":137617408,"data_stored":137458430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.34899999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":24.600000000000001},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":0.91300000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":3.847}]}]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571292,"kb_used_data":265172,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93800548,"statfs":{"total":96636764160,"available":96051761152,"internally_reserved":0,"allocated":271536128,"data_stored":271377150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":4.0780000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.117},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":1.125}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.26300000000000001},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":4.1399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271642624,"data_stored":271642624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137355264,"data_stored":137355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":271589376,"data_stored":271589376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":269189120,"data_stored":269189120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:13.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:13.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4185", 'reported_seq': 4225, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:09.606629+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:09.606629+0000', 'last_peered': '2024-09-18T11:46:09.606629+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:09.606629+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4185, 'log_dups_size': 0, 'ondisk_log_size': 4185, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34283520, 'num_objects': 4185, 'num_object_clones': 0, 'num_object_copies': 16740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4185, 'num_write_kb': 33480, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2055}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:13.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:13.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4172", 'reported_seq': 4235, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:11.150553+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:11.150553+0000', 'last_peered': '2024-09-18T11:46:11.150553+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:11.150553+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4172, 'log_dups_size': 0, 'ondisk_log_size': 4172, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34141430, 'num_objects': 4168, 'num_object_clones': 0, 'num_object_copies': 16672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4170, 'num_write_kb': 33356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2066}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:13.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4114", 'reported_seq': 4196, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:11.203833+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:11.203833+0000', 'last_peered': '2024-09-18T11:46:11.203833+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:11.203833+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4114, 'log_dups_size': 0, 'ondisk_log_size': 4114, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33675899, 'num_objects': 4111, 'num_object_clones': 0, 'num_object_copies': 16444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4113, 'num_write_kb': 32894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2057}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4177", 'reported_seq': 4239, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:11.270412+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:11.270412+0000', 'last_peered': '2024-09-18T11:46:11.270412+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:11.270412+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4177, 'log_dups_size': 0, 'ondisk_log_size': 4177, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34182390, 'num_objects': 4173, 'num_object_clones': 0, 'num_object_copies': 16692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4175, 'num_write_kb': 33389, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4085", 'reported_seq': 4140, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:09.566959+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:09.566959+0000', 'last_peered': '2024-09-18T11:46:09.566959+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:09.566959+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4085, 'log_dups_size': 0, 'ondisk_log_size': 4085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33436918, 'num_objects': 4082, 'num_object_clones': 0, 'num_object_copies': 16328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4082, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4083, 'num_write_kb': 32661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2000}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4090", 'reported_seq': 4133, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:09.399036+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:09.399036+0000', 'last_peered': '2024-09-18T11:46:09.399036+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:09.399036+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4090, 'log_dups_size': 0, 'ondisk_log_size': 4090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33505280, 'num_objects': 4090, 'num_object_clones': 0, 'num_object_copies': 16360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4090, 'num_write_kb': 32720, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1936}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4181", 'reported_seq': 4245, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:11.131419+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:11.131419+0000', 'last_peered': '2024-09-18T11:46:11.131419+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:11.131419+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4181, 'log_dups_size': 0, 'ondisk_log_size': 4181, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34198774, 'num_objects': 4175, 'num_object_clones': 0, 'num_object_copies': 16700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4175, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4179, 'num_write_kb': 33412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2041}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:13.239 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4066", 'reported_seq': 4129, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:11.101752+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:11.101752+0000', 'last_peered': '2024-09-18T11:46:11.101752+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:11.101752+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4066, 'log_dups_size': 0, 'ondisk_log_size': 4066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33264886, 'num_objects': 4061, 'num_object_clones': 0, 'num_object_copies': 16244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4064, 'num_write_kb': 32500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2027}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:13.240 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:13.240 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4217", 'reported_seq': 4275, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:09.580380+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:09.580380+0000', 'last_peered': '2024-09-18T11:46:09.580380+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:09.580380+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4217, 'log_dups_size': 0, 'ondisk_log_size': 4217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34474476, 'num_objects': 4209, 'num_object_clones': 0, 'num_object_copies': 16836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4214, 'num_write_kb': 33689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2074}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:13.240 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:13.240 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4030", 'reported_seq': 4114, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:11.200520+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:11.200520+0000', 'last_peered': '2024-09-18T11:46:11.200520+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:11.200520+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4030, 'log_dups_size': 0, 'ondisk_log_size': 4030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32994550, 'num_objects': 4028, 'num_object_clones': 0, 'num_object_copies': 16112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4028, 'num_write_kb': 32222, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2053}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:13.240 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:13.240 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4115", 'reported_seq': 4172, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:09.504509+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:09.504509+0000', 'last_peered': '2024-09-18T11:46:09.504509+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:09.504509+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4115, 'log_dups_size': 0, 'ondisk_log_size': 4115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33664881, 'num_objects': 4110, 'num_object_clones': 0, 'num_object_copies': 16440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4112, 'num_write_kb': 32884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2047}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:13.240 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:13.240 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4185", 'reported_seq': 4222, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:09.581589+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:09.581589+0000', 'last_peered': '2024-09-18T11:46:09.581589+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:09.581589+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4185, 'log_dups_size': 0, 'ondisk_log_size': 4185, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34273915, 'num_objects': 4184, 'num_object_clones': 0, 'num_object_copies': 16736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4184, 'num_write_kb': 33471, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2075}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:13.240 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:13.240 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4159", 'reported_seq': 4235, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:11.167075+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:11.167075+0000', 'last_peered': '2024-09-18T11:46:11.167075+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:11.167075+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4159, 'log_dups_size': 0, 'ondisk_log_size': 4159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34060923, 'num_objects': 4158, 'num_object_clones': 0, 'num_object_copies': 16632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4158, 'num_write_kb': 33263, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:13.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:13.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4158", 'reported_seq': 4207, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:09.452054+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:09.452054+0000', 'last_peered': '2024-09-18T11:46:09.452054+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:09.452054+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4158, 'log_dups_size': 0, 'ondisk_log_size': 4158, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34026742, 'num_objects': 4154, 'num_object_clones': 0, 'num_object_copies': 16616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4156, 'num_write_kb': 33244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1996}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:13.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:13.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4117", 'reported_seq': 6680, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:11.203048+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:11.203048+0000', 'last_peered': '2024-09-18T11:46:11.203048+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:11.203048+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4117, 'log_dups_size': 0, 'ondisk_log_size': 4117, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33699062, 'num_objects': 4114, 'num_object_clones': 0, 'num_object_copies': 16456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 851, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4115, 'num_write_kb': 32917, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1242, 'num_bytes_recovered': 10158080, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3263}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:13.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:13.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4068", 'reported_seq': 4130, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:11.214585+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:11.214585+0000', 'last_peered': '2024-09-18T11:46:11.214585+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:11.214585+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4068, 'log_dups_size': 0, 'ondisk_log_size': 4068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33281270, 'num_objects': 4063, 'num_object_clones': 0, 'num_object_copies': 16252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4066, 'num_write_kb': 32516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2050}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:13.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:13.866 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:13.866 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32187/264260 objects degraded (12.180%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":171,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66065,"data_bytes":541164916,"bytes_used":2205769728,"bytes_avail":384341286912,"bytes_total":386547056640,"degraded_objects":32187,"degraded_total":264260,"degraded_ratio":0.12180049950806024,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":153534,"recovering_keys_per_sec":0,"num_objects_recovered":225,"num_bytes_recovered":1843200,"num_keys_recovered":0,"write_bytes_sec":3074794,"read_op_per_sec":0,"write_op_per_sec":375},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:13.889 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:13.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:14.234 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 117 20 67299 67279 4.49132 2.51562 0.031476 0.0347756 2024-09-18T11:46:14.491 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:14.492 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-18T11:46:13.860007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":541164916,"num_objects":66065,"num_object_clones":0,"num_object_copies":264260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66092,"num_write_kb":528618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66119,"ondisk_log_size":66119,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2154072,"kb_used_data":925812,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375333288,"statfs":{"total":386547056640,"available":384341286912,"internally_reserved":0,"allocated":948031488,"data_stored":947395576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,12,7,4,9,13,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28311552,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3456,"num_write_kb":27648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005124"},"pg_stats":[{"pgid":"1.f","version":"31'4185","reported_seq":4225,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.606629+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:09.606629+0000","last_peered":"2024-09-18T11:46:09.606629+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:09.606629+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34283520,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4172","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.150553+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:11.150553+0000","last_peered":"2024-09-18T11:46:11.150553+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:11.150553+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34141430,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4170,"num_write_kb":33356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4114","reported_seq":4196,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.203833+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:11.203833+0000","last_peered":"2024-09-18T11:46:11.203833+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:11.203833+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33675899,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4113,"num_write_kb":32894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2057}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4177","reported_seq":4239,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.270412+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:11.270412+0000","last_peered":"2024-09-18T11:46:11.270412+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:11.270412+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4177,"log_dups_size":0,"ondisk_log_size":4177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34182390,"num_objects":4173,"num_object_clones":0,"num_object_copies":16692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":33389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4085","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.566959+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:09.566959+0000","last_peered":"2024-09-18T11:46:09.566959+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:09.566959+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33436918,"num_objects":4082,"num_object_clones":0,"num_object_copies":16328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4083,"num_write_kb":32661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4090","reported_seq":4133,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.399036+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:09.399036+0000","last_peered":"2024-09-18T11:46:09.399036+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:09.399036+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":4090,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4090,"num_write_kb":32720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4181","reported_seq":4245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.131419+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:11.131419+0000","last_peered":"2024-09-18T11:46:11.131419+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:11.131419+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34198774,"num_objects":4175,"num_object_clones":0,"num_object_copies":16700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4179,"num_write_kb":33412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4066","reported_seq":4129,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.101752+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:11.101752+0000","last_peered":"2024-09-18T11:46:11.101752+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:11.101752+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4066,"log_dups_size":0,"ondisk_log_size":4066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264886,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":32500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2027}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4217","reported_seq":4275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.580380+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:09.580380+0000","last_peered":"2024-09-18T11:46:09.580380+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:09.580380+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34474476,"num_objects":4209,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4214,"num_write_kb":33689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4030","reported_seq":4114,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.200520+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:11.200520+0000","last_peered":"2024-09-18T11:46:11.200520+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:11.200520+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32994550,"num_objects":4028,"num_object_clones":0,"num_object_copies":16112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4028,"num_write_kb":32222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2053}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4115","reported_seq":4172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.504509+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:09.504509+0000","last_peered":"2024-09-18T11:46:09.504509+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:09.504509+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4115,"log_dups_size":0,"ondisk_log_size":4115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33664881,"num_objects":4110,"num_object_clones":0,"num_object_copies":16440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4185","reported_seq":4222,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.581589+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:09.581589+0000","last_peered":"2024-09-18T11:46:09.581589+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:09.581589+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34273915,"num_objects":4184,"num_object_clones":0,"num_object_copies":16736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4184,"num_write_kb":33471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2075}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4159","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.167075+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:11.167075+0000","last_peered":"2024-09-18T11:46:11.167075+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:11.167075+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060923,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4158","reported_seq":4207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:09.452054+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:09.452054+0000","last_peered":"2024-09-18T11:46:09.452054+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:09.452054+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34026742,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4117","reported_seq":6680,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.203048+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:11.203048+0000","last_peered":"2024-09-18T11:46:11.203048+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:11.203048+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4117,"log_dups_size":0,"ondisk_log_size":4117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33699062,"num_objects":4114,"num_object_clones":0,"num_object_copies":16456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4115,"num_write_kb":32917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1242,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3263}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4068","reported_seq":4130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:11.214585+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:11.214585+0000","last_peered":"2024-09-18T11:46:11.214585+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:11.214585+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33281270,"num_objects":4063,"num_object_clones":0,"num_object_copies":16252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4066,"num_write_kb":32516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":541164916,"num_objects":66065,"num_object_clones":0,"num_object_copies":264260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66092,"num_write_kb":528618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":949776384,"data_stored":949776384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66119,"ondisk_log_size":66119,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564012,"kb_used_data":261988,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93807828,"statfs":{"total":96636764160,"available":96059215872,"internally_reserved":0,"allocated":268275712,"data_stored":268116734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,11,3,1,1,2,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":10.08},{"interface":"front","average":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.538,"5min":12.538,"15min":12.538},"last":10.111000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":6.8849999999999998},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":11.904}]},{"osd":2,"last update":"Wed Sep 18 11:45:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":2.23},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.834,"5min":10.834,"15min":10.834},"last":4.3360000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574096,"kb_used_data":264260,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93797744,"statfs":{"total":96636764160,"available":96048889856,"internally_reserved":0,"allocated":270602240,"data_stored":270443262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.042},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":14.537000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":14.601000000000001},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":1.0669999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":0.21099999999999999},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":7.4429999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444672,"kb_used_data":134392,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93927168,"statfs":{"total":96636764160,"available":96181420032,"internally_reserved":0,"allocated":137617408,"data_stored":137458430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.34899999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":24.600000000000001},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":0.91300000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":3.847}]}]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571292,"kb_used_data":265172,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93800548,"statfs":{"total":96636764160,"available":96051761152,"internally_reserved":0,"allocated":271536128,"data_stored":271377150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":4.0780000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.117},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":1.125}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.26300000000000001},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":4.1399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271642624,"data_stored":271642624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137355264,"data_stored":137355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":271589376,"data_stored":271589376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":269189120,"data_stored":269189120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:15.234 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 118 20 67639 67619 4.47577 2.65625 0.0427449 0.0348975 2024-09-18T11:46:16.234 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 119 20 67994 67974 4.46147 2.77344 0.113519 0.0350058 2024-09-18T11:46:17.235 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:46:17.234289+0000 min lat: 0.00272854 max lat: 2.42146 avg lat: 0.0351177 2024-09-18T11:46:17.235 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:46:17.235 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 120 20 68353 68333 4.44766 2.80469 0.0795335 0.0351177 2024-09-18T11:46:17.518 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:18.065 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:18.068 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:18.089 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-18T11:46:17.862480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":555328884,"num_objects":67794,"num_object_clones":0,"num_object_copies":271176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67821,"num_write_kb":542450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1349,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67848,"ondisk_log_size":67848,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2164956,"kb_used_data":953340,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375322404,"statfs":{"total":386547056640,"available":384330141696,"internally_reserved":0,"allocated":976220160,"data_stored":975584248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,10,14,11,21,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36192256,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":1925120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005611"},"pg_stats":[{"pgid":"1.f","version":"31'4314","reported_seq":4353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.633951+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:14.633951+0000","last_peered":"2024-09-18T11:46:14.633951+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:14.633951+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35340288,"num_objects":4314,"num_object_clones":0,"num_object_copies":17256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4314,"num_write_kb":34512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4282","reported_seq":4345,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.195170+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:16.195170+0000","last_peered":"2024-09-18T11:46:16.195170+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:16.195170+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35042550,"num_objects":4278,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4221","reported_seq":4303,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.257709+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:16.257709+0000","last_peered":"2024-09-18T11:46:16.257709+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:16.257709+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34552443,"num_objects":4218,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4220,"num_write_kb":33750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4267","reported_seq":4329,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.241803+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:16.241803+0000","last_peered":"2024-09-18T11:46:16.241803+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:16.241803+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34919670,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4265,"num_write_kb":34109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4197","reported_seq":4250,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.669345+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:14.669345+0000","last_peered":"2024-09-18T11:46:14.669345+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:14.669345+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34354422,"num_objects":4194,"num_object_clones":0,"num_object_copies":16776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4188","reported_seq":4231,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.626538+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:14.626538+0000","last_peered":"2024-09-18T11:46:14.626538+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:14.626538+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34308096,"num_objects":4188,"num_object_clones":0,"num_object_copies":16752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4188,"num_write_kb":33504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2034}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4284","reported_seq":4349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.165484+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:16.165484+0000","last_peered":"2024-09-18T11:46:16.165484+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:16.165484+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35042550,"num_objects":4278,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":34236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2144}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4187","reported_seq":4250,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.144638+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:16.144638+0000","last_peered":"2024-09-18T11:46:16.144638+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:16.144638+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256118,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4323","reported_seq":4381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.640983+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:14.640983+0000","last_peered":"2024-09-18T11:46:14.640983+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:14.640983+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35342828,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4320,"num_write_kb":34537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4146","reported_seq":4229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.246652+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:16.246652+0000","last_peered":"2024-09-18T11:46:16.246652+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:16.246652+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4146,"log_dups_size":0,"ondisk_log_size":4146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33944822,"num_objects":4144,"num_object_clones":0,"num_object_copies":16576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4144,"num_write_kb":33150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4221","reported_seq":4278,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.402507+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:14.402507+0000","last_peered":"2024-09-18T11:46:14.402507+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:14.402507+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34533233,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":33732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4292","reported_seq":4329,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.597652+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:14.597652+0000","last_peered":"2024-09-18T11:46:14.597652+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:14.597652+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4292,"log_dups_size":0,"ondisk_log_size":4292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35150459,"num_objects":4291,"num_object_clones":0,"num_object_copies":17164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":34327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2182}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4281","reported_seq":4357,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.120506+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:16.120506+0000","last_peered":"2024-09-18T11:46:16.120506+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:16.120506+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35060347,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4265","reported_seq":4314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.525336+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:14.525336+0000","last_peered":"2024-09-18T11:46:14.525336+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:14.525336+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34903286,"num_objects":4261,"num_object_clones":0,"num_object_copies":17044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4263,"num_write_kb":34100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4217","reported_seq":6944,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.247818+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:16.247818+0000","last_peered":"2024-09-18T11:46:16.247818+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:16.247818+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34518262,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4215,"num_write_kb":33717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4163","reported_seq":4223,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.259741+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:16.259741+0000","last_peered":"2024-09-18T11:46:16.259741+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:16.259741+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059510,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4161,"num_write_kb":33276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":555328884,"num_objects":67794,"num_object_clones":0,"num_object_copies":271176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67821,"num_write_kb":542450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1349,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":978116608,"data_stored":978116608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67848,"ondisk_log_size":67848,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575108,"kb_used_data":268924,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93796732,"statfs":{"total":96636764160,"available":96047853568,"internally_reserved":0,"allocated":275378176,"data_stored":275219198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,10,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":2.677},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.7000000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":1.548},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":2.3929999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.372},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":12.734}]}]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568280,"kb_used_data":270928,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93803560,"statfs":{"total":96636764160,"available":96054845440,"internally_reserved":0,"allocated":277430272,"data_stored":277271294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.335}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":21.626999999999999},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":14.31}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":14.34},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.28799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439264,"kb_used_data":141464,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93932576,"statfs":{"total":96636764160,"available":96186957824,"internally_reserved":0,"allocated":144859136,"data_stored":144700158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,5,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":1.6850000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":2.5339999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":9.3369999999999997},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":1.5469999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":2.5699999999999998},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":4.625}]}]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582304,"kb_used_data":272024,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93789536,"statfs":{"total":96636764160,"available":96040484864,"internally_reserved":0,"allocated":278552576,"data_stored":278393598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,6,5,0,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":7.681},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":7.75}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.23999999999999999},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":4.1779999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":7.7089999999999996},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":0.26900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278605824,"data_stored":278605824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144732160,"data_stored":144732160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278540288,"data_stored":278540288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":276238336,"data_stored":276238336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:18.091 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:18.234 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 121 20 68685 68665 4.43234 2.59375 0.0415382 0.0352399 2024-09-18T11:46:18.676 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:18.677 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:18.704 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-18T11:46:17.862480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":555328884,"num_objects":67794,"num_object_clones":0,"num_object_copies":271176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67821,"num_write_kb":542450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1349,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67848,"ondisk_log_size":67848,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2164956,"kb_used_data":953340,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375322404,"statfs":{"total":386547056640,"available":384330141696,"internally_reserved":0,"allocated":976220160,"data_stored":975584248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,10,14,11,21,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36192256,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":1925120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005611"},"pg_stats":[{"pgid":"1.f","version":"31'4314","reported_seq":4353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.633951+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:14.633951+0000","last_peered":"2024-09-18T11:46:14.633951+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:14.633951+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35340288,"num_objects":4314,"num_object_clones":0,"num_object_copies":17256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4314,"num_write_kb":34512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4282","reported_seq":4345,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.195170+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:16.195170+0000","last_peered":"2024-09-18T11:46:16.195170+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:16.195170+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35042550,"num_objects":4278,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4221","reported_seq":4303,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.257709+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:16.257709+0000","last_peered":"2024-09-18T11:46:16.257709+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:16.257709+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34552443,"num_objects":4218,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4220,"num_write_kb":33750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4267","reported_seq":4329,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.241803+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:16.241803+0000","last_peered":"2024-09-18T11:46:16.241803+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:16.241803+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34919670,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4265,"num_write_kb":34109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4197","reported_seq":4250,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.669345+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:14.669345+0000","last_peered":"2024-09-18T11:46:14.669345+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:14.669345+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34354422,"num_objects":4194,"num_object_clones":0,"num_object_copies":16776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4188","reported_seq":4231,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.626538+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:14.626538+0000","last_peered":"2024-09-18T11:46:14.626538+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:14.626538+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34308096,"num_objects":4188,"num_object_clones":0,"num_object_copies":16752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4188,"num_write_kb":33504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2034}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4284","reported_seq":4349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.165484+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:16.165484+0000","last_peered":"2024-09-18T11:46:16.165484+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:16.165484+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35042550,"num_objects":4278,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":34236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2144}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4187","reported_seq":4250,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.144638+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:16.144638+0000","last_peered":"2024-09-18T11:46:16.144638+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:16.144638+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256118,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4323","reported_seq":4381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.640983+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:14.640983+0000","last_peered":"2024-09-18T11:46:14.640983+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:14.640983+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35342828,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4320,"num_write_kb":34537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4146","reported_seq":4229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.246652+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:16.246652+0000","last_peered":"2024-09-18T11:46:16.246652+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:16.246652+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4146,"log_dups_size":0,"ondisk_log_size":4146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33944822,"num_objects":4144,"num_object_clones":0,"num_object_copies":16576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4144,"num_write_kb":33150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4221","reported_seq":4278,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.402507+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:14.402507+0000","last_peered":"2024-09-18T11:46:14.402507+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:14.402507+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34533233,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":33732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4292","reported_seq":4329,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.597652+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:14.597652+0000","last_peered":"2024-09-18T11:46:14.597652+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:14.597652+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4292,"log_dups_size":0,"ondisk_log_size":4292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35150459,"num_objects":4291,"num_object_clones":0,"num_object_copies":17164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":34327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2182}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4281","reported_seq":4357,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.120506+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:16.120506+0000","last_peered":"2024-09-18T11:46:16.120506+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:16.120506+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35060347,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4265","reported_seq":4314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:14.525336+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:14.525336+0000","last_peered":"2024-09-18T11:46:14.525336+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:14.525336+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34903286,"num_objects":4261,"num_object_clones":0,"num_object_copies":17044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4263,"num_write_kb":34100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4217","reported_seq":6944,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.247818+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:16.247818+0000","last_peered":"2024-09-18T11:46:16.247818+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:16.247818+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34518262,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4215,"num_write_kb":33717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4163","reported_seq":4223,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.259741+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:16.259741+0000","last_peered":"2024-09-18T11:46:16.259741+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:16.259741+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059510,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4161,"num_write_kb":33276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":555328884,"num_objects":67794,"num_object_clones":0,"num_object_copies":271176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67821,"num_write_kb":542450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1349,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":978116608,"data_stored":978116608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67848,"ondisk_log_size":67848,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575108,"kb_used_data":268924,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93796732,"statfs":{"total":96636764160,"available":96047853568,"internally_reserved":0,"allocated":275378176,"data_stored":275219198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,10,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":2.677},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.7000000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":1.548},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":2.3929999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.372},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":12.734}]}]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568280,"kb_used_data":270928,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93803560,"statfs":{"total":96636764160,"available":96054845440,"internally_reserved":0,"allocated":277430272,"data_stored":277271294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.335}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":21.626999999999999},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":14.31}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":14.34},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.28799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439264,"kb_used_data":141464,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93932576,"statfs":{"total":96636764160,"available":96186957824,"internally_reserved":0,"allocated":144859136,"data_stored":144700158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,5,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":1.6850000000000001},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":2.5339999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":9.3369999999999997},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":1.5469999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":2.5699999999999998},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":4.625}]}]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582304,"kb_used_data":272024,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93789536,"statfs":{"total":96636764160,"available":96040484864,"internally_reserved":0,"allocated":278552576,"data_stored":278393598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,6,5,0,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":7.681},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":7.75}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.23999999999999999},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":4.1779999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":7.7089999999999996},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":0.26900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278605824,"data_stored":278605824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144732160,"data_stored":144732160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278540288,"data_stored":278540288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":276238336,"data_stored":276238336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:18.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:18.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4314", 'reported_seq': 4353, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:14.633951+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:14.633951+0000', 'last_peered': '2024-09-18T11:46:14.633951+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:14.633951+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4314, 'log_dups_size': 0, 'ondisk_log_size': 4314, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35340288, 'num_objects': 4314, 'num_object_clones': 0, 'num_object_copies': 17256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4314, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4314, 'num_write_kb': 34512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2184}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:18.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:18.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4282", 'reported_seq': 4345, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:16.195170+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:16.195170+0000', 'last_peered': '2024-09-18T11:46:16.195170+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:16.195170+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4282, 'log_dups_size': 0, 'ondisk_log_size': 4282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35042550, 'num_objects': 4278, 'num_object_clones': 0, 'num_object_copies': 17112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4280, 'num_write_kb': 34236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:18.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:18.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4221", 'reported_seq': 4303, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:16.257709+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:16.257709+0000', 'last_peered': '2024-09-18T11:46:16.257709+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:16.257709+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4221, 'log_dups_size': 0, 'ondisk_log_size': 4221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34552443, 'num_objects': 4218, 'num_object_clones': 0, 'num_object_copies': 16872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4220, 'num_write_kb': 33750, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2164}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:18.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:18.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4267", 'reported_seq': 4329, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:16.241803+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:16.241803+0000', 'last_peered': '2024-09-18T11:46:16.241803+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:16.241803+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4267, 'log_dups_size': 0, 'ondisk_log_size': 4267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34919670, 'num_objects': 4263, 'num_object_clones': 0, 'num_object_copies': 17052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4265, 'num_write_kb': 34109, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2149}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:18.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:18.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4197", 'reported_seq': 4250, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:14.669345+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:14.669345+0000', 'last_peered': '2024-09-18T11:46:14.669345+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:14.669345+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4197, 'log_dups_size': 0, 'ondisk_log_size': 4197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34354422, 'num_objects': 4194, 'num_object_clones': 0, 'num_object_copies': 16776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4195, 'num_write_kb': 33557, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2112}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:18.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:18.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4188", 'reported_seq': 4231, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:14.626538+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:14.626538+0000', 'last_peered': '2024-09-18T11:46:14.626538+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:14.626538+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4188, 'log_dups_size': 0, 'ondisk_log_size': 4188, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34308096, 'num_objects': 4188, 'num_object_clones': 0, 'num_object_copies': 16752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4188, 'num_write_kb': 33504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2034}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:18.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:18.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4284", 'reported_seq': 4349, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:16.165484+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:16.165484+0000', 'last_peered': '2024-09-18T11:46:16.165484+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:16.165484+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4284, 'log_dups_size': 0, 'ondisk_log_size': 4284, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35042550, 'num_objects': 4278, 'num_object_clones': 0, 'num_object_copies': 17112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4282, 'num_write_kb': 34236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2144}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:18.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:18.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4187", 'reported_seq': 4250, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:16.144638+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:16.144638+0000', 'last_peered': '2024-09-18T11:46:16.144638+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:16.144638+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4187, 'log_dups_size': 0, 'ondisk_log_size': 4187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34256118, 'num_objects': 4182, 'num_object_clones': 0, 'num_object_copies': 16728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4185, 'num_write_kb': 33468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2148}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:18.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:18.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4323", 'reported_seq': 4381, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:14.640983+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:14.640983+0000', 'last_peered': '2024-09-18T11:46:14.640983+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:14.640983+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4323, 'log_dups_size': 0, 'ondisk_log_size': 4323, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35342828, 'num_objects': 4315, 'num_object_clones': 0, 'num_object_copies': 17260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4320, 'num_write_kb': 34537, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2180}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:18.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:18.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4146", 'reported_seq': 4229, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:16.246652+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:16.246652+0000', 'last_peered': '2024-09-18T11:46:16.246652+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:16.246652+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4146, 'log_dups_size': 0, 'ondisk_log_size': 4146, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33944822, 'num_objects': 4144, 'num_object_clones': 0, 'num_object_copies': 16576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4144, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4144, 'num_write_kb': 33150, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2169}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:18.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:18.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4221", 'reported_seq': 4278, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:14.402507+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:14.402507+0000', 'last_peered': '2024-09-18T11:46:14.402507+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:14.402507+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4221, 'log_dups_size': 0, 'ondisk_log_size': 4221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34533233, 'num_objects': 4216, 'num_object_clones': 0, 'num_object_copies': 16864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4218, 'num_write_kb': 33732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2153}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:18.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:18.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4292", 'reported_seq': 4329, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:14.597652+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:14.597652+0000', 'last_peered': '2024-09-18T11:46:14.597652+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:14.597652+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4292, 'log_dups_size': 0, 'ondisk_log_size': 4292, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35150459, 'num_objects': 4291, 'num_object_clones': 0, 'num_object_copies': 17164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4291, 'num_write_kb': 34327, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2182}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:18.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:18.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4281", 'reported_seq': 4357, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:16.120506+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:16.120506+0000', 'last_peered': '2024-09-18T11:46:16.120506+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:16.120506+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4281, 'log_dups_size': 0, 'ondisk_log_size': 4281, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35060347, 'num_objects': 4280, 'num_object_clones': 0, 'num_object_copies': 17120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4280, 'num_write_kb': 34239, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:18.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:18.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4265", 'reported_seq': 4314, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:14.525336+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:14.525336+0000', 'last_peered': '2024-09-18T11:46:14.525336+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:14.525336+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4265, 'log_dups_size': 0, 'ondisk_log_size': 4265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34903286, 'num_objects': 4261, 'num_object_clones': 0, 'num_object_copies': 17044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4263, 'num_write_kb': 34100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:18.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:18.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4217", 'reported_seq': 6944, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:16.247818+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:16.247818+0000', 'last_peered': '2024-09-18T11:46:16.247818+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:16.247818+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4217, 'log_dups_size': 0, 'ondisk_log_size': 4217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34518262, 'num_objects': 4214, 'num_object_clones': 0, 'num_object_copies': 16856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 769, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4215, 'num_write_kb': 33717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1324, 'num_bytes_recovered': 10829824, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:18.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:18.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4163", 'reported_seq': 4223, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:16.259741+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:16.259741+0000', 'last_peered': '2024-09-18T11:46:16.259741+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:16.259741+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4163, 'log_dups_size': 0, 'ondisk_log_size': 4163, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34059510, 'num_objects': 4158, 'num_object_clones': 0, 'num_object_copies': 16632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4161, 'num_write_kb': 33276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2145}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:18.709 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:19.235 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 122 20 68993 68973 4.41573 2.40625 0.0547527 0.0353714 2024-09-18T11:46:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:19.377 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32105/271176 objects degraded (11.839%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":177,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67794,"data_bytes":555328884,"bytes_used":2216914944,"bytes_avail":384330141696,"bytes_total":386547056640,"degraded_objects":32105,"degraded_total":271176,"degraded_ratio":0.11839174558220492,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":160351,"recovering_keys_per_sec":0,"num_objects_recovered":235,"num_bytes_recovered":1925120,"num_keys_recovered":0,"write_bytes_sec":3014611,"read_op_per_sec":0,"write_op_per_sec":367},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:19.410 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:19.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:20.005 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:20.005 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:20.032 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-18T11:46:19.863141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":561284468,"num_objects":68521,"num_object_clones":0,"num_object_copies":274084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68548,"num_write_kb":548266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1349,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68575,"ondisk_log_size":68575,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2187080,"kb_used_data":967144,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375300280,"statfs":{"total":386547056640,"available":384307486720,"internally_reserved":0,"allocated":990355456,"data_stored":989719544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,9,16,14,18,15,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34086912,"num_objects":4161,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4161,"num_write_kb":33288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005397"},"pg_stats":[{"pgid":"1.f","version":"31'4420","reported_seq":4460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.685345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:19.685345+0000","last_peered":"2024-09-18T11:46:19.685345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:19.685345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36208640,"num_objects":4420,"num_object_clones":0,"num_object_copies":17680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4420,"num_write_kb":35360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4282","reported_seq":4345,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.195170+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:16.195170+0000","last_peered":"2024-09-18T11:46:16.195170+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:16.195170+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35042550,"num_objects":4278,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4221","reported_seq":4303,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.257709+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:16.257709+0000","last_peered":"2024-09-18T11:46:16.257709+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:16.257709+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34552443,"num_objects":4218,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4220,"num_write_kb":33750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4267","reported_seq":4329,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.241803+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:16.241803+0000","last_peered":"2024-09-18T11:46:16.241803+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:16.241803+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34919670,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4265,"num_write_kb":34109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4298","reported_seq":4353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.552411+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:19.552411+0000","last_peered":"2024-09-18T11:46:19.552411+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:19.552411+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35181814,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4296,"num_write_kb":34365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4281","reported_seq":4323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.628098+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:19.628098+0000","last_peered":"2024-09-18T11:46:19.628098+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:19.628098+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35069952,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4281,"num_write_kb":34248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4284","reported_seq":4349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.165484+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:16.165484+0000","last_peered":"2024-09-18T11:46:16.165484+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:16.165484+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35042550,"num_objects":4278,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":34236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2144}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4187","reported_seq":4250,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.144638+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:16.144638+0000","last_peered":"2024-09-18T11:46:16.144638+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:16.144638+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256118,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4440","reported_seq":4498,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.630424+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:19.630424+0000","last_peered":"2024-09-18T11:46:19.630424+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:19.630424+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36301292,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2297}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4146","reported_seq":4229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.246652+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:16.246652+0000","last_peered":"2024-09-18T11:46:16.246652+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:16.246652+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4146,"log_dups_size":0,"ondisk_log_size":4146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33944822,"num_objects":4144,"num_object_clones":0,"num_object_copies":16576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4144,"num_write_kb":33150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4332","reported_seq":4389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.610277+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:19.610277+0000","last_peered":"2024-09-18T11:46:19.610277+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:19.610277+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35442545,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4329,"num_write_kb":34620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4397","reported_seq":4433,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.623725+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:19.623725+0000","last_peered":"2024-09-18T11:46:19.623725+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:19.623725+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36010619,"num_objects":4396,"num_object_clones":0,"num_object_copies":17584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":35167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2287}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4281","reported_seq":4357,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.120506+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:16.120506+0000","last_peered":"2024-09-18T11:46:16.120506+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:16.120506+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35060347,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4359","reported_seq":4408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.641803+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:19.641803+0000","last_peered":"2024-09-18T11:46:19.641803+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:19.641803+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4359,"log_dups_size":0,"ondisk_log_size":4359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35673334,"num_objects":4355,"num_object_clones":0,"num_object_copies":17420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4357,"num_write_kb":34852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2197}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4217","reported_seq":6944,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.247818+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:16.247818+0000","last_peered":"2024-09-18T11:46:16.247818+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:16.247818+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34518262,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4215,"num_write_kb":33717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4163","reported_seq":4223,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:16.259741+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:16.259741+0000","last_peered":"2024-09-18T11:46:16.259741+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:16.259741+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059510,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4161,"num_write_kb":33276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":561284468,"num_objects":68521,"num_object_clones":0,"num_object_copies":274084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68548,"num_write_kb":548266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1349,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":992444416,"data_stored":992444416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68575,"ondisk_log_size":68575,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585980,"kb_used_data":275636,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93785860,"statfs":{"total":96636764160,"available":96036720640,"internally_reserved":0,"allocated":282251264,"data_stored":282092286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,4,9,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":1.9099999999999999},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.944}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.2410000000000001},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":2.4889999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.0429999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568280,"kb_used_data":270928,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93803560,"statfs":{"total":96636764160,"available":96054845440,"internally_reserved":0,"allocated":277430272,"data_stored":277271294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.335}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":21.626999999999999},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":14.31}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":14.34},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.28799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450516,"kb_used_data":148556,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93921324,"statfs":{"total":96636764160,"available":96175435776,"internally_reserved":0,"allocated":152121344,"data_stored":151962366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,9,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":5.6449999999999996},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":5.0839999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":1.8839999999999999},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":5.6059999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":5.1180000000000003},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":11.303000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582304,"kb_used_data":272024,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93789536,"statfs":{"total":96636764160,"available":96040484864,"internally_reserved":0,"allocated":278552576,"data_stored":278393598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,6,5,0,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":7.681},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":7.75}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.23999999999999999},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":4.1779999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":7.7089999999999996},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":0.26900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278605824,"data_stored":278605824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152080384,"data_stored":152080384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278540288,"data_stored":278540288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283217920,"data_stored":283217920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:20.235 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 123 20 69313 69293 4.40016 2.5 0.0404432 0.0354956 2024-09-18T11:46:21.235 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 124 20 69662 69642 4.38666 2.72656 0.0358027 0.0356051 2024-09-18T11:46:22.235 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 125 20 70015 69995 4.37362 2.75781 0.0510449 0.035711 2024-09-18T11:46:23.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:23.235 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 126 20 70362 70342 4.36043 2.71094 0.0639338 0.0358207 2024-09-18T11:46:23.609 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:23.609 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:23.642 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-09-18T11:46:21.863825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569009524,"num_objects":69464,"num_object_clones":0,"num_object_copies":277856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69491,"num_write_kb":555810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1431,"num_bytes_recovered":11501568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69518,"ondisk_log_size":69518,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2196336,"kb_used_data":980560,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":375291024,"statfs":{"total":386547056640,"available":384298008576,"internally_reserved":0,"allocated":1004093440,"data_stored":1003457528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,4,11,23,28,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35569664,"num_objects":4342,"num_object_clones":0,"num_object_copies":17368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4342,"num_write_kb":34736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":1982464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005348"},"pg_stats":[{"pgid":"1.f","version":"31'4420","reported_seq":4460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.685345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:19.685345+0000","last_peered":"2024-09-18T11:46:19.685345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:19.685345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36208640,"num_objects":4420,"num_object_clones":0,"num_object_copies":17680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4420,"num_write_kb":35360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4397","reported_seq":4460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.253108+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:21.253108+0000","last_peered":"2024-09-18T11:46:21.253108+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:21.253108+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35984630,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4395,"num_write_kb":35156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2291}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4313","reported_seq":4393,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.047214+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:21.047214+0000","last_peered":"2024-09-18T11:46:21.047214+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:21.047214+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35306107,"num_objects":4310,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4312,"num_write_kb":34486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4381","reported_seq":4443,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.112192+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:21.112192+0000","last_peered":"2024-09-18T11:46:21.112192+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:21.112192+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853558,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4379,"num_write_kb":35021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4298","reported_seq":4353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.552411+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:19.552411+0000","last_peered":"2024-09-18T11:46:19.552411+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:19.552411+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35181814,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4296,"num_write_kb":34365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4281","reported_seq":4323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.628098+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:19.628098+0000","last_peered":"2024-09-18T11:46:19.628098+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:19.628098+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35069952,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4281,"num_write_kb":34248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4401","reported_seq":4465,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.272942+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:21.272942+0000","last_peered":"2024-09-18T11:46:21.272942+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:21.272942+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36001014,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4399,"num_write_kb":35172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2261}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4299","reported_seq":4362,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.257672+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:21.257672+0000","last_peered":"2024-09-18T11:46:21.257672+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:21.257672+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35173622,"num_objects":4294,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4297,"num_write_kb":34364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4440","reported_seq":4498,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.630424+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:19.630424+0000","last_peered":"2024-09-18T11:46:19.630424+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:19.630424+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36301292,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2297}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4250","reported_seq":4334,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.125813+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:21.125813+0000","last_peered":"2024-09-18T11:46:21.125813+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:21.125813+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34796790,"num_objects":4248,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4248,"num_write_kb":33982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2273}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4332","reported_seq":4389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.610277+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:19.610277+0000","last_peered":"2024-09-18T11:46:19.610277+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:19.610277+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35442545,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4329,"num_write_kb":34620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4397","reported_seq":4433,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.623725+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:19.623725+0000","last_peered":"2024-09-18T11:46:19.623725+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:19.623725+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36010619,"num_objects":4396,"num_object_clones":0,"num_object_copies":17584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":35167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2287}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4390","reported_seq":4466,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.243347+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:21.243347+0000","last_peered":"2024-09-18T11:46:21.243347+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:21.243347+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35953275,"num_objects":4389,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":35111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4359","reported_seq":4408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.641803+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:19.641803+0000","last_peered":"2024-09-18T11:46:19.641803+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:19.641803+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4359,"log_dups_size":0,"ondisk_log_size":4359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35673334,"num_objects":4355,"num_object_clones":0,"num_object_copies":17420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4357,"num_write_kb":34852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2197}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4312","reported_seq":7202,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.260338+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:21.260338+0000","last_peered":"2024-09-18T11:46:21.260338+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:21.260338+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35296502,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":34477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1406,"num_bytes_recovered":11501568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3622}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4248","reported_seq":4310,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.172348+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:21.172348+0000","last_peered":"2024-09-18T11:46:21.172348+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:21.172348+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34755830,"num_objects":4243,"num_object_clones":0,"num_object_copies":16972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4246,"num_write_kb":33956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":569009524,"num_objects":69464,"num_object_clones":0,"num_object_copies":277856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69491,"num_write_kb":555810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1431,"num_bytes_recovered":11501568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1006141440,"data_stored":1006141440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69518,"ondisk_log_size":69518,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585980,"kb_used_data":275636,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93785860,"statfs":{"total":96636764160,"available":96036720640,"internally_reserved":0,"allocated":282251264,"data_stored":282092286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,4,9,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":1.9099999999999999},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.944}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.2410000000000001},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":2.4889999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.0429999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579124,"kb_used_data":277612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93792716,"statfs":{"total":96636764160,"available":96043741184,"internally_reserved":0,"allocated":284274688,"data_stored":284115710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":8.5830000000000002},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":13.119999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":8.0470000000000006},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":13.194000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":13.986000000000001},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":14.442}]}]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450516,"kb_used_data":148556,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93921324,"statfs":{"total":96636764160,"available":96175435776,"internally_reserved":0,"allocated":152121344,"data_stored":151962366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,9,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":5.6449999999999996},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":5.0839999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":1.8839999999999999},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":5.6059999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":5.1180000000000003},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":11.303000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580716,"kb_used_data":278756,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93791124,"statfs":{"total":96636764160,"available":96042110976,"internally_reserved":0,"allocated":285446144,"data_stored":285287166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,4,14,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":5.0149999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.9850000000000003},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":7.4749999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":1.833},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":5.0439999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285446144,"data_stored":285446144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152080384,"data_stored":152080384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":285396992,"data_stored":285396992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283217920,"data_stored":283217920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:23.644 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:24.216 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:24.217 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:24.235 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 127 20 70665 70645 4.34473 2.36719 0.0834773 0.0359482 2024-09-18T11:46:24.245 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-18T11:46:23.864758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569009524,"num_objects":69464,"num_object_clones":0,"num_object_copies":277856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69491,"num_write_kb":555810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1431,"num_bytes_recovered":11501568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69518,"ondisk_log_size":69518,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2196336,"kb_used_data":980560,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":375291024,"statfs":{"total":386547056640,"available":384298008576,"internally_reserved":0,"allocated":1004093440,"data_stored":1003457528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,4,11,23,28,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27844608,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":27192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005601"},"pg_stats":[{"pgid":"1.f","version":"31'4420","reported_seq":4460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.685345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:19.685345+0000","last_peered":"2024-09-18T11:46:19.685345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:19.685345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36208640,"num_objects":4420,"num_object_clones":0,"num_object_copies":17680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4420,"num_write_kb":35360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4397","reported_seq":4460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.253108+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:21.253108+0000","last_peered":"2024-09-18T11:46:21.253108+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:21.253108+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35984630,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4395,"num_write_kb":35156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2291}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4313","reported_seq":4393,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.047214+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:21.047214+0000","last_peered":"2024-09-18T11:46:21.047214+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:21.047214+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35306107,"num_objects":4310,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4312,"num_write_kb":34486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4381","reported_seq":4443,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.112192+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:21.112192+0000","last_peered":"2024-09-18T11:46:21.112192+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:21.112192+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853558,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4379,"num_write_kb":35021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4298","reported_seq":4353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.552411+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:19.552411+0000","last_peered":"2024-09-18T11:46:19.552411+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:19.552411+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35181814,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4296,"num_write_kb":34365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4281","reported_seq":4323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.628098+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:19.628098+0000","last_peered":"2024-09-18T11:46:19.628098+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:19.628098+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35069952,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4281,"num_write_kb":34248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4401","reported_seq":4465,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.272942+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:21.272942+0000","last_peered":"2024-09-18T11:46:21.272942+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:21.272942+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36001014,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4399,"num_write_kb":35172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2261}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4299","reported_seq":4362,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.257672+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:21.257672+0000","last_peered":"2024-09-18T11:46:21.257672+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:21.257672+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35173622,"num_objects":4294,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4297,"num_write_kb":34364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4440","reported_seq":4498,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.630424+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:19.630424+0000","last_peered":"2024-09-18T11:46:19.630424+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:19.630424+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36301292,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2297}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4250","reported_seq":4334,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.125813+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:21.125813+0000","last_peered":"2024-09-18T11:46:21.125813+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:21.125813+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34796790,"num_objects":4248,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4248,"num_write_kb":33982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2273}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4332","reported_seq":4389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.610277+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:19.610277+0000","last_peered":"2024-09-18T11:46:19.610277+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:19.610277+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35442545,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4329,"num_write_kb":34620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4397","reported_seq":4433,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.623725+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:19.623725+0000","last_peered":"2024-09-18T11:46:19.623725+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:19.623725+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36010619,"num_objects":4396,"num_object_clones":0,"num_object_copies":17584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":35167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2287}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4390","reported_seq":4466,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.243347+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:21.243347+0000","last_peered":"2024-09-18T11:46:21.243347+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:21.243347+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35953275,"num_objects":4389,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":35111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4359","reported_seq":4408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.641803+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:19.641803+0000","last_peered":"2024-09-18T11:46:19.641803+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:19.641803+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4359,"log_dups_size":0,"ondisk_log_size":4359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35673334,"num_objects":4355,"num_object_clones":0,"num_object_copies":17420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4357,"num_write_kb":34852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2197}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4312","reported_seq":7202,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.260338+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:21.260338+0000","last_peered":"2024-09-18T11:46:21.260338+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:21.260338+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35296502,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":34477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1406,"num_bytes_recovered":11501568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3622}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4248","reported_seq":4310,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.172348+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:21.172348+0000","last_peered":"2024-09-18T11:46:21.172348+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:21.172348+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34755830,"num_objects":4243,"num_object_clones":0,"num_object_copies":16972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4246,"num_write_kb":33956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":569009524,"num_objects":69464,"num_object_clones":0,"num_object_copies":277856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69491,"num_write_kb":555810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1431,"num_bytes_recovered":11501568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1006141440,"data_stored":1006141440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69518,"ondisk_log_size":69518,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585980,"kb_used_data":275636,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93785860,"statfs":{"total":96636764160,"available":96036720640,"internally_reserved":0,"allocated":282251264,"data_stored":282092286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,4,9,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":1.9099999999999999},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.944}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.2410000000000001},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":2.4889999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.0429999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579124,"kb_used_data":277612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93792716,"statfs":{"total":96636764160,"available":96043741184,"internally_reserved":0,"allocated":284274688,"data_stored":284115710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":8.5830000000000002},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":13.119999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":8.0470000000000006},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":13.194000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":13.986000000000001},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":14.442}]}]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450516,"kb_used_data":148556,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93921324,"statfs":{"total":96636764160,"available":96175435776,"internally_reserved":0,"allocated":152121344,"data_stored":151962366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,9,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":5.6449999999999996},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":5.0839999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":1.8839999999999999},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":5.6059999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":5.1180000000000003},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":11.303000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580716,"kb_used_data":278756,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93791124,"statfs":{"total":96636764160,"available":96042110976,"internally_reserved":0,"allocated":285446144,"data_stored":285287166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,4,14,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":5.0149999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.9850000000000003},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":7.4749999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":1.833},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":5.0439999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285446144,"data_stored":285446144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152080384,"data_stored":152080384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":285396992,"data_stored":285396992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283217920,"data_stored":283217920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:24.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:24.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4420", 'reported_seq': 4460, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:19.685345+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:19.685345+0000', 'last_peered': '2024-09-18T11:46:19.685345+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:19.685345+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4420, 'log_dups_size': 0, 'ondisk_log_size': 4420, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36208640, 'num_objects': 4420, 'num_object_clones': 0, 'num_object_copies': 17680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4420, 'num_write_kb': 35360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2290}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:24.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:24.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4397", 'reported_seq': 4460, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:21.253108+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:21.253108+0000', 'last_peered': '2024-09-18T11:46:21.253108+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:21.253108+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4397, 'log_dups_size': 0, 'ondisk_log_size': 4397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35984630, 'num_objects': 4393, 'num_object_clones': 0, 'num_object_copies': 17572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4395, 'num_write_kb': 35156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2291}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:24.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:24.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4313", 'reported_seq': 4393, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:21.047214+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:21.047214+0000', 'last_peered': '2024-09-18T11:46:21.047214+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:21.047214+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4313, 'log_dups_size': 0, 'ondisk_log_size': 4313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35306107, 'num_objects': 4310, 'num_object_clones': 0, 'num_object_copies': 17240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4312, 'num_write_kb': 34486, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2256}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:24.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:24.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4381", 'reported_seq': 4443, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:21.112192+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:21.112192+0000', 'last_peered': '2024-09-18T11:46:21.112192+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:21.112192+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4381, 'log_dups_size': 0, 'ondisk_log_size': 4381, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35853558, 'num_objects': 4377, 'num_object_clones': 0, 'num_object_copies': 17508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4379, 'num_write_kb': 35021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2263}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:24.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:24.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4298", 'reported_seq': 4353, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:19.552411+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:19.552411+0000', 'last_peered': '2024-09-18T11:46:19.552411+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:19.552411+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4298, 'log_dups_size': 0, 'ondisk_log_size': 4298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35181814, 'num_objects': 4295, 'num_object_clones': 0, 'num_object_copies': 17180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4296, 'num_write_kb': 34365, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2213}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:24.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:24.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4281", 'reported_seq': 4323, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:19.628098+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:19.628098+0000', 'last_peered': '2024-09-18T11:46:19.628098+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:19.628098+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4281, 'log_dups_size': 0, 'ondisk_log_size': 4281, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35069952, 'num_objects': 4281, 'num_object_clones': 0, 'num_object_copies': 17124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4281, 'num_write_kb': 34248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2127}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:24.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:24.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4401", 'reported_seq': 4465, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:21.272942+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:21.272942+0000', 'last_peered': '2024-09-18T11:46:21.272942+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:21.272942+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4401, 'log_dups_size': 0, 'ondisk_log_size': 4401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36001014, 'num_objects': 4395, 'num_object_clones': 0, 'num_object_copies': 17580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4399, 'num_write_kb': 35172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2261}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:24.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:24.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4299", 'reported_seq': 4362, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:21.257672+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:21.257672+0000', 'last_peered': '2024-09-18T11:46:21.257672+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:21.257672+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4299, 'log_dups_size': 0, 'ondisk_log_size': 4299, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35173622, 'num_objects': 4294, 'num_object_clones': 0, 'num_object_copies': 17176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4297, 'num_write_kb': 34364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2260}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:24.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:24.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4440", 'reported_seq': 4498, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:19.630424+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:19.630424+0000', 'last_peered': '2024-09-18T11:46:19.630424+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:19.630424+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4440, 'log_dups_size': 0, 'ondisk_log_size': 4440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36301292, 'num_objects': 4432, 'num_object_clones': 0, 'num_object_copies': 17728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4437, 'num_write_kb': 35473, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2297}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:24.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:24.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4250", 'reported_seq': 4334, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:21.125813+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:21.125813+0000', 'last_peered': '2024-09-18T11:46:21.125813+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:21.125813+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4250, 'log_dups_size': 0, 'ondisk_log_size': 4250, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34796790, 'num_objects': 4248, 'num_object_clones': 0, 'num_object_copies': 16992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4248, 'num_write_kb': 33982, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2273}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:24.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:24.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4332", 'reported_seq': 4389, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:19.610277+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:19.610277+0000', 'last_peered': '2024-09-18T11:46:19.610277+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:19.610277+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4332, 'log_dups_size': 0, 'ondisk_log_size': 4332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35442545, 'num_objects': 4327, 'num_object_clones': 0, 'num_object_copies': 17308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4329, 'num_write_kb': 34620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2264}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:24.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:24.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4397", 'reported_seq': 4433, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:19.623725+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:19.623725+0000', 'last_peered': '2024-09-18T11:46:19.623725+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:19.623725+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4397, 'log_dups_size': 0, 'ondisk_log_size': 4397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36010619, 'num_objects': 4396, 'num_object_clones': 0, 'num_object_copies': 17584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4396, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4396, 'num_write_kb': 35167, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:24.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:24.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4390", 'reported_seq': 4466, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:21.243347+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:21.243347+0000', 'last_peered': '2024-09-18T11:46:21.243347+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:21.243347+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4390, 'log_dups_size': 0, 'ondisk_log_size': 4390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35953275, 'num_objects': 4389, 'num_object_clones': 0, 'num_object_copies': 17556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4389, 'num_write_kb': 35111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:24.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:24.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4359", 'reported_seq': 4408, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:19.641803+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:19.641803+0000', 'last_peered': '2024-09-18T11:46:19.641803+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:19.641803+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4359, 'log_dups_size': 0, 'ondisk_log_size': 4359, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35673334, 'num_objects': 4355, 'num_object_clones': 0, 'num_object_copies': 17420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4357, 'num_write_kb': 34852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2197}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:24.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:24.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4312", 'reported_seq': 7202, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:21.260338+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:21.260338+0000', 'last_peered': '2024-09-18T11:46:21.260338+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:21.260338+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4312, 'log_dups_size': 0, 'ondisk_log_size': 4312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35296502, 'num_objects': 4309, 'num_object_clones': 0, 'num_object_copies': 17236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 687, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4310, 'num_write_kb': 34477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1406, 'num_bytes_recovered': 11501568, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3622}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:24.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:24.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4248", 'reported_seq': 4310, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:21.172348+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:21.172348+0000', 'last_peered': '2024-09-18T11:46:21.172348+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:21.172348+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4248, 'log_dups_size': 0, 'ondisk_log_size': 4248, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34755830, 'num_objects': 4243, 'num_object_clones': 0, 'num_object_copies': 16972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4243, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4246, 'num_write_kb': 33956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:24.250 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:24.890 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:24.890 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32023/277856 objects degraded (11.525%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":182,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":69464,"data_bytes":569009524,"bytes_used":2249048064,"bytes_avail":384298008576,"bytes_total":386547056640,"degraded_objects":32023,"degraded_total":277856,"degraded_ratio":0.11525034550270644,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":111905,"recovering_keys_per_sec":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"write_bytes_sec":2319301,"read_op_per_sec":0,"write_op_per_sec":283},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:24.912 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:24.913 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:25.235 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 128 20 70972 70952 4.32953 2.39844 0.0659222 0.0360766 2024-09-18T11:46:25.474 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:25.474 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:25.495 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-18T11:46:23.864758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569009524,"num_objects":69464,"num_object_clones":0,"num_object_copies":277856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69491,"num_write_kb":555810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1431,"num_bytes_recovered":11501568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69518,"ondisk_log_size":69518,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2196336,"kb_used_data":980560,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":375291024,"statfs":{"total":386547056640,"available":384298008576,"internally_reserved":0,"allocated":1004093440,"data_stored":1003457528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,4,11,23,28,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27844608,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":27192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005601"},"pg_stats":[{"pgid":"1.f","version":"31'4420","reported_seq":4460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.685345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:19.685345+0000","last_peered":"2024-09-18T11:46:19.685345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:19.685345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36208640,"num_objects":4420,"num_object_clones":0,"num_object_copies":17680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4420,"num_write_kb":35360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4397","reported_seq":4460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.253108+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:21.253108+0000","last_peered":"2024-09-18T11:46:21.253108+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:21.253108+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35984630,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4395,"num_write_kb":35156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2291}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4313","reported_seq":4393,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.047214+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:21.047214+0000","last_peered":"2024-09-18T11:46:21.047214+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:21.047214+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35306107,"num_objects":4310,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4312,"num_write_kb":34486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4381","reported_seq":4443,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.112192+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:21.112192+0000","last_peered":"2024-09-18T11:46:21.112192+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:21.112192+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853558,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4379,"num_write_kb":35021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4298","reported_seq":4353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.552411+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:19.552411+0000","last_peered":"2024-09-18T11:46:19.552411+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:19.552411+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35181814,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4296,"num_write_kb":34365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4281","reported_seq":4323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.628098+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:19.628098+0000","last_peered":"2024-09-18T11:46:19.628098+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:19.628098+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35069952,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4281,"num_write_kb":34248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4401","reported_seq":4465,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.272942+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:21.272942+0000","last_peered":"2024-09-18T11:46:21.272942+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:21.272942+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36001014,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4399,"num_write_kb":35172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2261}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4299","reported_seq":4362,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.257672+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:21.257672+0000","last_peered":"2024-09-18T11:46:21.257672+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:21.257672+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35173622,"num_objects":4294,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4297,"num_write_kb":34364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4440","reported_seq":4498,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.630424+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:19.630424+0000","last_peered":"2024-09-18T11:46:19.630424+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:19.630424+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36301292,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2297}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4250","reported_seq":4334,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.125813+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:21.125813+0000","last_peered":"2024-09-18T11:46:21.125813+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:21.125813+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34796790,"num_objects":4248,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4248,"num_write_kb":33982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2273}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4332","reported_seq":4389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.610277+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:19.610277+0000","last_peered":"2024-09-18T11:46:19.610277+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:19.610277+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35442545,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4329,"num_write_kb":34620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4397","reported_seq":4433,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.623725+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:19.623725+0000","last_peered":"2024-09-18T11:46:19.623725+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:19.623725+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36010619,"num_objects":4396,"num_object_clones":0,"num_object_copies":17584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":35167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2287}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4390","reported_seq":4466,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.243347+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:21.243347+0000","last_peered":"2024-09-18T11:46:21.243347+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:21.243347+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35953275,"num_objects":4389,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":35111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4359","reported_seq":4408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:19.641803+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:19.641803+0000","last_peered":"2024-09-18T11:46:19.641803+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:19.641803+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4359,"log_dups_size":0,"ondisk_log_size":4359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35673334,"num_objects":4355,"num_object_clones":0,"num_object_copies":17420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4357,"num_write_kb":34852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2197}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4312","reported_seq":7202,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.260338+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:21.260338+0000","last_peered":"2024-09-18T11:46:21.260338+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:21.260338+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35296502,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":34477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1406,"num_bytes_recovered":11501568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3622}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4248","reported_seq":4310,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:21.172348+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:21.172348+0000","last_peered":"2024-09-18T11:46:21.172348+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:21.172348+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34755830,"num_objects":4243,"num_object_clones":0,"num_object_copies":16972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4246,"num_write_kb":33956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":569009524,"num_objects":69464,"num_object_clones":0,"num_object_copies":277856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69491,"num_write_kb":555810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1431,"num_bytes_recovered":11501568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1006141440,"data_stored":1006141440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69518,"ondisk_log_size":69518,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585980,"kb_used_data":275636,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93785860,"statfs":{"total":96636764160,"available":96036720640,"internally_reserved":0,"allocated":282251264,"data_stored":282092286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,4,9,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":1.9099999999999999},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.944}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.2410000000000001},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":2.4889999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.0429999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579124,"kb_used_data":277612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93792716,"statfs":{"total":96636764160,"available":96043741184,"internally_reserved":0,"allocated":284274688,"data_stored":284115710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":8.5830000000000002},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":13.119999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":8.0470000000000006},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":13.194000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":13.986000000000001},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":14.442}]}]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450516,"kb_used_data":148556,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93921324,"statfs":{"total":96636764160,"available":96175435776,"internally_reserved":0,"allocated":152121344,"data_stored":151962366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,9,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":5.6449999999999996},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":5.0839999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":1.8839999999999999},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":5.6059999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":5.1180000000000003},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":11.303000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580716,"kb_used_data":278756,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93791124,"statfs":{"total":96636764160,"available":96042110976,"internally_reserved":0,"allocated":285446144,"data_stored":285287166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,4,14,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":5.0149999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.9850000000000003},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":7.4749999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":1.833},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":5.0439999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285446144,"data_stored":285446144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152080384,"data_stored":152080384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":285396992,"data_stored":285396992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283217920,"data_stored":283217920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:26.236 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 129 20 71302 71282 4.31595 2.57812 0.0442328 0.036184 2024-09-18T11:46:27.236 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 130 20 71653 71633 4.30384 2.74219 0.0680435 0.0362923 2024-09-18T11:46:28.236 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 131 20 72003 71983 4.29186 2.73438 0.0628878 0.0363926 2024-09-18T11:46:28.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:29.026 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:29.026 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:29.050 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-18T11:46:27.867791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":582509940,"num_objects":71112,"num_object_clones":0,"num_object_copies":284448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71139,"num_write_kb":568994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71166,"ondisk_log_size":71166,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2227684,"kb_used_data":1007748,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375259676,"statfs":{"total":386547056640,"available":384265908224,"internally_reserved":0,"allocated":1031933952,"data_stored":1031298040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,11,3,21,28,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35078144,"num_objects":4282,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":34256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":1990656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006046"},"pg_stats":[{"pgid":"1.f","version":"31'4522","reported_seq":4563,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.691257+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:24.691257+0000","last_peered":"2024-09-18T11:46:24.691257+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:24.691257+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37044224,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4493","reported_seq":4556,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.148481+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:26.148481+0000","last_peered":"2024-09-18T11:46:26.148481+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:26.148481+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36771062,"num_objects":4489,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2387}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4407","reported_seq":4489,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.117712+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:26.117712+0000","last_peered":"2024-09-18T11:46:26.117712+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:26.117712+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36076155,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":35238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2350}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4466","reported_seq":4528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.147647+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:26.147647+0000","last_peered":"2024-09-18T11:46:26.147647+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:26.147647+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36549878,"num_objects":4462,"num_object_clones":0,"num_object_copies":17848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":35701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4393","reported_seq":4446,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.736386+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:24.736386+0000","last_peered":"2024-09-18T11:46:24.736386+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:24.736386+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35960054,"num_objects":4390,"num_object_clones":0,"num_object_copies":17560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":35125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4388","reported_seq":4431,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.641472+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:24.641472+0000","last_peered":"2024-09-18T11:46:24.641472+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:24.641472+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4388,"log_dups_size":0,"ondisk_log_size":4388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35946496,"num_objects":4388,"num_object_clones":0,"num_object_copies":17552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":35104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4518","reported_seq":4581,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.308134+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:26.308134+0000","last_peered":"2024-09-18T11:46:26.308134+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:26.308134+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36959478,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4516,"num_write_kb":36108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2378}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4390","reported_seq":4452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.164086+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:26.164086+0000","last_peered":"2024-09-18T11:46:26.164086+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:26.164086+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35919094,"num_objects":4385,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":35092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4546","reported_seq":4603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.644306+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:24.644306+0000","last_peered":"2024-09-18T11:46:24.644306+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:24.644306+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37169644,"num_objects":4538,"num_object_clones":0,"num_object_copies":18152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2403}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4337","reported_seq":4421,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.044764+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:26.044764+0000","last_peered":"2024-09-18T11:46:26.044764+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:26.044764+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4337,"log_dups_size":0,"ondisk_log_size":4337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35509494,"num_objects":4335,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":34678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4438","reported_seq":4495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.674025+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:24.674025+0000","last_peered":"2024-09-18T11:46:24.674025+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:24.674025+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36310897,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":35468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2370}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4524","reported_seq":4560,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.738747+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:24.738747+0000","last_peered":"2024-09-18T11:46:24.738747+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:24.738747+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4524,"log_dups_size":0,"ondisk_log_size":4524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37051003,"num_objects":4523,"num_object_clones":0,"num_object_copies":18092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4523,"num_write_kb":36183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2414}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4515","reported_seq":4590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.260779+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:26.260779+0000","last_peered":"2024-09-18T11:46:26.260779+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:26.260779+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4515,"log_dups_size":0,"ondisk_log_size":4515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36977275,"num_objects":4514,"num_object_clones":0,"num_object_copies":18056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4514,"num_write_kb":36111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2460}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4460","reported_seq":4509,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.587707+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:24.587707+0000","last_peered":"2024-09-18T11:46:24.587707+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:24.587707+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4460,"log_dups_size":0,"ondisk_log_size":4460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36500726,"num_objects":4456,"num_object_clones":0,"num_object_copies":17824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4458,"num_write_kb":35660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2298}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4424","reported_seq":7476,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.280607+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:26.280607+0000","last_peered":"2024-09-18T11:46:26.280607+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:26.280607+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4424,"log_dups_size":0,"ondisk_log_size":4424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36214006,"num_objects":4421,"num_object_clones":0,"num_object_copies":17684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":35373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1487,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4345","reported_seq":4407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.190816+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:26.190816+0000","last_peered":"2024-09-18T11:46:26.190816+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:26.190816+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35550454,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":34732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":582509940,"num_objects":71112,"num_object_clones":0,"num_object_copies":284448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71139,"num_write_kb":568994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1033674752,"data_stored":1033674752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71166,"ondisk_log_size":71166,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584452,"kb_used_data":282428,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93787388,"statfs":{"total":96636764160,"available":96038285312,"internally_reserved":0,"allocated":289206272,"data_stored":289047294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,12,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":6.71},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.7380000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.673},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":6.9470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.3419999999999996},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":6.8129999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589944,"kb_used_data":284272,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93781896,"statfs":{"total":96636764160,"available":96032661504,"internally_reserved":0,"allocated":291094528,"data_stored":290935550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,2,2,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":11.992000000000001},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":11.74}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":2.7599999999999998},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":5.4889999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":9.5329999999999995},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.6120000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461748,"kb_used_data":155628,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93910092,"statfs":{"total":96636764160,"available":96163934208,"internally_reserved":0,"allocated":159363072,"data_stored":159204094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":5.3090000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":1.2589999999999999},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":1.2729999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.625},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":1.2190000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591540,"kb_used_data":285420,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93780300,"statfs":{"total":96636764160,"available":96031027200,"internally_reserved":0,"allocated":292270080,"data_stored":292111102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,3,17,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.438},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":9.2669999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":5.2489999999999997},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":2.3969999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292167680,"data_stored":292167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159297536,"data_stored":159297536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292110336,"data_stored":292110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":290099200,"data_stored":290099200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:29.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:29.236 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 132 20 72319 72299 4.27805 2.46875 0.0335913 0.0365097 2024-09-18T11:46:29.636 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:29.636 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:29.656 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-18T11:46:27.867791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":582509940,"num_objects":71112,"num_object_clones":0,"num_object_copies":284448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71139,"num_write_kb":568994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71166,"ondisk_log_size":71166,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2227684,"kb_used_data":1007748,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375259676,"statfs":{"total":386547056640,"available":384265908224,"internally_reserved":0,"allocated":1031933952,"data_stored":1031298040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,11,3,21,28,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35078144,"num_objects":4282,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":34256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":1990656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006046"},"pg_stats":[{"pgid":"1.f","version":"31'4522","reported_seq":4563,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.691257+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:24.691257+0000","last_peered":"2024-09-18T11:46:24.691257+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:24.691257+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37044224,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4493","reported_seq":4556,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.148481+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:26.148481+0000","last_peered":"2024-09-18T11:46:26.148481+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:26.148481+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36771062,"num_objects":4489,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2387}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4407","reported_seq":4489,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.117712+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:26.117712+0000","last_peered":"2024-09-18T11:46:26.117712+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:26.117712+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36076155,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":35238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2350}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4466","reported_seq":4528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.147647+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:26.147647+0000","last_peered":"2024-09-18T11:46:26.147647+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:26.147647+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36549878,"num_objects":4462,"num_object_clones":0,"num_object_copies":17848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":35701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4393","reported_seq":4446,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.736386+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:24.736386+0000","last_peered":"2024-09-18T11:46:24.736386+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:24.736386+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35960054,"num_objects":4390,"num_object_clones":0,"num_object_copies":17560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":35125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4388","reported_seq":4431,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.641472+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:24.641472+0000","last_peered":"2024-09-18T11:46:24.641472+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:24.641472+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4388,"log_dups_size":0,"ondisk_log_size":4388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35946496,"num_objects":4388,"num_object_clones":0,"num_object_copies":17552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":35104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4518","reported_seq":4581,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.308134+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:26.308134+0000","last_peered":"2024-09-18T11:46:26.308134+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:26.308134+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36959478,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4516,"num_write_kb":36108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2378}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4390","reported_seq":4452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.164086+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:26.164086+0000","last_peered":"2024-09-18T11:46:26.164086+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:26.164086+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35919094,"num_objects":4385,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":35092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4546","reported_seq":4603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.644306+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:24.644306+0000","last_peered":"2024-09-18T11:46:24.644306+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:24.644306+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37169644,"num_objects":4538,"num_object_clones":0,"num_object_copies":18152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2403}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4337","reported_seq":4421,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.044764+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:26.044764+0000","last_peered":"2024-09-18T11:46:26.044764+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:26.044764+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4337,"log_dups_size":0,"ondisk_log_size":4337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35509494,"num_objects":4335,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":34678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4438","reported_seq":4495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.674025+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:24.674025+0000","last_peered":"2024-09-18T11:46:24.674025+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:24.674025+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36310897,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":35468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2370}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4524","reported_seq":4560,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.738747+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:24.738747+0000","last_peered":"2024-09-18T11:46:24.738747+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:24.738747+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4524,"log_dups_size":0,"ondisk_log_size":4524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37051003,"num_objects":4523,"num_object_clones":0,"num_object_copies":18092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4523,"num_write_kb":36183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2414}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4515","reported_seq":4590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.260779+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:26.260779+0000","last_peered":"2024-09-18T11:46:26.260779+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:26.260779+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4515,"log_dups_size":0,"ondisk_log_size":4515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36977275,"num_objects":4514,"num_object_clones":0,"num_object_copies":18056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4514,"num_write_kb":36111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2460}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4460","reported_seq":4509,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:24.587707+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:24.587707+0000","last_peered":"2024-09-18T11:46:24.587707+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:24.587707+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4460,"log_dups_size":0,"ondisk_log_size":4460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36500726,"num_objects":4456,"num_object_clones":0,"num_object_copies":17824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4458,"num_write_kb":35660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2298}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4424","reported_seq":7476,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.280607+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:26.280607+0000","last_peered":"2024-09-18T11:46:26.280607+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:26.280607+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4424,"log_dups_size":0,"ondisk_log_size":4424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36214006,"num_objects":4421,"num_object_clones":0,"num_object_copies":17684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":35373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1487,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4345","reported_seq":4407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.190816+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:26.190816+0000","last_peered":"2024-09-18T11:46:26.190816+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:26.190816+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35550454,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":34732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":582509940,"num_objects":71112,"num_object_clones":0,"num_object_copies":284448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71139,"num_write_kb":568994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1033674752,"data_stored":1033674752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71166,"ondisk_log_size":71166,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584452,"kb_used_data":282428,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93787388,"statfs":{"total":96636764160,"available":96038285312,"internally_reserved":0,"allocated":289206272,"data_stored":289047294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,12,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":6.71},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.7380000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.673},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":6.9470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.3419999999999996},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":6.8129999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589944,"kb_used_data":284272,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93781896,"statfs":{"total":96636764160,"available":96032661504,"internally_reserved":0,"allocated":291094528,"data_stored":290935550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,2,2,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":11.992000000000001},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":11.74}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":2.7599999999999998},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":5.4889999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":9.5329999999999995},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.6120000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461748,"kb_used_data":155628,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93910092,"statfs":{"total":96636764160,"available":96163934208,"internally_reserved":0,"allocated":159363072,"data_stored":159204094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":5.3090000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":1.2589999999999999},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":1.2729999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.625},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":1.2190000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591540,"kb_used_data":285420,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93780300,"statfs":{"total":96636764160,"available":96031027200,"internally_reserved":0,"allocated":292270080,"data_stored":292111102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,3,17,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.438},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":9.2669999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":5.2489999999999997},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":2.3969999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292167680,"data_stored":292167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159297536,"data_stored":159297536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292110336,"data_stored":292110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":290099200,"data_stored":290099200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:29.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:29.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4522", 'reported_seq': 4563, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:24.691257+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:24.691257+0000', 'last_peered': '2024-09-18T11:46:24.691257+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:24.691257+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4522, 'log_dups_size': 0, 'ondisk_log_size': 4522, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37044224, 'num_objects': 4522, 'num_object_clones': 0, 'num_object_copies': 18088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4522, 'num_write_kb': 36176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2392}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:29.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:29.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4493", 'reported_seq': 4556, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:26.148481+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:26.148481+0000', 'last_peered': '2024-09-18T11:46:26.148481+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:26.148481+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4493, 'log_dups_size': 0, 'ondisk_log_size': 4493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36771062, 'num_objects': 4489, 'num_object_clones': 0, 'num_object_copies': 17956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4491, 'num_write_kb': 35924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2387}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:29.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:29.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4407", 'reported_seq': 4489, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:26.117712+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:26.117712+0000', 'last_peered': '2024-09-18T11:46:26.117712+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:26.117712+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4407, 'log_dups_size': 0, 'ondisk_log_size': 4407, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36076155, 'num_objects': 4404, 'num_object_clones': 0, 'num_object_copies': 17616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4406, 'num_write_kb': 35238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2350}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:29.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:29.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4466", 'reported_seq': 4528, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:26.147647+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:26.147647+0000', 'last_peered': '2024-09-18T11:46:26.147647+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:26.147647+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4466, 'log_dups_size': 0, 'ondisk_log_size': 4466, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36549878, 'num_objects': 4462, 'num_object_clones': 0, 'num_object_copies': 17848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4464, 'num_write_kb': 35701, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:29.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:29.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4393", 'reported_seq': 4446, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:24.736386+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:24.736386+0000', 'last_peered': '2024-09-18T11:46:24.736386+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:24.736386+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4393, 'log_dups_size': 0, 'ondisk_log_size': 4393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35960054, 'num_objects': 4390, 'num_object_clones': 0, 'num_object_copies': 17560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4391, 'num_write_kb': 35125, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2308}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:29.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:29.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4388", 'reported_seq': 4431, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:24.641472+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:24.641472+0000', 'last_peered': '2024-09-18T11:46:24.641472+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:24.641472+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4388, 'log_dups_size': 0, 'ondisk_log_size': 4388, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35946496, 'num_objects': 4388, 'num_object_clones': 0, 'num_object_copies': 17552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4388, 'num_write_kb': 35104, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2234}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4518", 'reported_seq': 4581, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:26.308134+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:26.308134+0000', 'last_peered': '2024-09-18T11:46:26.308134+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:26.308134+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4518, 'log_dups_size': 0, 'ondisk_log_size': 4518, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36959478, 'num_objects': 4512, 'num_object_clones': 0, 'num_object_copies': 18048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4516, 'num_write_kb': 36108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2378}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4390", 'reported_seq': 4452, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:26.164086+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:26.164086+0000', 'last_peered': '2024-09-18T11:46:26.164086+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:26.164086+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4390, 'log_dups_size': 0, 'ondisk_log_size': 4390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35919094, 'num_objects': 4385, 'num_object_clones': 0, 'num_object_copies': 17540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4388, 'num_write_kb': 35092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2351}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4546", 'reported_seq': 4603, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:24.644306+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:24.644306+0000', 'last_peered': '2024-09-18T11:46:24.644306+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:24.644306+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4546, 'log_dups_size': 0, 'ondisk_log_size': 4546, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37169644, 'num_objects': 4538, 'num_object_clones': 0, 'num_object_copies': 18152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4543, 'num_write_kb': 36321, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2403}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4337", 'reported_seq': 4421, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:26.044764+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:26.044764+0000', 'last_peered': '2024-09-18T11:46:26.044764+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:26.044764+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4337, 'log_dups_size': 0, 'ondisk_log_size': 4337, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35509494, 'num_objects': 4335, 'num_object_clones': 0, 'num_object_copies': 17340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4335, 'num_write_kb': 34678, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2360}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4438", 'reported_seq': 4495, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:24.674025+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:24.674025+0000', 'last_peered': '2024-09-18T11:46:24.674025+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:24.674025+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4438, 'log_dups_size': 0, 'ondisk_log_size': 4438, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36310897, 'num_objects': 4433, 'num_object_clones': 0, 'num_object_copies': 17732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4435, 'num_write_kb': 35468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2370}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:29.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:29.660 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4524", 'reported_seq': 4560, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:24.738747+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:24.738747+0000', 'last_peered': '2024-09-18T11:46:24.738747+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:24.738747+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4524, 'log_dups_size': 0, 'ondisk_log_size': 4524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37051003, 'num_objects': 4523, 'num_object_clones': 0, 'num_object_copies': 18092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4523, 'num_write_kb': 36183, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2414}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:29.660 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:29.660 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4515", 'reported_seq': 4590, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:26.260779+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:26.260779+0000', 'last_peered': '2024-09-18T11:46:26.260779+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:26.260779+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4515, 'log_dups_size': 0, 'ondisk_log_size': 4515, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36977275, 'num_objects': 4514, 'num_object_clones': 0, 'num_object_copies': 18056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4514, 'num_write_kb': 36111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2460}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:29.660 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:29.660 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4460", 'reported_seq': 4509, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:24.587707+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:24.587707+0000', 'last_peered': '2024-09-18T11:46:24.587707+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:24.587707+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4460, 'log_dups_size': 0, 'ondisk_log_size': 4460, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36500726, 'num_objects': 4456, 'num_object_clones': 0, 'num_object_copies': 17824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4458, 'num_write_kb': 35660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2298}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:29.660 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:29.660 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4424", 'reported_seq': 7476, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:26.280607+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:26.280607+0000', 'last_peered': '2024-09-18T11:46:26.280607+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:26.280607+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4424, 'log_dups_size': 0, 'ondisk_log_size': 4424, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36214006, 'num_objects': 4421, 'num_object_clones': 0, 'num_object_copies': 17684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 608, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4422, 'num_write_kb': 35373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1487, 'num_bytes_recovered': 12148736, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:29.660 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:29.660 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4345", 'reported_seq': 4407, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:26.190816+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:26.190816+0000', 'last_peered': '2024-09-18T11:46:26.190816+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:26.190816+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4345, 'log_dups_size': 0, 'ondisk_log_size': 4345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35550454, 'num_objects': 4340, 'num_object_clones': 0, 'num_object_copies': 17360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4343, 'num_write_kb': 34732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2327}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:29.661 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:30.238 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 133 20 72620 72600 4.26357 2.35156 0.0486018 0.0366307 2024-09-18T11:46:30.300 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:30.300 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31944/287448 objects degraded (11.113%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":188,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71862,"data_bytes":588653940,"bytes_used":2304696320,"bytes_avail":384242360320,"bytes_total":386547056640,"degraded_objects":31944,"degraded_total":287448,"degraded_ratio":0.11112966519161727,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":109851,"recovering_keys_per_sec":0,"num_objects_recovered":163,"num_bytes_recovered":1318912,"num_keys_recovered":0,"write_bytes_sec":2775636,"read_op_per_sec":0,"write_op_per_sec":338},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:30.331 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:30.331 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:30.871 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:30.871 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:30.902 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-18T11:46:29.868757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":588653940,"num_objects":71862,"num_object_clones":0,"num_object_copies":287448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71889,"num_write_kb":574994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71916,"ondisk_log_size":71916,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2250680,"kb_used_data":1022424,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375236680,"statfs":{"total":386547056640,"available":384242360320,"internally_reserved":0,"allocated":1046962176,"data_stored":1046326264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,8,9,19,28,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33325056,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4068,"num_write_kb":32544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":1318912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006277"},"pg_stats":[{"pgid":"1.f","version":"31'4625","reported_seq":4666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.624703+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:29.624703+0000","last_peered":"2024-09-18T11:46:29.624703+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:29.624703+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37888000,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":37000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4493","reported_seq":4556,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.148481+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:26.148481+0000","last_peered":"2024-09-18T11:46:26.148481+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:26.148481+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36771062,"num_objects":4489,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2387}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4407","reported_seq":4489,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.117712+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:26.117712+0000","last_peered":"2024-09-18T11:46:26.117712+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:26.117712+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36076155,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":35238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2350}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4466","reported_seq":4528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.147647+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:26.147647+0000","last_peered":"2024-09-18T11:46:26.147647+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:26.147647+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36549878,"num_objects":4462,"num_object_clones":0,"num_object_copies":17848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":35701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4506","reported_seq":4559,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.738450+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:29.738450+0000","last_peered":"2024-09-18T11:46:29.738450+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:29.738450+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4506,"log_dups_size":0,"ondisk_log_size":4506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885750,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4504,"num_write_kb":36029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4515","reported_seq":4556,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.741558+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:29.741558+0000","last_peered":"2024-09-18T11:46:29.741558+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:29.741558+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4515,"log_dups_size":0,"ondisk_log_size":4515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36986880,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4515,"num_write_kb":36120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4518","reported_seq":4581,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.308134+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:26.308134+0000","last_peered":"2024-09-18T11:46:26.308134+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:26.308134+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36959478,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4516,"num_write_kb":36108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2378}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4390","reported_seq":4452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.164086+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:26.164086+0000","last_peered":"2024-09-18T11:46:26.164086+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:26.164086+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35919094,"num_objects":4385,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":35092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4652","reported_seq":4710,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.738956+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:29.738956+0000","last_peered":"2024-09-18T11:46:29.738956+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:29.738956+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38037996,"num_objects":4644,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":37169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4337","reported_seq":4421,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.044764+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:26.044764+0000","last_peered":"2024-09-18T11:46:26.044764+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:26.044764+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4337,"log_dups_size":0,"ondisk_log_size":4337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35509494,"num_objects":4335,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":34678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4546","reported_seq":4603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.689323+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:29.689323+0000","last_peered":"2024-09-18T11:46:29.689323+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:29.689323+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37195633,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4632","reported_seq":4669,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.722296+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:29.722296+0000","last_peered":"2024-09-18T11:46:29.722296+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:29.722296+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37935739,"num_objects":4631,"num_object_clones":0,"num_object_copies":18524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4631,"num_write_kb":37047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4515","reported_seq":4590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.260779+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:26.260779+0000","last_peered":"2024-09-18T11:46:26.260779+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:26.260779+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4515,"log_dups_size":0,"ondisk_log_size":4515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36977275,"num_objects":4514,"num_object_clones":0,"num_object_copies":18056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4514,"num_write_kb":36111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2460}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4545","reported_seq":4594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.735396+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:29.735396+0000","last_peered":"2024-09-18T11:46:29.735396+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:29.735396+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4545,"log_dups_size":0,"ondisk_log_size":4545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37197046,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2383}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4424","reported_seq":7476,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.280607+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:26.280607+0000","last_peered":"2024-09-18T11:46:26.280607+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:26.280607+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4424,"log_dups_size":0,"ondisk_log_size":4424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36214006,"num_objects":4421,"num_object_clones":0,"num_object_copies":17684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":35373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1487,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4345","reported_seq":4407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:26.190816+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:26.190816+0000","last_peered":"2024-09-18T11:46:26.190816+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:26.190816+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35550454,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":34732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":588653940,"num_objects":71862,"num_object_clones":0,"num_object_copies":287448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71889,"num_write_kb":574994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1047691264,"data_stored":1047691264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71916,"ondisk_log_size":71916,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596256,"kb_used_data":290072,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93775584,"statfs":{"total":96636764160,"available":96026198016,"internally_reserved":0,"allocated":297033728,"data_stored":296874750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,12,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":6.71},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.7380000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.673},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":6.9470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.3419999999999996},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":6.8129999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589944,"kb_used_data":284272,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93781896,"statfs":{"total":96636764160,"available":96032661504,"internally_reserved":0,"allocated":291094528,"data_stored":290935550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,2,2,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":11.992000000000001},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":11.74}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":2.7599999999999998},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":5.4889999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":9.5329999999999995},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.6120000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472940,"kb_used_data":162660,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93898900,"statfs":{"total":96636764160,"available":96152473600,"internally_reserved":0,"allocated":166563840,"data_stored":166404862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,7,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":2.282},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":8.2270000000000003},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":8.5229999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.31900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591540,"kb_used_data":285420,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93780300,"statfs":{"total":96636764160,"available":96031027200,"internally_reserved":0,"allocated":292270080,"data_stored":292111102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,3,17,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.438},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":9.2669999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":5.2489999999999997},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":2.3969999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292167680,"data_stored":292167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":166526976,"data_stored":166526976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292110336,"data_stored":292110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":296886272,"data_stored":296886272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:31.237 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 134 20 72941 72921 4.25044 2.50781 0.0409278 0.0367478 2024-09-18T11:46:32.237 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 135 20 73278 73258 4.23846 2.63281 0.0488171 0.0368502 2024-09-18T11:46:33.237 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 136 20 73631 73611 4.22757 2.75781 0.0400833 0.0369462 2024-09-18T11:46:33.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:34.237 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 137 20 73961 73941 4.21553 2.57812 0.0458332 0.0370516 2024-09-18T11:46:34.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:34.459 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:34.496 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-18T11:46:33.870360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596059508,"num_objects":72766,"num_object_clones":0,"num_object_copies":291064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72793,"num_write_kb":582226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1599,"num_bytes_recovered":12861440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72820,"ondisk_log_size":72820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2272168,"kb_used_data":1035592,"kb_used_omap":14,"kb_used_meta":1236529,"kb_avail":375215192,"statfs":{"total":386547056640,"available":384220356608,"internally_reserved":0,"allocated":1060446208,"data_stored":1059810296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1266205723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,3,8,19,22,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27049984,"num_objects":3302,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3302,"num_write_kb":26416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006534"},"pg_stats":[{"pgid":"1.f","version":"31'4625","reported_seq":4666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.624703+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:29.624703+0000","last_peered":"2024-09-18T11:46:29.624703+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:29.624703+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37888000,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":37000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4593","reported_seq":4656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.296017+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:31.296017+0000","last_peered":"2024-09-18T11:46:31.296017+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:31.296017+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590262,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":36724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4497","reported_seq":4579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.059418+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:31.059418+0000","last_peered":"2024-09-18T11:46:31.059418+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:31.059418+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36813435,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2440}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4564","reported_seq":4626,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.301153+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:31.301153+0000","last_peered":"2024-09-18T11:46:31.301153+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:31.301153+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37352694,"num_objects":4560,"num_object_clones":0,"num_object_copies":18240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":36485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4506","reported_seq":4559,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.738450+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:29.738450+0000","last_peered":"2024-09-18T11:46:29.738450+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:29.738450+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4506,"log_dups_size":0,"ondisk_log_size":4506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885750,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4504,"num_write_kb":36029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4515","reported_seq":4556,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.741558+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:29.741558+0000","last_peered":"2024-09-18T11:46:29.741558+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:29.741558+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4515,"log_dups_size":0,"ondisk_log_size":4515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36986880,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4515,"num_write_kb":36120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4627","reported_seq":4692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.288013+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:31.288013+0000","last_peered":"2024-09-18T11:46:31.288013+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:31.288013+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37852406,"num_objects":4621,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":36980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2487}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4488","reported_seq":4551,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.154477+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:31.154477+0000","last_peered":"2024-09-18T11:46:31.154477+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:31.154477+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36721910,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4486,"num_write_kb":35876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4652","reported_seq":4710,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.738956+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:29.738956+0000","last_peered":"2024-09-18T11:46:29.738956+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:29.738956+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38037996,"num_objects":4644,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":37169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4443","reported_seq":4526,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.283865+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:31.283865+0000","last_peered":"2024-09-18T11:46:31.283865+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:31.283865+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4443,"log_dups_size":0,"ondisk_log_size":4443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377846,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4441,"num_write_kb":35526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2466}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4546","reported_seq":4603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.689323+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:29.689323+0000","last_peered":"2024-09-18T11:46:29.689323+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:29.689323+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37195633,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4632","reported_seq":4669,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.722296+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:29.722296+0000","last_peered":"2024-09-18T11:46:29.722296+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:29.722296+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37935739,"num_objects":4631,"num_object_clones":0,"num_object_copies":18524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4631,"num_write_kb":37047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4623","reported_seq":4699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.257220+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:31.257220+0000","last_peered":"2024-09-18T11:46:31.257220+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:31.257220+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37862011,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4622,"num_write_kb":36975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2568}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4545","reported_seq":4594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.735396+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:29.735396+0000","last_peered":"2024-09-18T11:46:29.735396+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:29.735396+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4545,"log_dups_size":0,"ondisk_log_size":4545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37197046,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2383}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4516","reported_seq":7743,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.288088+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:31.288088+0000","last_peered":"2024-09-18T11:46:31.288088+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:31.288088+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4516,"log_dups_size":0,"ondisk_log_size":4516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967670,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4514,"num_write_kb":36109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1574,"num_bytes_recovered":12861440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4448","reported_seq":4510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.250099+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:31.250099+0000","last_peered":"2024-09-18T11:46:31.250099+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:31.250099+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36394230,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4446,"num_write_kb":35556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2430}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":596059508,"num_objects":72766,"num_object_clones":0,"num_object_copies":291064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72793,"num_write_kb":582226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1599,"num_bytes_recovered":12861440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1061134336,"data_stored":1061134336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72820,"ondisk_log_size":72820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596256,"kb_used_data":290072,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93775584,"statfs":{"total":96636764160,"available":96026198016,"internally_reserved":0,"allocated":297033728,"data_stored":296874750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,12,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":6.71},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.7380000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.673},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":6.9470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.3419999999999996},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":6.8129999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600752,"kb_used_data":290920,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93771088,"statfs":{"total":96636764160,"available":96021594112,"internally_reserved":0,"allocated":297902080,"data_stored":297743102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.091},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.313}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":0.439}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":0.70299999999999996},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.97799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472940,"kb_used_data":162660,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93898900,"statfs":{"total":96636764160,"available":96152473600,"internally_reserved":0,"allocated":166563840,"data_stored":166404862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,7,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":2.282},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":8.2270000000000003},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":8.5229999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.31900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602220,"kb_used_data":291940,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93769620,"statfs":{"total":96636764160,"available":96020090880,"internally_reserved":0,"allocated":298946560,"data_stored":298787582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,3,17,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.438},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":9.2669999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":5.2489999999999997},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":2.3969999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298893312,"data_stored":298893312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":166526976,"data_stored":166526976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298827776,"data_stored":298827776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":296886272,"data_stored":296886272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:34.498 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:35.010 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:35.011 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:35.031 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-18T11:46:33.870360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596059508,"num_objects":72766,"num_object_clones":0,"num_object_copies":291064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72793,"num_write_kb":582226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1599,"num_bytes_recovered":12861440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72820,"ondisk_log_size":72820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2272168,"kb_used_data":1035592,"kb_used_omap":14,"kb_used_meta":1236529,"kb_avail":375215192,"statfs":{"total":386547056640,"available":384220356608,"internally_reserved":0,"allocated":1060446208,"data_stored":1059810296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1266205723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,3,8,19,22,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27049984,"num_objects":3302,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3302,"num_write_kb":26416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006534"},"pg_stats":[{"pgid":"1.f","version":"31'4625","reported_seq":4666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.624703+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:29.624703+0000","last_peered":"2024-09-18T11:46:29.624703+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:29.624703+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37888000,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":37000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4593","reported_seq":4656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.296017+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:31.296017+0000","last_peered":"2024-09-18T11:46:31.296017+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:31.296017+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590262,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":36724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4497","reported_seq":4579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.059418+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:31.059418+0000","last_peered":"2024-09-18T11:46:31.059418+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:31.059418+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36813435,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2440}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4564","reported_seq":4626,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.301153+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:31.301153+0000","last_peered":"2024-09-18T11:46:31.301153+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:31.301153+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37352694,"num_objects":4560,"num_object_clones":0,"num_object_copies":18240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":36485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4506","reported_seq":4559,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.738450+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:29.738450+0000","last_peered":"2024-09-18T11:46:29.738450+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:29.738450+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4506,"log_dups_size":0,"ondisk_log_size":4506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885750,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4504,"num_write_kb":36029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4515","reported_seq":4556,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.741558+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:29.741558+0000","last_peered":"2024-09-18T11:46:29.741558+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:29.741558+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4515,"log_dups_size":0,"ondisk_log_size":4515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36986880,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4515,"num_write_kb":36120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4627","reported_seq":4692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.288013+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:31.288013+0000","last_peered":"2024-09-18T11:46:31.288013+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:31.288013+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37852406,"num_objects":4621,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":36980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2487}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4488","reported_seq":4551,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.154477+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:31.154477+0000","last_peered":"2024-09-18T11:46:31.154477+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:31.154477+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36721910,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4486,"num_write_kb":35876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4652","reported_seq":4710,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.738956+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:29.738956+0000","last_peered":"2024-09-18T11:46:29.738956+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:29.738956+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38037996,"num_objects":4644,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":37169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4443","reported_seq":4526,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.283865+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:31.283865+0000","last_peered":"2024-09-18T11:46:31.283865+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:31.283865+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4443,"log_dups_size":0,"ondisk_log_size":4443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377846,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4441,"num_write_kb":35526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2466}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4546","reported_seq":4603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.689323+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:29.689323+0000","last_peered":"2024-09-18T11:46:29.689323+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:29.689323+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37195633,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4632","reported_seq":4669,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.722296+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:29.722296+0000","last_peered":"2024-09-18T11:46:29.722296+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:29.722296+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37935739,"num_objects":4631,"num_object_clones":0,"num_object_copies":18524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4631,"num_write_kb":37047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4623","reported_seq":4699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.257220+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:31.257220+0000","last_peered":"2024-09-18T11:46:31.257220+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:31.257220+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37862011,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4622,"num_write_kb":36975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2568}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4545","reported_seq":4594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:29.735396+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:29.735396+0000","last_peered":"2024-09-18T11:46:29.735396+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:29.735396+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4545,"log_dups_size":0,"ondisk_log_size":4545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37197046,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2383}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4516","reported_seq":7743,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.288088+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:31.288088+0000","last_peered":"2024-09-18T11:46:31.288088+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:31.288088+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4516,"log_dups_size":0,"ondisk_log_size":4516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967670,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4514,"num_write_kb":36109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1574,"num_bytes_recovered":12861440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4448","reported_seq":4510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.250099+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:31.250099+0000","last_peered":"2024-09-18T11:46:31.250099+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:31.250099+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36394230,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4446,"num_write_kb":35556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2430}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":596059508,"num_objects":72766,"num_object_clones":0,"num_object_copies":291064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72793,"num_write_kb":582226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1599,"num_bytes_recovered":12861440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1061134336,"data_stored":1061134336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72820,"ondisk_log_size":72820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596256,"kb_used_data":290072,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93775584,"statfs":{"total":96636764160,"available":96026198016,"internally_reserved":0,"allocated":297033728,"data_stored":296874750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,12,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":6.71},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.7380000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":2.673},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":6.9470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.3419999999999996},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":6.8129999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600752,"kb_used_data":290920,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93771088,"statfs":{"total":96636764160,"available":96021594112,"internally_reserved":0,"allocated":297902080,"data_stored":297743102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.091},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.313}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":0.439}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":0.70299999999999996},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.97799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472940,"kb_used_data":162660,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93898900,"statfs":{"total":96636764160,"available":96152473600,"internally_reserved":0,"allocated":166563840,"data_stored":166404862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,7,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":2.282},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":8.2270000000000003},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":8.5229999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.31900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602220,"kb_used_data":291940,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93769620,"statfs":{"total":96636764160,"available":96020090880,"internally_reserved":0,"allocated":298946560,"data_stored":298787582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,3,17,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.438},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":9.2669999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":5.2489999999999997},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":2.3969999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298893312,"data_stored":298893312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":166526976,"data_stored":166526976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298827776,"data_stored":298827776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":296886272,"data_stored":296886272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:35.032 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:35.032 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4625", 'reported_seq': 4666, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:29.624703+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:29.624703+0000', 'last_peered': '2024-09-18T11:46:29.624703+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:29.624703+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4625, 'log_dups_size': 0, 'ondisk_log_size': 4625, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37888000, 'num_objects': 4625, 'num_object_clones': 0, 'num_object_copies': 18500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4625, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4625, 'num_write_kb': 37000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2495}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:35.032 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:35.032 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4593", 'reported_seq': 4656, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:31.296017+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:31.296017+0000', 'last_peered': '2024-09-18T11:46:31.296017+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:31.296017+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37590262, 'num_objects': 4589, 'num_object_clones': 0, 'num_object_copies': 18356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4591, 'num_write_kb': 36724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2487}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:35.032 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4497", 'reported_seq': 4579, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:31.059418+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:31.059418+0000', 'last_peered': '2024-09-18T11:46:31.059418+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:31.059418+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4497, 'log_dups_size': 0, 'ondisk_log_size': 4497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36813435, 'num_objects': 4494, 'num_object_clones': 0, 'num_object_copies': 17976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4496, 'num_write_kb': 35958, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2440}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4564", 'reported_seq': 4626, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:31.301153+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:31.301153+0000', 'last_peered': '2024-09-18T11:46:31.301153+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:31.301153+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4564, 'log_dups_size': 0, 'ondisk_log_size': 4564, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37352694, 'num_objects': 4560, 'num_object_clones': 0, 'num_object_copies': 18240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4562, 'num_write_kb': 36485, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2446}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4506", 'reported_seq': 4559, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:29.738450+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:29.738450+0000', 'last_peered': '2024-09-18T11:46:29.738450+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:29.738450+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4506, 'log_dups_size': 0, 'ondisk_log_size': 4506, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36885750, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 18012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4504, 'num_write_kb': 36029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2421}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4515", 'reported_seq': 4556, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:29.741558+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:29.741558+0000', 'last_peered': '2024-09-18T11:46:29.741558+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:29.741558+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4515, 'log_dups_size': 0, 'ondisk_log_size': 4515, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36986880, 'num_objects': 4515, 'num_object_clones': 0, 'num_object_copies': 18060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4515, 'num_write_kb': 36120, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2361}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4627", 'reported_seq': 4692, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:31.288013+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:31.288013+0000', 'last_peered': '2024-09-18T11:46:31.288013+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:31.288013+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4627, 'log_dups_size': 0, 'ondisk_log_size': 4627, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37852406, 'num_objects': 4621, 'num_object_clones': 0, 'num_object_copies': 18484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4625, 'num_write_kb': 36980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2487}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:35.033 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4488", 'reported_seq': 4551, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:31.154477+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:31.154477+0000', 'last_peered': '2024-09-18T11:46:31.154477+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:31.154477+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4488, 'log_dups_size': 0, 'ondisk_log_size': 4488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36721910, 'num_objects': 4483, 'num_object_clones': 0, 'num_object_copies': 17932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4486, 'num_write_kb': 35876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:35.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:35.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4652", 'reported_seq': 4710, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:29.738956+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:29.738956+0000', 'last_peered': '2024-09-18T11:46:29.738956+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:29.738956+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4652, 'log_dups_size': 0, 'ondisk_log_size': 4652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38037996, 'num_objects': 4644, 'num_object_clones': 0, 'num_object_copies': 18576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4649, 'num_write_kb': 37169, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2509}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:35.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:35.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4443", 'reported_seq': 4526, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:31.283865+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:31.283865+0000', 'last_peered': '2024-09-18T11:46:31.283865+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:31.283865+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4443, 'log_dups_size': 0, 'ondisk_log_size': 4443, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36377846, 'num_objects': 4441, 'num_object_clones': 0, 'num_object_copies': 17764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4441, 'num_write_kb': 35526, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2466}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:35.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:35.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4546", 'reported_seq': 4603, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:29.689323+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:29.689323+0000', 'last_peered': '2024-09-18T11:46:29.689323+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:29.689323+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4546, 'log_dups_size': 0, 'ondisk_log_size': 4546, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37195633, 'num_objects': 4541, 'num_object_clones': 0, 'num_object_copies': 18164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4543, 'num_write_kb': 36332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2478}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:35.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:35.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4632", 'reported_seq': 4669, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:29.722296+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:29.722296+0000', 'last_peered': '2024-09-18T11:46:29.722296+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:29.722296+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4632, 'log_dups_size': 0, 'ondisk_log_size': 4632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37935739, 'num_objects': 4631, 'num_object_clones': 0, 'num_object_copies': 18524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4631, 'num_write_kb': 37047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:35.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:35.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4623", 'reported_seq': 4699, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:31.257220+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:31.257220+0000', 'last_peered': '2024-09-18T11:46:31.257220+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:31.257220+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4623, 'log_dups_size': 0, 'ondisk_log_size': 4623, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37862011, 'num_objects': 4622, 'num_object_clones': 0, 'num_object_copies': 18488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4622, 'num_write_kb': 36975, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2568}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:35.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:35.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4545", 'reported_seq': 4594, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:29.735396+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:29.735396+0000', 'last_peered': '2024-09-18T11:46:29.735396+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:29.735396+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4545, 'log_dups_size': 0, 'ondisk_log_size': 4545, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37197046, 'num_objects': 4541, 'num_object_clones': 0, 'num_object_copies': 18164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4543, 'num_write_kb': 36340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:35.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:35.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4516", 'reported_seq': 7743, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:31.288088+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:31.288088+0000', 'last_peered': '2024-09-18T11:46:31.288088+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:31.288088+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4516, 'log_dups_size': 0, 'ondisk_log_size': 4516, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36967670, 'num_objects': 4513, 'num_object_clones': 0, 'num_object_copies': 18052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 521, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4514, 'num_write_kb': 36109, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1574, 'num_bytes_recovered': 12861440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3992}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:35.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:35.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4448", 'reported_seq': 4510, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:31.250099+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:31.250099+0000', 'last_peered': '2024-09-18T11:46:31.250099+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:31.250099+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4448, 'log_dups_size': 0, 'ondisk_log_size': 4448, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36394230, 'num_objects': 4443, 'num_object_clones': 0, 'num_object_copies': 17772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4446, 'num_write_kb': 35556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2430}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:35.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:35.237 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 138 20 74261 74241 4.20197 2.34375 0.0524841 0.0371705 2024-09-18T11:46:35.705 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:35.706 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31857/291064 objects degraded (10.945%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":193,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72766,"data_bytes":596059508,"bytes_used":2326700032,"bytes_avail":384220356608,"bytes_total":386547056640,"degraded_objects":31857,"degraded_total":291064,"degraded_ratio":0.10945015529230685,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":113260,"recovering_keys_per_sec":0,"num_objects_recovered":168,"num_bytes_recovered":1359872,"num_keys_recovered":0,"write_bytes_sec":2252938,"read_op_per_sec":0,"write_op_per_sec":275},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:35.736 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:35.736 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:36.237 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 139 20 74554 74534 4.18821 2.28906 0.0628175 0.0372922 2024-09-18T11:46:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:36.284 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:36.304 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-18T11:46:35.870803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602129780,"num_objects":73507,"num_object_clones":0,"num_object_copies":294028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73534,"num_write_kb":588154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1599,"num_bytes_recovered":12861440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73561,"ondisk_log_size":73561,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2277252,"kb_used_data":1048996,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375210108,"statfs":{"total":386547056640,"available":384215150592,"internally_reserved":0,"allocated":1074171904,"data_stored":1073535992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,6,17,27,20,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33120256,"num_objects":4043,"num_object_clones":0,"num_object_copies":16172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":32344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006044"},"pg_stats":[{"pgid":"1.f","version":"31'4742","reported_seq":4783,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.691676+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:34.691676+0000","last_peered":"2024-09-18T11:46:34.691676+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:34.691676+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38846464,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4742,"num_write_kb":37936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4593","reported_seq":4656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.296017+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:31.296017+0000","last_peered":"2024-09-18T11:46:31.296017+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:31.296017+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590262,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":36724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4497","reported_seq":4579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.059418+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:31.059418+0000","last_peered":"2024-09-18T11:46:31.059418+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:31.059418+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36813435,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2440}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4564","reported_seq":4626,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.301153+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:31.301153+0000","last_peered":"2024-09-18T11:46:31.301153+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:31.301153+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37352694,"num_objects":4560,"num_object_clones":0,"num_object_copies":18240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":36485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4610","reported_seq":4664,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.703348+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:34.703348+0000","last_peered":"2024-09-18T11:46:34.703348+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:34.703348+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37737718,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4608,"num_write_kb":36861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4621","reported_seq":4664,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.716235+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:34.716235+0000","last_peered":"2024-09-18T11:46:34.716235+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:34.716235+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37855232,"num_objects":4621,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":36968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4627","reported_seq":4692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.288013+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:31.288013+0000","last_peered":"2024-09-18T11:46:31.288013+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:31.288013+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37852406,"num_objects":4621,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":36980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2487}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4488","reported_seq":4551,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.154477+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:31.154477+0000","last_peered":"2024-09-18T11:46:31.154477+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:31.154477+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36721910,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4486,"num_write_kb":35876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4754","reported_seq":4812,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.720430+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:34.720430+0000","last_peered":"2024-09-18T11:46:34.720430+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:34.720430+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4754,"log_dups_size":0,"ondisk_log_size":4754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38873580,"num_objects":4746,"num_object_clones":0,"num_object_copies":18984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4751,"num_write_kb":37985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2611}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4443","reported_seq":4526,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.283865+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:31.283865+0000","last_peered":"2024-09-18T11:46:31.283865+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:31.283865+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4443,"log_dups_size":0,"ondisk_log_size":4443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377846,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4441,"num_write_kb":35526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2466}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4652","reported_seq":4709,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.702497+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:34.702497+0000","last_peered":"2024-09-18T11:46:34.702497+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:34.702497+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38063985,"num_objects":4647,"num_object_clones":0,"num_object_copies":18588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":37180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2584}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4735","reported_seq":4772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.580782+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:34.580782+0000","last_peered":"2024-09-18T11:46:34.580782+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:34.580782+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38779515,"num_objects":4734,"num_object_clones":0,"num_object_copies":18936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2625}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4623","reported_seq":4699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.257220+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:31.257220+0000","last_peered":"2024-09-18T11:46:31.257220+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:31.257220+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37862011,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4622,"num_write_kb":36975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2568}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4648","reported_seq":4697,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.695209+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:34.695209+0000","last_peered":"2024-09-18T11:46:34.695209+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:34.695209+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4648,"log_dups_size":0,"ondisk_log_size":4648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38040822,"num_objects":4644,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4646,"num_write_kb":37164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2486}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4516","reported_seq":7743,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.288088+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:31.288088+0000","last_peered":"2024-09-18T11:46:31.288088+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:31.288088+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4516,"log_dups_size":0,"ondisk_log_size":4516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967670,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4514,"num_write_kb":36109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1574,"num_bytes_recovered":12861440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4448","reported_seq":4510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:31.250099+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:31.250099+0000","last_peered":"2024-09-18T11:46:31.250099+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:31.250099+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36394230,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4446,"num_write_kb":35556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2430}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":602129780,"num_objects":73507,"num_object_clones":0,"num_object_copies":294028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73534,"num_write_kb":588154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1599,"num_bytes_recovered":12861440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1074941952,"data_stored":1074941952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73561,"ondisk_log_size":73561,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606912,"kb_used_data":296568,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93764928,"statfs":{"total":96636764160,"available":96015286272,"internally_reserved":0,"allocated":303685632,"data_stored":303526654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,8,4,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":8.7100000000000009},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":8.7449999999999992}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":11.307},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.1950000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.159},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":13.672000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600752,"kb_used_data":290920,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93771088,"statfs":{"total":96636764160,"available":96021594112,"internally_reserved":0,"allocated":297902080,"data_stored":297743102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.091},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.313}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":0.439}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":0.70299999999999996},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.97799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467368,"kb_used_data":169568,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93904472,"statfs":{"total":96636764160,"available":96158179328,"internally_reserved":0,"allocated":173637632,"data_stored":173478654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":8.8420000000000005},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.33200000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":3.851},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":3.8860000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":3.6280000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602220,"kb_used_data":291940,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93769620,"statfs":{"total":96636764160,"available":96020090880,"internally_reserved":0,"allocated":298946560,"data_stored":298787582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,3,17,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.438},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":9.2669999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":5.2489999999999997},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":2.3969999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298893312,"data_stored":298893312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173613056,"data_stored":173613056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298827776,"data_stored":298827776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303607808,"data_stored":303607808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:37.237 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:46:37.237082+0000 min lat: 0.00272854 max lat: 2.42146 avg lat: 0.0373897 2024-09-18T11:46:37.237 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:46:37.238 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 140 20 74907 74887 4.17799 2.75781 0.0481107 0.0373897 2024-09-18T11:46:38.237 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 141 20 75240 75220 4.16681 2.60156 0.0541713 0.0374872 2024-09-18T11:46:39.237 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 142 20 75577 75557 4.15601 2.63281 0.0642817 0.0375819 2024-09-18T11:46:39.306 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:39.850 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:39.851 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:39.884 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-18T11:46:37.872165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":609281396,"num_objects":74380,"num_object_clones":0,"num_object_copies":297520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74407,"num_write_kb":595138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1683,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74434,"ondisk_log_size":74434,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2269504,"kb_used_data":1062048,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375217856,"statfs":{"total":386547056640,"available":384223084544,"internally_reserved":0,"allocated":1087537152,"data_stored":1086901240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,6,8,22,17,25,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34177024,"num_objects":4172,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4172,"num_write_kb":33376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":252,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005367"},"pg_stats":[{"pgid":"1.f","version":"31'4742","reported_seq":4783,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.691676+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:34.691676+0000","last_peered":"2024-09-18T11:46:34.691676+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:34.691676+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38846464,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4742,"num_write_kb":37936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4709","reported_seq":4772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:35.943914+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:35.943914+0000","last_peered":"2024-09-18T11:46:35.943914+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:35.943914+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38540534,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4588","reported_seq":4670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.203740+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:36.203740+0000","last_peered":"2024-09-18T11:46:36.203740+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:36.203740+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37558907,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4587,"num_write_kb":36686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2531}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4653","reported_seq":4715,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.201492+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:36.201492+0000","last_peered":"2024-09-18T11:46:36.201492+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:36.201492+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38081782,"num_objects":4649,"num_object_clones":0,"num_object_copies":18596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":37197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4610","reported_seq":4664,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.703348+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:34.703348+0000","last_peered":"2024-09-18T11:46:34.703348+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:34.703348+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37737718,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4608,"num_write_kb":36861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4621","reported_seq":4664,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.716235+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:34.716235+0000","last_peered":"2024-09-18T11:46:34.716235+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:34.716235+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37855232,"num_objects":4621,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":36968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4725","reported_seq":4789,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.346883+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:36.346883+0000","last_peered":"2024-09-18T11:46:36.346883+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:36.346883+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38655222,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4723,"num_write_kb":37764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2585}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4598","reported_seq":4661,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.320932+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:36.320932+0000","last_peered":"2024-09-18T11:46:36.320932+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:36.320932+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37623030,"num_objects":4593,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4596,"num_write_kb":36756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4754","reported_seq":4812,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.720430+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:34.720430+0000","last_peered":"2024-09-18T11:46:34.720430+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:34.720430+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4754,"log_dups_size":0,"ondisk_log_size":4754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38873580,"num_objects":4746,"num_object_clones":0,"num_object_copies":18984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4751,"num_write_kb":37985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2611}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4531","reported_seq":4615,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.131747+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:36.131747+0000","last_peered":"2024-09-18T11:46:36.131747+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:36.131747+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37098742,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4529,"num_write_kb":36230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2554}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4652","reported_seq":4709,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.702497+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:34.702497+0000","last_peered":"2024-09-18T11:46:34.702497+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:34.702497+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38063985,"num_objects":4647,"num_object_clones":0,"num_object_copies":18588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":37180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2584}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4735","reported_seq":4772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.580782+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:34.580782+0000","last_peered":"2024-09-18T11:46:34.580782+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:34.580782+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38779515,"num_objects":4734,"num_object_clones":0,"num_object_copies":18936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2625}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4723","reported_seq":4799,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.218160+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:36.218160+0000","last_peered":"2024-09-18T11:46:36.218160+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:36.218160+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4723,"log_dups_size":0,"ondisk_log_size":4723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38681211,"num_objects":4722,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4722,"num_write_kb":37775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2668}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4648","reported_seq":4697,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:34.695209+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:34.695209+0000","last_peered":"2024-09-18T11:46:34.695209+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:34.695209+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4648,"log_dups_size":0,"ondisk_log_size":4648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38040822,"num_objects":4644,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4646,"num_write_kb":37164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2486}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4627","reported_seq":8021,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.252238+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:36.252238+0000","last_peered":"2024-09-18T11:46:36.252238+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:36.252238+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876982,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":36997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1658,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4518","reported_seq":4580,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.089977+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:36.089977+0000","last_peered":"2024-09-18T11:46:36.089977+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:36.089977+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967670,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4516,"num_write_kb":36116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":609281396,"num_objects":74380,"num_object_clones":0,"num_object_copies":297520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74407,"num_write_kb":595138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1683,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1088204800,"data_stored":1088204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74434,"ondisk_log_size":74434,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606912,"kb_used_data":296568,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93764928,"statfs":{"total":96636764160,"available":96015286272,"internally_reserved":0,"allocated":303685632,"data_stored":303526654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,8,4,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":8.7100000000000009},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":8.7449999999999992}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":11.307},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.1950000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.159},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":13.672000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599028,"kb_used_data":297516,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93772812,"statfs":{"total":96636764160,"available":96023359488,"internally_reserved":0,"allocated":304656384,"data_stored":304497406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,6,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":2.3540000000000001},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.3210000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":3.722},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":2.4529999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":2.4209999999999998},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.125}]}]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467368,"kb_used_data":169568,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93904472,"statfs":{"total":96636764160,"available":96158179328,"internally_reserved":0,"allocated":173637632,"data_stored":173478654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":8.8420000000000005},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":0.33200000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":3.851},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":3.8860000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":3.6280000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596196,"kb_used_data":298396,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93775644,"statfs":{"total":96636764160,"available":96026259456,"internally_reserved":0,"allocated":305557504,"data_stored":305398526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":4.9509999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.9909999999999997},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.63300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":2.9220000000000002},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":5.016}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305557504,"data_stored":305557504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173613056,"data_stored":173613056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":305426432,"data_stored":305426432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303607808,"data_stored":303607808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:39.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:40.240 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 143 20 75873 75853 4.14312 2.3125 0.0727882 0.0376994 2024-09-18T11:46:40.424 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:40.424 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:40.441 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-18T11:46:39.873180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615245172,"num_objects":75108,"num_object_clones":0,"num_object_copies":300432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75135,"num_write_kb":600962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1683,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75162,"ondisk_log_size":75162,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2274440,"kb_used_data":1075304,"kb_used_omap":14,"kb_used_meta":1199089,"kb_avail":375212920,"statfs":{"total":386547056640,"available":384218030080,"internally_reserved":0,"allocated":1101111296,"data_stored":1100475384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227867163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,11,18,16,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32735232,"num_objects":3996,"num_object_clones":0,"num_object_copies":15984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3996,"num_write_kb":31968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005388"},"pg_stats":[{"pgid":"1.f","version":"31'4851","reported_seq":4892,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.383097+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:39.383097+0000","last_peered":"2024-09-18T11:46:39.383097+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:39.383097+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39739392,"num_objects":4851,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4851,"num_write_kb":38808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4709","reported_seq":4772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:35.943914+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:35.943914+0000","last_peered":"2024-09-18T11:46:35.943914+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:35.943914+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38540534,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4588","reported_seq":4670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.203740+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:36.203740+0000","last_peered":"2024-09-18T11:46:36.203740+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:36.203740+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37558907,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4587,"num_write_kb":36686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2531}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4653","reported_seq":4715,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.201492+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:36.201492+0000","last_peered":"2024-09-18T11:46:36.201492+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:36.201492+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38081782,"num_objects":4649,"num_object_clones":0,"num_object_copies":18596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":37197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4702","reported_seq":4757,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.742606+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:39.742606+0000","last_peered":"2024-09-18T11:46:39.742606+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:39.742606+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38491382,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":37597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4734","reported_seq":4777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.805340+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:39.805340+0000","last_peered":"2024-09-18T11:46:39.805340+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:39.805340+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38780928,"num_objects":4734,"num_object_clones":0,"num_object_copies":18936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4725","reported_seq":4789,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.346883+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:36.346883+0000","last_peered":"2024-09-18T11:46:36.346883+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:36.346883+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38655222,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4723,"num_write_kb":37764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2585}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4598","reported_seq":4661,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.320932+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:36.320932+0000","last_peered":"2024-09-18T11:46:36.320932+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:36.320932+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37623030,"num_objects":4593,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4596,"num_write_kb":36756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4858","reported_seq":4916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.816054+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:39.816054+0000","last_peered":"2024-09-18T11:46:39.816054+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:39.816054+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39725548,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":38817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4531","reported_seq":4615,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.131747+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:36.131747+0000","last_peered":"2024-09-18T11:46:36.131747+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:36.131747+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37098742,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4529,"num_write_kb":36230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2554}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4758","reported_seq":4814,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.702528+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:39.702528+0000","last_peered":"2024-09-18T11:46:39.702528+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:39.702528+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4758,"log_dups_size":0,"ondisk_log_size":4758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38932337,"num_objects":4753,"num_object_clones":0,"num_object_copies":19012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":38028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4832","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.829377+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:39.829377+0000","last_peered":"2024-09-18T11:46:39.829377+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:39.829377+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39574139,"num_objects":4831,"num_object_clones":0,"num_object_copies":19324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4831,"num_write_kb":38647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2722}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4723","reported_seq":4799,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.218160+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:36.218160+0000","last_peered":"2024-09-18T11:46:36.218160+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:36.218160+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4723,"log_dups_size":0,"ondisk_log_size":4723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38681211,"num_objects":4722,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4722,"num_write_kb":37775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2668}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4755","reported_seq":4803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.796030+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:39.796030+0000","last_peered":"2024-09-18T11:46:39.796030+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:39.796030+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38917366,"num_objects":4751,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4753,"num_write_kb":38020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4627","reported_seq":8021,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.252238+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:36.252238+0000","last_peered":"2024-09-18T11:46:36.252238+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:36.252238+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876982,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":36997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1658,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4518","reported_seq":4580,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.089977+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:36.089977+0000","last_peered":"2024-09-18T11:46:36.089977+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:36.089977+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967670,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4516,"num_write_kb":36116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":615245172,"num_objects":75108,"num_object_clones":0,"num_object_copies":300432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75135,"num_write_kb":600962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1683,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1101918208,"data_stored":1101918208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75162,"ondisk_log_size":75162,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600876,"kb_used_data":303012,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93770964,"statfs":{"total":96636764160,"available":96021467136,"internally_reserved":0,"allocated":310284288,"data_stored":310125310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":0.46200000000000002},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.48699999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":0.71999999999999997},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":8.9019999999999992},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.179}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599028,"kb_used_data":297516,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93772812,"statfs":{"total":96636764160,"available":96023359488,"internally_reserved":0,"allocated":304656384,"data_stored":304497406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,6,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":2.3540000000000001},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.3210000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":3.722},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":2.4529999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":2.4209999999999998},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.125}]}]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478340,"kb_used_data":176380,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93893500,"statfs":{"total":96636764160,"available":96146944000,"internally_reserved":0,"allocated":180613120,"data_stored":180454142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":10.48},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":7.2549999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":0.254},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":26.719000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":14.185},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":6.7329999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596196,"kb_used_data":298396,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93775644,"statfs":{"total":96636764160,"available":96026259456,"internally_reserved":0,"allocated":305557504,"data_stored":305398526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":4.9509999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.9909999999999997},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.63300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":2.9220000000000002},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":5.016}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305557504,"data_stored":305557504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180649984,"data_stored":180649984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":305426432,"data_stored":305426432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310284288,"data_stored":310284288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:40.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:40.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4851", 'reported_seq': 4892, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.383097+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:39.383097+0000', 'last_peered': '2024-09-18T11:46:39.383097+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:39.383097+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4851, 'log_dups_size': 0, 'ondisk_log_size': 4851, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39739392, 'num_objects': 4851, 'num_object_clones': 0, 'num_object_copies': 19404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4851, 'num_write_kb': 38808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2721}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:40.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:40.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4709", 'reported_seq': 4772, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:35.943914+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:35.943914+0000', 'last_peered': '2024-09-18T11:46:35.943914+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:35.943914+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4709, 'log_dups_size': 0, 'ondisk_log_size': 4709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38540534, 'num_objects': 4705, 'num_object_clones': 0, 'num_object_copies': 18820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4707, 'num_write_kb': 37652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2603}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:40.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:40.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4588", 'reported_seq': 4670, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:36.203740+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:36.203740+0000', 'last_peered': '2024-09-18T11:46:36.203740+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:36.203740+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4588, 'log_dups_size': 0, 'ondisk_log_size': 4588, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37558907, 'num_objects': 4585, 'num_object_clones': 0, 'num_object_copies': 18340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4587, 'num_write_kb': 36686, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2531}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:40.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:40.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4653", 'reported_seq': 4715, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:36.201492+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:36.201492+0000', 'last_peered': '2024-09-18T11:46:36.201492+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:36.201492+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4653, 'log_dups_size': 0, 'ondisk_log_size': 4653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38081782, 'num_objects': 4649, 'num_object_clones': 0, 'num_object_copies': 18596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4651, 'num_write_kb': 37197, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:40.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:40.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4702", 'reported_seq': 4757, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.742606+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:39.742606+0000', 'last_peered': '2024-09-18T11:46:39.742606+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:39.742606+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4702, 'log_dups_size': 0, 'ondisk_log_size': 4702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38491382, 'num_objects': 4699, 'num_object_clones': 0, 'num_object_copies': 18796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4700, 'num_write_kb': 37597, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2617}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:40.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:40.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4734", 'reported_seq': 4777, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.805340+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:39.805340+0000', 'last_peered': '2024-09-18T11:46:39.805340+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:39.805340+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4734, 'log_dups_size': 0, 'ondisk_log_size': 4734, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38780928, 'num_objects': 4734, 'num_object_clones': 0, 'num_object_copies': 18936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4734, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4734, 'num_write_kb': 37872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2580}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:40.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:40.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4725", 'reported_seq': 4789, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:36.346883+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:36.346883+0000', 'last_peered': '2024-09-18T11:46:36.346883+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:36.346883+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4725, 'log_dups_size': 0, 'ondisk_log_size': 4725, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38655222, 'num_objects': 4719, 'num_object_clones': 0, 'num_object_copies': 18876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4723, 'num_write_kb': 37764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2585}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:40.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:40.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4598", 'reported_seq': 4661, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:36.320932+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:36.320932+0000', 'last_peered': '2024-09-18T11:46:36.320932+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:36.320932+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4598, 'log_dups_size': 0, 'ondisk_log_size': 4598, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37623030, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 18372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4596, 'num_write_kb': 36756, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:40.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:40.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4858", 'reported_seq': 4916, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.816054+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:39.816054+0000', 'last_peered': '2024-09-18T11:46:39.816054+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:39.816054+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4858, 'log_dups_size': 0, 'ondisk_log_size': 4858, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39725548, 'num_objects': 4850, 'num_object_clones': 0, 'num_object_copies': 19400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4855, 'num_write_kb': 38817, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2715}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:40.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:40.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4531", 'reported_seq': 4615, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:36.131747+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:36.131747+0000', 'last_peered': '2024-09-18T11:46:36.131747+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:36.131747+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37098742, 'num_objects': 4529, 'num_object_clones': 0, 'num_object_copies': 18116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4529, 'num_write_kb': 36230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2554}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:40.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:40.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4758", 'reported_seq': 4814, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.702528+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:39.702528+0000', 'last_peered': '2024-09-18T11:46:39.702528+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:39.702528+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4758, 'log_dups_size': 0, 'ondisk_log_size': 4758, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38932337, 'num_objects': 4753, 'num_object_clones': 0, 'num_object_copies': 19012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4753, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4755, 'num_write_kb': 38028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2690}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:40.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:40.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4832", 'reported_seq': 4867, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.829377+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:39.829377+0000', 'last_peered': '2024-09-18T11:46:39.829377+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:39.829377+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4832, 'log_dups_size': 0, 'ondisk_log_size': 4832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39574139, 'num_objects': 4831, 'num_object_clones': 0, 'num_object_copies': 19324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4831, 'num_write_kb': 38647, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2722}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:40.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:40.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4723", 'reported_seq': 4799, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:36.218160+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:36.218160+0000', 'last_peered': '2024-09-18T11:46:36.218160+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:36.218160+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4723, 'log_dups_size': 0, 'ondisk_log_size': 4723, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38681211, 'num_objects': 4722, 'num_object_clones': 0, 'num_object_copies': 18888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4722, 'num_write_kb': 37775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2668}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:40.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:40.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4755", 'reported_seq': 4803, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.796030+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:39.796030+0000', 'last_peered': '2024-09-18T11:46:39.796030+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:39.796030+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4755, 'log_dups_size': 0, 'ondisk_log_size': 4755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38917366, 'num_objects': 4751, 'num_object_clones': 0, 'num_object_copies': 19004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4753, 'num_write_kb': 38020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:40.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:40.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4627", 'reported_seq': 8021, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:36.252238+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:36.252238+0000', 'last_peered': '2024-09-18T11:46:36.252238+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:36.252238+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4627, 'log_dups_size': 0, 'ondisk_log_size': 4627, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37876982, 'num_objects': 4624, 'num_object_clones': 0, 'num_object_copies': 18496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 437, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4625, 'num_write_kb': 36997, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1658, 'num_bytes_recovered': 13549568, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4187}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:40.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:40.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4518", 'reported_seq': 4580, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:36.089977+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:36.089977+0000', 'last_peered': '2024-09-18T11:46:36.089977+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:36.089977+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4518, 'log_dups_size': 0, 'ondisk_log_size': 4518, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36967670, 'num_objects': 4513, 'num_object_clones': 0, 'num_object_copies': 18052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4516, 'num_write_kb': 36116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2500}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:40.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:41.146 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:41.146 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31773/300432 objects degraded (10.576%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":199,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":75108,"data_bytes":615245172,"bytes_used":2329026560,"bytes_avail":384218030080,"bytes_total":386547056640,"degraded_objects":31773,"degraded_total":300432,"degraded_ratio":0.10575770889918518,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":116683,"recovering_keys_per_sec":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"write_bytes_sec":2726711,"read_op_per_sec":0,"write_op_per_sec":332},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:41.172 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:41.172 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:41.241 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 144 20 76170 76150 4.13046 2.32031 0.101478 0.0378146 2024-09-18T11:46:41.719 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:41.720 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:41.738 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-18T11:46:39.873180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615245172,"num_objects":75108,"num_object_clones":0,"num_object_copies":300432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75135,"num_write_kb":600962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1683,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75162,"ondisk_log_size":75162,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2274440,"kb_used_data":1075304,"kb_used_omap":14,"kb_used_meta":1199089,"kb_avail":375212920,"statfs":{"total":386547056640,"available":384218030080,"internally_reserved":0,"allocated":1101111296,"data_stored":1100475384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227867163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,11,18,16,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32735232,"num_objects":3996,"num_object_clones":0,"num_object_copies":15984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3996,"num_write_kb":31968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005388"},"pg_stats":[{"pgid":"1.f","version":"31'4851","reported_seq":4892,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.383097+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:39.383097+0000","last_peered":"2024-09-18T11:46:39.383097+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:39.383097+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39739392,"num_objects":4851,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4851,"num_write_kb":38808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4709","reported_seq":4772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:35.943914+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:35.943914+0000","last_peered":"2024-09-18T11:46:35.943914+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:35.943914+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38540534,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4588","reported_seq":4670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.203740+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:36.203740+0000","last_peered":"2024-09-18T11:46:36.203740+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:36.203740+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37558907,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4587,"num_write_kb":36686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2531}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4653","reported_seq":4715,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.201492+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:36.201492+0000","last_peered":"2024-09-18T11:46:36.201492+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:36.201492+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38081782,"num_objects":4649,"num_object_clones":0,"num_object_copies":18596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":37197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4702","reported_seq":4757,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.742606+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:39.742606+0000","last_peered":"2024-09-18T11:46:39.742606+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:39.742606+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38491382,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":37597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4734","reported_seq":4777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.805340+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:39.805340+0000","last_peered":"2024-09-18T11:46:39.805340+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:39.805340+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38780928,"num_objects":4734,"num_object_clones":0,"num_object_copies":18936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4725","reported_seq":4789,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.346883+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:36.346883+0000","last_peered":"2024-09-18T11:46:36.346883+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:36.346883+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38655222,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4723,"num_write_kb":37764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2585}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4598","reported_seq":4661,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.320932+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:36.320932+0000","last_peered":"2024-09-18T11:46:36.320932+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:36.320932+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37623030,"num_objects":4593,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4596,"num_write_kb":36756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4858","reported_seq":4916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.816054+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:39.816054+0000","last_peered":"2024-09-18T11:46:39.816054+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:39.816054+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39725548,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":38817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4531","reported_seq":4615,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.131747+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:36.131747+0000","last_peered":"2024-09-18T11:46:36.131747+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:36.131747+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37098742,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4529,"num_write_kb":36230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2554}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4758","reported_seq":4814,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.702528+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:39.702528+0000","last_peered":"2024-09-18T11:46:39.702528+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:39.702528+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4758,"log_dups_size":0,"ondisk_log_size":4758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38932337,"num_objects":4753,"num_object_clones":0,"num_object_copies":19012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":38028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4832","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.829377+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:39.829377+0000","last_peered":"2024-09-18T11:46:39.829377+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:39.829377+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39574139,"num_objects":4831,"num_object_clones":0,"num_object_copies":19324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4831,"num_write_kb":38647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2722}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4723","reported_seq":4799,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.218160+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:36.218160+0000","last_peered":"2024-09-18T11:46:36.218160+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:36.218160+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4723,"log_dups_size":0,"ondisk_log_size":4723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38681211,"num_objects":4722,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4722,"num_write_kb":37775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2668}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4755","reported_seq":4803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.796030+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:39.796030+0000","last_peered":"2024-09-18T11:46:39.796030+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:39.796030+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38917366,"num_objects":4751,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4753,"num_write_kb":38020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4627","reported_seq":8021,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.252238+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:36.252238+0000","last_peered":"2024-09-18T11:46:36.252238+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:36.252238+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876982,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":36997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1658,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4518","reported_seq":4580,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:36.089977+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:36.089977+0000","last_peered":"2024-09-18T11:46:36.089977+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:36.089977+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967670,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4516,"num_write_kb":36116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":615245172,"num_objects":75108,"num_object_clones":0,"num_object_copies":300432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75135,"num_write_kb":600962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1683,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1101918208,"data_stored":1101918208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75162,"ondisk_log_size":75162,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600876,"kb_used_data":303012,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93770964,"statfs":{"total":96636764160,"available":96021467136,"internally_reserved":0,"allocated":310284288,"data_stored":310125310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":0.46200000000000002},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.48699999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":0.71999999999999997},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":8.9019999999999992},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.179}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599028,"kb_used_data":297516,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93772812,"statfs":{"total":96636764160,"available":96023359488,"internally_reserved":0,"allocated":304656384,"data_stored":304497406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,6,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":2.3540000000000001},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.3210000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":3.722},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":2.4529999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":2.4209999999999998},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.125}]}]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478340,"kb_used_data":176380,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93893500,"statfs":{"total":96636764160,"available":96146944000,"internally_reserved":0,"allocated":180613120,"data_stored":180454142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":10.48},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":7.2549999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":0.254},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":26.719000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":14.185},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":6.7329999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596196,"kb_used_data":298396,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93775644,"statfs":{"total":96636764160,"available":96026259456,"internally_reserved":0,"allocated":305557504,"data_stored":305398526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":4.9509999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.9909999999999997},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.63300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":2.9220000000000002},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":5.016}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305557504,"data_stored":305557504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180649984,"data_stored":180649984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":305426432,"data_stored":305426432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310284288,"data_stored":310284288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:42.238 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 145 20 76489 76469 4.11916 2.49219 0.0412058 0.0379198 2024-09-18T11:46:43.238 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 146 20 76825 76805 4.10893 2.625 0.0392949 0.0380131 2024-09-18T11:46:44.238 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 147 20 77166 77146 4.09909 2.66406 0.0661651 0.0381049 2024-09-18T11:46:44.740 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:45.240 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 148 20 77473 77453 4.0876 2.39844 0.0468431 0.0382097 2024-09-18T11:46:45.273 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:45.273 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:45.303 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-18T11:46:43.875132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622912884,"num_objects":76044,"num_object_clones":0,"num_object_copies":304176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76071,"num_write_kb":608450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1774,"num_bytes_recovered":14295040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76098,"ondisk_log_size":76098,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2295700,"kb_used_data":1088244,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375191660,"statfs":{"total":386547056640,"available":384196259840,"internally_reserved":0,"allocated":1114361856,"data_stored":1113725944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,1,18,26,17,14,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26853376,"num_objects":3278,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3278,"num_write_kb":26224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005741"},"pg_stats":[{"pgid":"1.f","version":"31'4851","reported_seq":4892,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.383097+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:39.383097+0000","last_peered":"2024-09-18T11:46:39.383097+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:39.383097+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39739392,"num_objects":4851,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4851,"num_write_kb":38808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4815","reported_seq":4878,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.213714+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:41.213714+0000","last_peered":"2024-09-18T11:46:41.213714+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:41.213714+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39408886,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2709}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4700","reported_seq":4782,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.209645+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:41.209645+0000","last_peered":"2024-09-18T11:46:41.209645+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:41.209645+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38476411,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2643}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4748","reported_seq":4810,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.281713+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:41.281713+0000","last_peered":"2024-09-18T11:46:41.281713+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:41.281713+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38860022,"num_objects":4744,"num_object_clones":0,"num_object_copies":18976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4746,"num_write_kb":37957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4702","reported_seq":4757,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.742606+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:39.742606+0000","last_peered":"2024-09-18T11:46:39.742606+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:39.742606+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38491382,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":37597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4734","reported_seq":4777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.805340+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:39.805340+0000","last_peered":"2024-09-18T11:46:39.805340+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:39.805340+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38780928,"num_objects":4734,"num_object_clones":0,"num_object_copies":18936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4819","reported_seq":4883,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.354919+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:41.354919+0000","last_peered":"2024-09-18T11:46:41.354919+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:41.354919+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39425270,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4817,"num_write_kb":38516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2679}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4697","reported_seq":4760,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.339303+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:41.339303+0000","last_peered":"2024-09-18T11:46:41.339303+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:41.339303+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38434038,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":37548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4858","reported_seq":4916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.816054+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:39.816054+0000","last_peered":"2024-09-18T11:46:39.816054+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:39.816054+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39725548,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":38817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4653","reported_seq":4736,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.209843+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:41.209843+0000","last_peered":"2024-09-18T11:46:41.209843+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:41.209843+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38098166,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":37206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4758","reported_seq":4814,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.702528+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:39.702528+0000","last_peered":"2024-09-18T11:46:39.702528+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:39.702528+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4758,"log_dups_size":0,"ondisk_log_size":4758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38932337,"num_objects":4753,"num_object_clones":0,"num_object_copies":19012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":38028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4832","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.829377+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:39.829377+0000","last_peered":"2024-09-18T11:46:39.829377+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:39.829377+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39574139,"num_objects":4831,"num_object_clones":0,"num_object_copies":19324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4831,"num_write_kb":38647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2722}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4814","reported_seq":4890,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.266180+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:41.266180+0000","last_peered":"2024-09-18T11:46:41.266180+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:41.266180+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39426683,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2759}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4755","reported_seq":4803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.796030+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:39.796030+0000","last_peered":"2024-09-18T11:46:39.796030+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:39.796030+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38917366,"num_objects":4751,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4753,"num_write_kb":38020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4735","reported_seq":8312,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.265166+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:41.265166+0000","last_peered":"2024-09-18T11:46:41.265166+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:41.265166+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38761718,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1749,"num_bytes_recovered":14295040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4627","reported_seq":4689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.349105+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:41.349105+0000","last_peered":"2024-09-18T11:46:41.349105+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:41.349105+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37860598,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":36988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2609}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622912884,"num_objects":76044,"num_object_clones":0,"num_object_copies":304176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76071,"num_write_kb":608450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1774,"num_bytes_recovered":14295040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1115213824,"data_stored":1115213824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76098,"ondisk_log_size":76098,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600876,"kb_used_data":303012,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93770964,"statfs":{"total":96636764160,"available":96021467136,"internally_reserved":0,"allocated":310284288,"data_stored":310125310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":0.46200000000000002},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.48699999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":0.71999999999999997},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":8.9019999999999992},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.179}]}]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609632,"kb_used_data":303960,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93762208,"statfs":{"total":96636764160,"available":96012500992,"internally_reserved":0,"allocated":311255040,"data_stored":311096062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":3.79},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.7570000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":18.646999999999998},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":3.8250000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":8.6050000000000004},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.877}]}]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478340,"kb_used_data":176380,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93893500,"statfs":{"total":96636764160,"available":96146944000,"internally_reserved":0,"allocated":180613120,"data_stored":180454142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":10.48},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":7.2549999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":0.254},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":26.719000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":14.185},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":6.7329999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606852,"kb_used_data":304892,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93764988,"statfs":{"total":96636764160,"available":96015347712,"internally_reserved":0,"allocated":312209408,"data_stored":312050430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,9,17,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":9.5679999999999996},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.9909999999999997},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.63300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":9.5289999999999999},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":4.4989999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312201216,"data_stored":312201216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180649984,"data_stored":180649984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312078336,"data_stored":312078336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310284288,"data_stored":310284288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:45.304 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:45.840 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:45.840 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:45.859 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-18T11:46:43.875132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622912884,"num_objects":76044,"num_object_clones":0,"num_object_copies":304176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76071,"num_write_kb":608450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1774,"num_bytes_recovered":14295040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76098,"ondisk_log_size":76098,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2295700,"kb_used_data":1088244,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375191660,"statfs":{"total":386547056640,"available":384196259840,"internally_reserved":0,"allocated":1114361856,"data_stored":1113725944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,1,18,26,17,14,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26853376,"num_objects":3278,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3278,"num_write_kb":26224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005741"},"pg_stats":[{"pgid":"1.f","version":"31'4851","reported_seq":4892,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.383097+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:39.383097+0000","last_peered":"2024-09-18T11:46:39.383097+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:39.383097+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39739392,"num_objects":4851,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4851,"num_write_kb":38808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4815","reported_seq":4878,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.213714+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:41.213714+0000","last_peered":"2024-09-18T11:46:41.213714+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:41.213714+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39408886,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2709}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4700","reported_seq":4782,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.209645+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:41.209645+0000","last_peered":"2024-09-18T11:46:41.209645+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:41.209645+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38476411,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2643}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4748","reported_seq":4810,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.281713+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:41.281713+0000","last_peered":"2024-09-18T11:46:41.281713+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:41.281713+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38860022,"num_objects":4744,"num_object_clones":0,"num_object_copies":18976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4746,"num_write_kb":37957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4702","reported_seq":4757,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.742606+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:39.742606+0000","last_peered":"2024-09-18T11:46:39.742606+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:39.742606+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38491382,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":37597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4734","reported_seq":4777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.805340+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:39.805340+0000","last_peered":"2024-09-18T11:46:39.805340+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:39.805340+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38780928,"num_objects":4734,"num_object_clones":0,"num_object_copies":18936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4819","reported_seq":4883,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.354919+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:41.354919+0000","last_peered":"2024-09-18T11:46:41.354919+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:41.354919+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39425270,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4817,"num_write_kb":38516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2679}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4697","reported_seq":4760,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.339303+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:41.339303+0000","last_peered":"2024-09-18T11:46:41.339303+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:41.339303+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38434038,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":37548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4858","reported_seq":4916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.816054+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:39.816054+0000","last_peered":"2024-09-18T11:46:39.816054+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:39.816054+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39725548,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":38817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4653","reported_seq":4736,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.209843+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:41.209843+0000","last_peered":"2024-09-18T11:46:41.209843+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:41.209843+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38098166,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":37206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4758","reported_seq":4814,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.702528+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:39.702528+0000","last_peered":"2024-09-18T11:46:39.702528+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:39.702528+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4758,"log_dups_size":0,"ondisk_log_size":4758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38932337,"num_objects":4753,"num_object_clones":0,"num_object_copies":19012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":38028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4832","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.829377+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:39.829377+0000","last_peered":"2024-09-18T11:46:39.829377+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:39.829377+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39574139,"num_objects":4831,"num_object_clones":0,"num_object_copies":19324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4831,"num_write_kb":38647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2722}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4814","reported_seq":4890,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.266180+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:41.266180+0000","last_peered":"2024-09-18T11:46:41.266180+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:41.266180+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39426683,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2759}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4755","reported_seq":4803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:39.796030+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:39.796030+0000","last_peered":"2024-09-18T11:46:39.796030+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:39.796030+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38917366,"num_objects":4751,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4753,"num_write_kb":38020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4735","reported_seq":8312,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.265166+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:41.265166+0000","last_peered":"2024-09-18T11:46:41.265166+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:41.265166+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38761718,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1749,"num_bytes_recovered":14295040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4627","reported_seq":4689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.349105+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:41.349105+0000","last_peered":"2024-09-18T11:46:41.349105+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:41.349105+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37860598,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":36988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2609}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622912884,"num_objects":76044,"num_object_clones":0,"num_object_copies":304176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76071,"num_write_kb":608450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1774,"num_bytes_recovered":14295040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1115213824,"data_stored":1115213824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76098,"ondisk_log_size":76098,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600876,"kb_used_data":303012,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93770964,"statfs":{"total":96636764160,"available":96021467136,"internally_reserved":0,"allocated":310284288,"data_stored":310125310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":0.46200000000000002},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.48699999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":0.71999999999999997},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":8.9019999999999992},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.179}]}]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609632,"kb_used_data":303960,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93762208,"statfs":{"total":96636764160,"available":96012500992,"internally_reserved":0,"allocated":311255040,"data_stored":311096062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":3.79},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.7570000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":18.646999999999998},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":3.8250000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":8.6050000000000004},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.877}]}]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478340,"kb_used_data":176380,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93893500,"statfs":{"total":96636764160,"available":96146944000,"internally_reserved":0,"allocated":180613120,"data_stored":180454142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":10.48},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":7.2549999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":0.254},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":26.719000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":14.185},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":6.7329999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606852,"kb_used_data":304892,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93764988,"statfs":{"total":96636764160,"available":96015347712,"internally_reserved":0,"allocated":312209408,"data_stored":312050430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,9,17,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":9.5679999999999996},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.9909999999999997},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.63300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":9.5289999999999999},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":4.4989999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312201216,"data_stored":312201216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180649984,"data_stored":180649984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312078336,"data_stored":312078336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310284288,"data_stored":310284288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:45.860 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:45.860 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4851", 'reported_seq': 4892, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.383097+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:39.383097+0000', 'last_peered': '2024-09-18T11:46:39.383097+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:39.383097+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4851, 'log_dups_size': 0, 'ondisk_log_size': 4851, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39739392, 'num_objects': 4851, 'num_object_clones': 0, 'num_object_copies': 19404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4851, 'num_write_kb': 38808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2721}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:45.860 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:45.860 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4815", 'reported_seq': 4878, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:41.213714+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:41.213714+0000', 'last_peered': '2024-09-18T11:46:41.213714+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:41.213714+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4815, 'log_dups_size': 0, 'ondisk_log_size': 4815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39408886, 'num_objects': 4811, 'num_object_clones': 0, 'num_object_copies': 19244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4811, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4813, 'num_write_kb': 38500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2709}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4700", 'reported_seq': 4782, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:41.209645+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:41.209645+0000', 'last_peered': '2024-09-18T11:46:41.209645+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:41.209645+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4700, 'log_dups_size': 0, 'ondisk_log_size': 4700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38476411, 'num_objects': 4697, 'num_object_clones': 0, 'num_object_copies': 18788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4699, 'num_write_kb': 37582, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2643}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4748", 'reported_seq': 4810, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:41.281713+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:41.281713+0000', 'last_peered': '2024-09-18T11:46:41.281713+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:41.281713+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4748, 'log_dups_size': 0, 'ondisk_log_size': 4748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38860022, 'num_objects': 4744, 'num_object_clones': 0, 'num_object_copies': 18976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4746, 'num_write_kb': 37957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2630}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4702", 'reported_seq': 4757, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.742606+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:39.742606+0000', 'last_peered': '2024-09-18T11:46:39.742606+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:39.742606+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4702, 'log_dups_size': 0, 'ondisk_log_size': 4702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38491382, 'num_objects': 4699, 'num_object_clones': 0, 'num_object_copies': 18796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4700, 'num_write_kb': 37597, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2617}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4734", 'reported_seq': 4777, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.805340+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:39.805340+0000', 'last_peered': '2024-09-18T11:46:39.805340+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:39.805340+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4734, 'log_dups_size': 0, 'ondisk_log_size': 4734, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38780928, 'num_objects': 4734, 'num_object_clones': 0, 'num_object_copies': 18936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4734, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4734, 'num_write_kb': 37872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2580}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4819", 'reported_seq': 4883, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:41.354919+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:41.354919+0000', 'last_peered': '2024-09-18T11:46:41.354919+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:41.354919+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4819, 'log_dups_size': 0, 'ondisk_log_size': 4819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39425270, 'num_objects': 4813, 'num_object_clones': 0, 'num_object_copies': 19252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4817, 'num_write_kb': 38516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2679}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:45.861 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:45.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4697", 'reported_seq': 4760, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:41.339303+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:41.339303+0000', 'last_peered': '2024-09-18T11:46:41.339303+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:41.339303+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4697, 'log_dups_size': 0, 'ondisk_log_size': 4697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38434038, 'num_objects': 4692, 'num_object_clones': 0, 'num_object_copies': 18768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4695, 'num_write_kb': 37548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2658}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:45.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:45.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4858", 'reported_seq': 4916, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.816054+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:39.816054+0000', 'last_peered': '2024-09-18T11:46:39.816054+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:39.816054+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4858, 'log_dups_size': 0, 'ondisk_log_size': 4858, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39725548, 'num_objects': 4850, 'num_object_clones': 0, 'num_object_copies': 19400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4855, 'num_write_kb': 38817, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2715}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:45.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:45.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4653", 'reported_seq': 4736, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:41.209843+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:41.209843+0000', 'last_peered': '2024-09-18T11:46:41.209843+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:41.209843+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4653, 'log_dups_size': 0, 'ondisk_log_size': 4653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38098166, 'num_objects': 4651, 'num_object_clones': 0, 'num_object_copies': 18604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4651, 'num_write_kb': 37206, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2676}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:45.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:45.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4758", 'reported_seq': 4814, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.702528+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:39.702528+0000', 'last_peered': '2024-09-18T11:46:39.702528+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:39.702528+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4758, 'log_dups_size': 0, 'ondisk_log_size': 4758, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38932337, 'num_objects': 4753, 'num_object_clones': 0, 'num_object_copies': 19012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4753, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4755, 'num_write_kb': 38028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2690}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:45.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:45.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4832", 'reported_seq': 4867, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.829377+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:39.829377+0000', 'last_peered': '2024-09-18T11:46:39.829377+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:39.829377+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4832, 'log_dups_size': 0, 'ondisk_log_size': 4832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39574139, 'num_objects': 4831, 'num_object_clones': 0, 'num_object_copies': 19324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4831, 'num_write_kb': 38647, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2722}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:45.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:45.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4814", 'reported_seq': 4890, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:41.266180+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:41.266180+0000', 'last_peered': '2024-09-18T11:46:41.266180+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:41.266180+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4814, 'log_dups_size': 0, 'ondisk_log_size': 4814, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39426683, 'num_objects': 4813, 'num_object_clones': 0, 'num_object_copies': 19252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4813, 'num_write_kb': 38503, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2759}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:45.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:45.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4755", 'reported_seq': 4803, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:39.796030+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:39.796030+0000', 'last_peered': '2024-09-18T11:46:39.796030+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:39.796030+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4755, 'log_dups_size': 0, 'ondisk_log_size': 4755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38917366, 'num_objects': 4751, 'num_object_clones': 0, 'num_object_copies': 19004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4753, 'num_write_kb': 38020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:45.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:45.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4735", 'reported_seq': 8312, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:41.265166+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:41.265166+0000', 'last_peered': '2024-09-18T11:46:41.265166+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:41.265166+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4735, 'log_dups_size': 0, 'ondisk_log_size': 4735, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38761718, 'num_objects': 4732, 'num_object_clones': 0, 'num_object_copies': 18928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 346, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4733, 'num_write_kb': 37861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1749, 'num_bytes_recovered': 14295040, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4386}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:45.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:45.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4627", 'reported_seq': 4689, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:41.349105+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:41.349105+0000', 'last_peered': '2024-09-18T11:46:41.349105+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:41.349105+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4627, 'log_dups_size': 0, 'ondisk_log_size': 4627, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37860598, 'num_objects': 4622, 'num_object_clones': 0, 'num_object_copies': 18488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4625, 'num_write_kb': 36988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2609}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:45.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:46.238 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 149 20 77772 77752 4.07584 2.33594 0.0612826 0.0383225 2024-09-18T11:46:46.550 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:46.550 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31682/306928 objects degraded (10.322%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":204,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76732,"data_bytes":628548980,"bytes_used":2372841472,"bytes_avail":384174215168,"bytes_total":386547056640,"degraded_objects":31682,"degraded_total":306928,"degraded_ratio":0.10322290569775322,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":119408,"recovering_keys_per_sec":0,"num_objects_recovered":175,"num_bytes_recovered":1433600,"num_keys_recovered":0,"write_bytes_sec":2706137,"read_op_per_sec":0,"write_op_per_sec":330},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:46.576 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:46.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:47.128 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:47.129 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:47.155 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-18T11:46:45.876209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":628548980,"num_objects":76732,"num_object_clones":0,"num_object_copies":306928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76759,"num_write_kb":613954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1774,"num_bytes_recovered":14295040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76786,"ondisk_log_size":76786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2317228,"kb_used_data":1101452,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":375170132,"statfs":{"total":386547056640,"available":384174215168,"internally_reserved":0,"allocated":1127886848,"data_stored":1127250936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,6,1,15,24,14,10,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32489472,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3966,"num_write_kb":31728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005848"},"pg_stats":[{"pgid":"1.f","version":"31'4952","reported_seq":4992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.797913+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:44.797913+0000","last_peered":"2024-09-18T11:46:44.797913+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:44.797913+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4952,"num_write_kb":39616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4815","reported_seq":4878,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.213714+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:41.213714+0000","last_peered":"2024-09-18T11:46:41.213714+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:41.213714+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39408886,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2709}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4700","reported_seq":4782,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.209645+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:41.209645+0000","last_peered":"2024-09-18T11:46:41.209645+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:41.209645+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38476411,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2643}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4748","reported_seq":4810,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.281713+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:41.281713+0000","last_peered":"2024-09-18T11:46:41.281713+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:41.281713+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38860022,"num_objects":4744,"num_object_clones":0,"num_object_copies":18976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4746,"num_write_kb":37957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4806","reported_seq":4861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.652758+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:44.652758+0000","last_peered":"2024-09-18T11:46:44.652758+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:44.652758+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343350,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4804,"num_write_kb":38429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4836","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.706229+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:44.706229+0000","last_peered":"2024-09-18T11:46:44.706229+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:44.706229+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39616512,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4819","reported_seq":4883,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.354919+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:41.354919+0000","last_peered":"2024-09-18T11:46:41.354919+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:41.354919+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39425270,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4817,"num_write_kb":38516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2679}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4697","reported_seq":4760,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.339303+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:41.339303+0000","last_peered":"2024-09-18T11:46:41.339303+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:41.339303+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38434038,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":37548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4954","reported_seq":5012,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.774248+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:44.774248+0000","last_peered":"2024-09-18T11:46:44.774248+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:44.774248+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40511980,"num_objects":4946,"num_object_clones":0,"num_object_copies":19784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":39585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4653","reported_seq":4736,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.209843+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:41.209843+0000","last_peered":"2024-09-18T11:46:41.209843+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:41.209843+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38098166,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":37206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4841","reported_seq":4897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.792385+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:44.792385+0000","last_peered":"2024-09-18T11:46:44.792385+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:44.792385+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39612273,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4838,"num_write_kb":38692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4925","reported_seq":4962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.696730+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:44.696730+0000","last_peered":"2024-09-18T11:46:44.696730+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:44.696730+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40335995,"num_objects":4924,"num_object_clones":0,"num_object_copies":19696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4924,"num_write_kb":39391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4814","reported_seq":4890,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.266180+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:41.266180+0000","last_peered":"2024-09-18T11:46:41.266180+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:41.266180+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39426683,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2759}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4864","reported_seq":4913,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.805568+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:44.805568+0000","last_peered":"2024-09-18T11:46:44.805568+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:44.805568+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39810294,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4735","reported_seq":8312,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.265166+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:41.265166+0000","last_peered":"2024-09-18T11:46:41.265166+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:41.265166+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38761718,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1749,"num_bytes_recovered":14295040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4627","reported_seq":4689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:41.349105+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:41.349105+0000","last_peered":"2024-09-18T11:46:41.349105+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:41.349105+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37860598,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":36988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2609}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":628548980,"num_objects":76732,"num_object_clones":0,"num_object_copies":306928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76759,"num_write_kb":613954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1774,"num_bytes_recovered":14295040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1128767488,"data_stored":1128767488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76786,"ondisk_log_size":76786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611384,"kb_used_data":309360,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93760456,"statfs":{"total":96636764160,"available":96010706944,"internally_reserved":0,"allocated":316784640,"data_stored":316625662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,1,1,2,0,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":3.4420000000000002},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.4649999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":3.2610000000000001},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.23999999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":10.24},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":5.9779999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609632,"kb_used_data":303960,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93762208,"statfs":{"total":96636764160,"available":96012500992,"internally_reserved":0,"allocated":311255040,"data_stored":311096062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":3.79},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.7570000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":18.646999999999998},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":3.8250000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":8.6050000000000004},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.877}]}]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489360,"kb_used_data":183240,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93882480,"statfs":{"total":96636764160,"available":96135659520,"internally_reserved":0,"allocated":187637760,"data_stored":187478782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":3.7400000000000002},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":17.605}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":13.949},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":11.851000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":5.7519999999999998},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.86199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606852,"kb_used_data":304892,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93764988,"statfs":{"total":96636764160,"available":96015347712,"internally_reserved":0,"allocated":312209408,"data_stored":312050430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,9,17,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":9.5679999999999996},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":4.9909999999999997},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.63300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":9.5289999999999999},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":4.4989999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312201216,"data_stored":312201216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187613184,"data_stored":187613184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312078336,"data_stored":312078336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316874752,"data_stored":316874752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:47.238 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 150 20 78063 78043 4.06383 2.27344 0.0666911 0.0384354 2024-09-18T11:46:48.238 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 151 20 78390 78370 4.05383 2.55469 0.0694102 0.0385302 2024-09-18T11:46:49.239 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 152 20 78719 78699 4.04407 2.57031 0.0923049 0.0386273 2024-09-18T11:46:50.156 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:50.239 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 153 20 79037 79017 4.03388 2.48438 0.110299 0.0387207 2024-09-18T11:46:50.719 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:50.719 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:50.737 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-18T11:46:49.878223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635856244,"num_objects":77624,"num_object_clones":0,"num_object_copies":310496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77651,"num_write_kb":621090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77678,"ondisk_log_size":77678,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2338292,"kb_used_data":1114196,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375149068,"statfs":{"total":386547056640,"available":384152645632,"internally_reserved":0,"allocated":1140936704,"data_stored":1140300792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,1,9,7,20,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26574848,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3244,"num_write_kb":25952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006058"},"pg_stats":[{"pgid":"1.f","version":"31'4952","reported_seq":4992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.797913+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:44.797913+0000","last_peered":"2024-09-18T11:46:44.797913+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:44.797913+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4952,"num_write_kb":39616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4899","reported_seq":4962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.265358+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:46.265358+0000","last_peered":"2024-09-18T11:46:46.265358+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:46.265358+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4899,"log_dups_size":0,"ondisk_log_size":4899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40097014,"num_objects":4895,"num_object_clones":0,"num_object_copies":19580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4897,"num_write_kb":39172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4802","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.255126+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:46.255126+0000","last_peered":"2024-09-18T11:46:46.255126+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:46.255126+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39311995,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4801,"num_write_kb":38398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4854","reported_seq":4916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.291379+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:46.291379+0000","last_peered":"2024-09-18T11:46:46.291379+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:46.291379+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728374,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":38805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2736}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4806","reported_seq":4861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.652758+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:44.652758+0000","last_peered":"2024-09-18T11:46:44.652758+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:44.652758+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343350,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4804,"num_write_kb":38429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4836","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.706229+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:44.706229+0000","last_peered":"2024-09-18T11:46:44.706229+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:44.706229+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39616512,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4915","reported_seq":4980,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.313949+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:46.313949+0000","last_peered":"2024-09-18T11:46:46.313949+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:46.313949+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211702,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":39284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4791","reported_seq":4854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.361576+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:46.361576+0000","last_peered":"2024-09-18T11:46:46.361576+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:46.361576+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4791,"log_dups_size":0,"ondisk_log_size":4791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39204086,"num_objects":4786,"num_object_clones":0,"num_object_copies":19144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4954","reported_seq":5012,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.774248+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:44.774248+0000","last_peered":"2024-09-18T11:46:44.774248+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:44.774248+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40511980,"num_objects":4946,"num_object_clones":0,"num_object_copies":19784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":39585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4730","reported_seq":4813,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.311372+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:46.311372+0000","last_peered":"2024-09-18T11:46:46.311372+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:46.311372+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4730,"log_dups_size":0,"ondisk_log_size":4730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38728950,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4728,"num_write_kb":37822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2753}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4841","reported_seq":4897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.792385+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:44.792385+0000","last_peered":"2024-09-18T11:46:44.792385+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:44.792385+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39612273,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4838,"num_write_kb":38692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4925","reported_seq":4962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.696730+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:44.696730+0000","last_peered":"2024-09-18T11:46:44.696730+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:44.696730+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40335995,"num_objects":4924,"num_object_clones":0,"num_object_copies":19696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4924,"num_write_kb":39391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4909","reported_seq":4985,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.229019+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:46.229019+0000","last_peered":"2024-09-18T11:46:46.229019+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:46.229019+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4909,"log_dups_size":0,"ondisk_log_size":4909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40204923,"num_objects":4908,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4908,"num_write_kb":39263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4864","reported_seq":4913,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.805568+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:44.805568+0000","last_peered":"2024-09-18T11:46:44.805568+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:44.805568+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39810294,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4865","reported_seq":8624,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.357177+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:46.357177+0000","last_peered":"2024-09-18T11:46:46.357177+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:46.357177+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4865,"log_dups_size":0,"ondisk_log_size":4865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39826678,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4863,"num_write_kb":38901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4607}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4735","reported_seq":4797,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.381283+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:46.381283+0000","last_peered":"2024-09-18T11:46:46.381283+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:46.381283+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745334,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":635856244,"num_objects":77624,"num_object_clones":0,"num_object_copies":310496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77651,"num_write_kb":621090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1141927936,"data_stored":1141927936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77678,"ondisk_log_size":77678,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611384,"kb_used_data":309360,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93760456,"statfs":{"total":96636764160,"available":96010706944,"internally_reserved":0,"allocated":316784640,"data_stored":316625662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,1,1,2,0,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":3.4420000000000002},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.4649999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":3.2610000000000001},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.23999999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":10.24},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":5.9779999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":620200,"kb_used_data":310368,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93751640,"statfs":{"total":96636764160,"available":96001679360,"internally_reserved":0,"allocated":317816832,"data_stored":317657854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":3.79},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.7570000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":18.646999999999998},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":3.8250000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":8.6050000000000004},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.877}]}]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489360,"kb_used_data":183240,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93882480,"statfs":{"total":96636764160,"available":96135659520,"internally_reserved":0,"allocated":187637760,"data_stored":187478782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":3.7400000000000002},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":17.605}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":13.949},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":11.851000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":5.7519999999999998},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.86199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617348,"kb_used_data":311228,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93754492,"statfs":{"total":96636764160,"available":96004599808,"internally_reserved":0,"allocated":318697472,"data_stored":318538494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,14,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":12.034000000000001},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.64300000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.70199999999999996},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":1.5649999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":318754816,"data_stored":318754816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187613184,"data_stored":187613184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318685184,"data_stored":318685184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316874752,"data_stored":316874752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:50.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:51.239 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 154 20 79328 79308 4.02245 2.27344 0.029011 0.0388281 2024-09-18T11:46:51.247 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:51.247 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:51.273 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-18T11:46:49.878223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635856244,"num_objects":77624,"num_object_clones":0,"num_object_copies":310496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77651,"num_write_kb":621090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77678,"ondisk_log_size":77678,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2338292,"kb_used_data":1114196,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375149068,"statfs":{"total":386547056640,"available":384152645632,"internally_reserved":0,"allocated":1140936704,"data_stored":1140300792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,1,9,7,20,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26574848,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3244,"num_write_kb":25952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006058"},"pg_stats":[{"pgid":"1.f","version":"31'4952","reported_seq":4992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.797913+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:44.797913+0000","last_peered":"2024-09-18T11:46:44.797913+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:44.797913+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4952,"num_write_kb":39616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4899","reported_seq":4962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.265358+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:46.265358+0000","last_peered":"2024-09-18T11:46:46.265358+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:46.265358+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4899,"log_dups_size":0,"ondisk_log_size":4899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40097014,"num_objects":4895,"num_object_clones":0,"num_object_copies":19580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4897,"num_write_kb":39172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4802","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.255126+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:46.255126+0000","last_peered":"2024-09-18T11:46:46.255126+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:46.255126+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39311995,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4801,"num_write_kb":38398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4854","reported_seq":4916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.291379+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:46.291379+0000","last_peered":"2024-09-18T11:46:46.291379+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:46.291379+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728374,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":38805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2736}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4806","reported_seq":4861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.652758+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:44.652758+0000","last_peered":"2024-09-18T11:46:44.652758+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:44.652758+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343350,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4804,"num_write_kb":38429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4836","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.706229+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:44.706229+0000","last_peered":"2024-09-18T11:46:44.706229+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:44.706229+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39616512,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4915","reported_seq":4980,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.313949+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:46.313949+0000","last_peered":"2024-09-18T11:46:46.313949+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:46.313949+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40211702,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":39284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4791","reported_seq":4854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.361576+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:46.361576+0000","last_peered":"2024-09-18T11:46:46.361576+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:46.361576+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4791,"log_dups_size":0,"ondisk_log_size":4791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39204086,"num_objects":4786,"num_object_clones":0,"num_object_copies":19144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4954","reported_seq":5012,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.774248+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:44.774248+0000","last_peered":"2024-09-18T11:46:44.774248+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:44.774248+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40511980,"num_objects":4946,"num_object_clones":0,"num_object_copies":19784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":39585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4730","reported_seq":4813,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.311372+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:46.311372+0000","last_peered":"2024-09-18T11:46:46.311372+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:46.311372+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4730,"log_dups_size":0,"ondisk_log_size":4730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38728950,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4728,"num_write_kb":37822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2753}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4841","reported_seq":4897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.792385+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:44.792385+0000","last_peered":"2024-09-18T11:46:44.792385+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:44.792385+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39612273,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4838,"num_write_kb":38692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4925","reported_seq":4962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.696730+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:44.696730+0000","last_peered":"2024-09-18T11:46:44.696730+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:44.696730+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40335995,"num_objects":4924,"num_object_clones":0,"num_object_copies":19696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4924,"num_write_kb":39391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4909","reported_seq":4985,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.229019+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:46.229019+0000","last_peered":"2024-09-18T11:46:46.229019+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:46.229019+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4909,"log_dups_size":0,"ondisk_log_size":4909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40204923,"num_objects":4908,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4908,"num_write_kb":39263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4864","reported_seq":4913,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:44.805568+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:44.805568+0000","last_peered":"2024-09-18T11:46:44.805568+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:44.805568+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39810294,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4865","reported_seq":8624,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.357177+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:46.357177+0000","last_peered":"2024-09-18T11:46:46.357177+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:46.357177+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4865,"log_dups_size":0,"ondisk_log_size":4865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39826678,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4863,"num_write_kb":38901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4607}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4735","reported_seq":4797,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:46.381283+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:46.381283+0000","last_peered":"2024-09-18T11:46:46.381283+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:46.381283+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745334,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":635856244,"num_objects":77624,"num_object_clones":0,"num_object_copies":310496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77651,"num_write_kb":621090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1141927936,"data_stored":1141927936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77678,"ondisk_log_size":77678,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611384,"kb_used_data":309360,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93760456,"statfs":{"total":96636764160,"available":96010706944,"internally_reserved":0,"allocated":316784640,"data_stored":316625662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,1,1,2,0,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":3.4420000000000002},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.4649999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":3.2610000000000001},{"interface":"front","average":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":14.941000000000001,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.23999999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":10.24},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":5.9779999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":620200,"kb_used_data":310368,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93751640,"statfs":{"total":96636764160,"available":96001679360,"internally_reserved":0,"allocated":317816832,"data_stored":317657854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":3.79},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.7570000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:45:45 2024","interfaces":[{"interface":"back","average":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":18.646999999999998},{"interface":"front","average":{"1min":3.6560000000000001,"5min":3.6560000000000001,"15min":3.6560000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.550000000000001,"5min":12.550000000000001,"15min":12.550000000000001},"last":3.8250000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":8.6050000000000004},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.877}]}]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489360,"kb_used_data":183240,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93882480,"statfs":{"total":96636764160,"available":96135659520,"internally_reserved":0,"allocated":187637760,"data_stored":187478782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":3.7400000000000002},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":17.605}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":13.949},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":11.851000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":5.7519999999999998},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.86199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617348,"kb_used_data":311228,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93754492,"statfs":{"total":96636764160,"available":96004599808,"internally_reserved":0,"allocated":318697472,"data_stored":318538494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,14,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.681,"5min":3.681,"15min":3.681},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.898999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":12.034000000000001},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.64300000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.70199999999999996},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":1.5649999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":318754816,"data_stored":318754816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187613184,"data_stored":187613184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318685184,"data_stored":318685184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316874752,"data_stored":316874752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:51.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:51.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4952", 'reported_seq': 4992, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:44.797913+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:44.797913+0000', 'last_peered': '2024-09-18T11:46:44.797913+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:44.797913+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4952, 'log_dups_size': 0, 'ondisk_log_size': 4952, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40566784, 'num_objects': 4952, 'num_object_clones': 0, 'num_object_copies': 19808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4952, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4952, 'num_write_kb': 39616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2822}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:51.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:51.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4899", 'reported_seq': 4962, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:46.265358+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:46.265358+0000', 'last_peered': '2024-09-18T11:46:46.265358+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:46.265358+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4899, 'log_dups_size': 0, 'ondisk_log_size': 4899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40097014, 'num_objects': 4895, 'num_object_clones': 0, 'num_object_copies': 19580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4895, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4897, 'num_write_kb': 39172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2793}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4802", 'reported_seq': 4879, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:46.255126+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:46.255126+0000', 'last_peered': '2024-09-18T11:46:46.255126+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:46.255126+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4802, 'log_dups_size': 0, 'ondisk_log_size': 4802, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39311995, 'num_objects': 4799, 'num_object_clones': 0, 'num_object_copies': 19196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4801, 'num_write_kb': 38398, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2745}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4854", 'reported_seq': 4916, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:46.291379+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:46.291379+0000', 'last_peered': '2024-09-18T11:46:46.291379+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:46.291379+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4854, 'log_dups_size': 0, 'ondisk_log_size': 4854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39728374, 'num_objects': 4850, 'num_object_clones': 0, 'num_object_copies': 19400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4852, 'num_write_kb': 38805, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2736}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4806", 'reported_seq': 4861, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:44.652758+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:44.652758+0000', 'last_peered': '2024-09-18T11:46:44.652758+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:44.652758+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4806, 'log_dups_size': 0, 'ondisk_log_size': 4806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39343350, 'num_objects': 4803, 'num_object_clones': 0, 'num_object_copies': 19212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4804, 'num_write_kb': 38429, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2721}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4836", 'reported_seq': 4879, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:44.706229+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:44.706229+0000', 'last_peered': '2024-09-18T11:46:44.706229+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:44.706229+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4836, 'log_dups_size': 0, 'ondisk_log_size': 4836, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39616512, 'num_objects': 4836, 'num_object_clones': 0, 'num_object_copies': 19344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4836, 'num_write_kb': 38688, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2682}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4915", 'reported_seq': 4980, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:46.313949+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:46.313949+0000', 'last_peered': '2024-09-18T11:46:46.313949+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:46.313949+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4915, 'log_dups_size': 0, 'ondisk_log_size': 4915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40211702, 'num_objects': 4909, 'num_object_clones': 0, 'num_object_copies': 19636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4913, 'num_write_kb': 39284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2775}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:51.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4791", 'reported_seq': 4854, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:46.361576+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:46.361576+0000', 'last_peered': '2024-09-18T11:46:46.361576+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:46.361576+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4791, 'log_dups_size': 0, 'ondisk_log_size': 4791, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39204086, 'num_objects': 4786, 'num_object_clones': 0, 'num_object_copies': 19144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4789, 'num_write_kb': 38300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4954", 'reported_seq': 5012, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:44.774248+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:44.774248+0000', 'last_peered': '2024-09-18T11:46:44.774248+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:44.774248+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4954, 'log_dups_size': 0, 'ondisk_log_size': 4954, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40511980, 'num_objects': 4946, 'num_object_clones': 0, 'num_object_copies': 19784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4951, 'num_write_kb': 39585, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4730", 'reported_seq': 4813, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:46.311372+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:46.311372+0000', 'last_peered': '2024-09-18T11:46:46.311372+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:46.311372+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4730, 'log_dups_size': 0, 'ondisk_log_size': 4730, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38728950, 'num_objects': 4728, 'num_object_clones': 0, 'num_object_copies': 18912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4728, 'num_write_kb': 37822, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2753}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4841", 'reported_seq': 4897, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:44.792385+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:44.792385+0000', 'last_peered': '2024-09-18T11:46:44.792385+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:44.792385+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4841, 'log_dups_size': 0, 'ondisk_log_size': 4841, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39612273, 'num_objects': 4836, 'num_object_clones': 0, 'num_object_copies': 19344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4838, 'num_write_kb': 38692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2773}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4925", 'reported_seq': 4962, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:44.696730+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:44.696730+0000', 'last_peered': '2024-09-18T11:46:44.696730+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:44.696730+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4925, 'log_dups_size': 0, 'ondisk_log_size': 4925, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40335995, 'num_objects': 4924, 'num_object_clones': 0, 'num_object_copies': 19696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4924, 'num_write_kb': 39391, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:51.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4909", 'reported_seq': 4985, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:46.229019+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:46.229019+0000', 'last_peered': '2024-09-18T11:46:46.229019+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:46.229019+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4909, 'log_dups_size': 0, 'ondisk_log_size': 4909, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40204923, 'num_objects': 4908, 'num_object_clones': 0, 'num_object_copies': 19632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4908, 'num_write_kb': 39263, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2854}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:51.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:51.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4864", 'reported_seq': 4913, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:44.805568+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:44.805568+0000', 'last_peered': '2024-09-18T11:46:44.805568+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:44.805568+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4864, 'log_dups_size': 0, 'ondisk_log_size': 4864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39810294, 'num_objects': 4860, 'num_object_clones': 0, 'num_object_copies': 19440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4862, 'num_write_kb': 38892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:51.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:51.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4865", 'reported_seq': 8624, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:46.357177+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:46.357177+0000', 'last_peered': '2024-09-18T11:46:46.357177+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:46.357177+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4865, 'log_dups_size': 0, 'ondisk_log_size': 4865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39826678, 'num_objects': 4862, 'num_object_clones': 0, 'num_object_copies': 19448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 255, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4863, 'num_write_kb': 38901, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1840, 'num_bytes_recovered': 15040512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4607}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:51.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:51.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4735", 'reported_seq': 4797, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:46.381283+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:46.381283+0000', 'last_peered': '2024-09-18T11:46:46.381283+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:46.381283+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4735, 'log_dups_size': 0, 'ondisk_log_size': 4735, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38745334, 'num_objects': 4730, 'num_object_clones': 0, 'num_object_copies': 18920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4733, 'num_write_kb': 37852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2717}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:51.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:51.935 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:51.935 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31591/310496 objects degraded (10.174%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":210,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77624,"data_bytes":635856244,"bytes_used":2394411008,"bytes_avail":384152645632,"bytes_total":386547056640,"degraded_objects":31591,"degraded_total":310496,"degraded_ratio":0.10174366175409669,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":124182,"recovering_keys_per_sec":0,"num_objects_recovered":182,"num_bytes_recovered":1490944,"num_keys_recovered":0,"write_bytes_sec":2213453,"read_op_per_sec":0,"write_op_per_sec":270},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:51.955 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:51.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:52.239 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 155 20 79609 79589 4.01066 2.19531 0.0578961 0.0389433 2024-09-18T11:46:52.500 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:52.500 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:52.535 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-18T11:46:51.878970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648709492,"num_objects":79193,"num_object_clones":0,"num_object_copies":316772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79220,"num_write_kb":633642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":15851520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79247,"ondisk_log_size":79247,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2363412,"kb_used_data":1139316,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375123948,"statfs":{"total":386547056640,"available":384126922752,"internally_reserved":0,"allocated":1166659584,"data_stored":1166023672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,16,17,19,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33464320,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4085,"num_write_kb":32680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":2301952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005790"},"pg_stats":[{"pgid":"1.f","version":"31'5065","reported_seq":5106,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:49.760860+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:49.760860+0000","last_peered":"2024-09-18T11:46:49.760860+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:49.760860+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41492480,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5065,"num_write_kb":40520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5009","reported_seq":5071,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.382084+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:51.382084+0000","last_peered":"2024-09-18T11:46:51.382084+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:51.382084+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40998134,"num_objects":5005,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":40052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4895","reported_seq":4977,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.296569+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:51.296569+0000","last_peered":"2024-09-18T11:46:51.296569+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:51.296569+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40073851,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4894,"num_write_kb":39142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4936","reported_seq":4998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.233435+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:51.233435+0000","last_peered":"2024-09-18T11:46:51.233435+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:51.233435+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40400118,"num_objects":4932,"num_object_clones":0,"num_object_copies":19728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4934,"num_write_kb":39461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4904","reported_seq":4958,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:49.767794+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:49.767794+0000","last_peered":"2024-09-18T11:46:49.767794+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:49.767794+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4904,"log_dups_size":0,"ondisk_log_size":4904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40146166,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4902,"num_write_kb":39213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2819}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4940","reported_seq":4983,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:49.807677+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:49.807677+0000","last_peered":"2024-09-18T11:46:49.807677+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:49.807677+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4940,"log_dups_size":0,"ondisk_log_size":4940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40468480,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4940,"num_write_kb":39520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5007","reported_seq":5072,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.463637+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:51.463637+0000","last_peered":"2024-09-18T11:46:51.463637+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:51.463637+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40965366,"num_objects":5001,"num_object_clones":0,"num_object_copies":20004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2867}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4890","reported_seq":4952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.406116+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:51.406116+0000","last_peered":"2024-09-18T11:46:51.406116+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:51.406116+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40015094,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5056","reported_seq":5114,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:49.829695+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:49.829695+0000","last_peered":"2024-09-18T11:46:49.829695+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:49.829695+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5056,"log_dups_size":0,"ondisk_log_size":5056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41347564,"num_objects":5048,"num_object_clones":0,"num_object_copies":20192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5053,"num_write_kb":40401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2913}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4819","reported_seq":4903,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.232238+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:51.232238+0000","last_peered":"2024-09-18T11:46:51.232238+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:51.232238+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458038,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4817,"num_write_kb":38534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2842}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4921","reported_seq":4978,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:49.799717+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:49.799717+0000","last_peered":"2024-09-18T11:46:49.799717+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:49.799717+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40267633,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":39332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5042","reported_seq":5078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:49.848471+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:49.848471+0000","last_peered":"2024-09-18T11:46:49.848471+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:49.848471+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41294459,"num_objects":5041,"num_object_clones":0,"num_object_copies":20164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2932}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5005","reported_seq":5080,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.301793+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:51.301793+0000","last_peered":"2024-09-18T11:46:51.301793+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:51.301793+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40991355,"num_objects":5004,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2950}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4956","reported_seq":5005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:49.786305+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:49.786305+0000","last_peered":"2024-09-18T11:46:49.786305+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:49.786305+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40563958,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4954,"num_write_kb":39628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2794}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4970","reported_seq":8927,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.376400+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:51.376400+0000","last_peered":"2024-09-18T11:46:51.376400+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:51.376400+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40686838,"num_objects":4967,"num_object_clones":0,"num_object_copies":19868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":39741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":15851520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4832","reported_seq":4894,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.387173+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:51.387173+0000","last_peered":"2024-09-18T11:46:51.387173+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:51.387173+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39539958,"num_objects":4827,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4830,"num_write_kb":38628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":648709492,"num_objects":79193,"num_object_clones":0,"num_object_copies":316772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79220,"num_write_kb":633642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":15851520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1167929344,"data_stored":1167929344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79247,"ondisk_log_size":79247,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621552,"kb_used_data":315368,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93750288,"statfs":{"total":96636764160,"available":96000294912,"internally_reserved":0,"allocated":322936832,"data_stored":322777854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,2,2,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":2.2349999999999999},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.2629999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":2.29},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":2.8460000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":2.8149999999999999},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.33300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614012,"kb_used_data":316660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93757828,"statfs":{"total":96636764160,"available":96008015872,"internally_reserved":0,"allocated":324259840,"data_stored":324100862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.262}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":2.0539999999999998},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":3.9969999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":1.2350000000000001},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.3939999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500104,"kb_used_data":189824,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93871736,"statfs":{"total":96636764160,"available":96124657664,"internally_reserved":0,"allocated":194379776,"data_stored":194220798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,7,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.29,"15min":3.29},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.808999999999999},"last":3.2229999999999999},{"interface":"front","average":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":18.221,"5min":18.221,"15min":18.221},"last":1.8979999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":15.015000000000001,"5min":15.015000000000001,"15min":15.015000000000001},"last":2.673},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":22.084,"5min":22.084,"15min":22.084},"last":3.2599999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":5.2729999999999997},{"interface":"front","average":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.701000000000001,"5min":11.701000000000001,"15min":11.701000000000001},"last":3.988}]}]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627744,"kb_used_data":317464,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93744096,"statfs":{"total":96636764160,"available":95993954304,"internally_reserved":0,"allocated":325083136,"data_stored":324924158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,7,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":5.1740000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.221}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.33900000000000002},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325193728,"data_stored":325193728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":194392064,"data_stored":194392064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":325087232,"data_stored":325087232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323256320,"data_stored":323256320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:53.242 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 156 20 79931 79911 4.00099 2.51562 0.0886195 0.0390404 2024-09-18T11:46:54.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 157 20 80258 80238 3.99178 2.55469 0.0451477 0.0391321 2024-09-18T11:46:55.242 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 158 20 80582 80562 3.98253 2.53125 0.053388 0.0392216 2024-09-18T11:46:55.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:56.113 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:56.114 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:56.126 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-18T11:46:55.880733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654239092,"num_objects":79868,"num_object_clones":0,"num_object_copies":319472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79895,"num_write_kb":639042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":15851520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79922,"ondisk_log_size":79922,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2367788,"kb_used_data":1152008,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":375119572,"statfs":{"total":386547056640,"available":384122441728,"internally_reserved":0,"allocated":1179656192,"data_stored":1179020280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,6,20,15,15,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31326208,"num_objects":3824,"num_object_clones":0,"num_object_copies":15296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3824,"num_write_kb":30592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005600"},"pg_stats":[{"pgid":"1.f","version":"31'5169","reported_seq":5209,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.882737+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:54.882737+0000","last_peered":"2024-09-18T11:46:54.882737+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:54.882737+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5169,"log_dups_size":0,"ondisk_log_size":5169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42344448,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":41352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5009","reported_seq":5071,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.382084+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:51.382084+0000","last_peered":"2024-09-18T11:46:51.382084+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:51.382084+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40998134,"num_objects":5005,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":40052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4895","reported_seq":4977,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.296569+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:51.296569+0000","last_peered":"2024-09-18T11:46:51.296569+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:51.296569+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40073851,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4894,"num_write_kb":39142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4936","reported_seq":4998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.233435+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:51.233435+0000","last_peered":"2024-09-18T11:46:51.233435+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:51.233435+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40400118,"num_objects":4932,"num_object_clones":0,"num_object_copies":19728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4934,"num_write_kb":39461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4991","reported_seq":5046,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.765465+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:54.765465+0000","last_peered":"2024-09-18T11:46:54.765465+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:54.765465+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40858870,"num_objects":4988,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4989,"num_write_kb":39909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5050","reported_seq":5093,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.815445+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:54.815445+0000","last_peered":"2024-09-18T11:46:54.815445+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:54.815445+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41369600,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":40400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5007","reported_seq":5072,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.463637+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:51.463637+0000","last_peered":"2024-09-18T11:46:51.463637+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:51.463637+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40965366,"num_objects":5001,"num_object_clones":0,"num_object_copies":20004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2867}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4890","reported_seq":4952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.406116+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:51.406116+0000","last_peered":"2024-09-18T11:46:51.406116+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:51.406116+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40015094,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5151","reported_seq":5209,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.829969+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:54.829969+0000","last_peered":"2024-09-18T11:46:54.829969+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:54.829969+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42125804,"num_objects":5143,"num_object_clones":0,"num_object_copies":20572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":41161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4819","reported_seq":4903,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.232238+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:51.232238+0000","last_peered":"2024-09-18T11:46:51.232238+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:51.232238+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458038,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4817,"num_write_kb":38534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2842}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5021","reported_seq":5078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.807609+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:54.807609+0000","last_peered":"2024-09-18T11:46:54.807609+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:54.807609+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41086833,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":40132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2953}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5134","reported_seq":5171,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.820321+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:54.820321+0000","last_peered":"2024-09-18T11:46:54.820321+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:54.820321+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5134,"log_dups_size":0,"ondisk_log_size":5134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42048123,"num_objects":5133,"num_object_clones":0,"num_object_copies":20532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5133,"num_write_kb":41063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3024}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5005","reported_seq":5080,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.301793+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:51.301793+0000","last_peered":"2024-09-18T11:46:51.301793+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:51.301793+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40991355,"num_objects":5004,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2950}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5043","reported_seq":5092,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.769246+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:54.769246+0000","last_peered":"2024-09-18T11:46:54.769246+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:54.769246+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41276662,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2881}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4970","reported_seq":8927,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.376400+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:51.376400+0000","last_peered":"2024-09-18T11:46:51.376400+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:51.376400+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40686838,"num_objects":4967,"num_object_clones":0,"num_object_copies":19868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":39741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":15851520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4832","reported_seq":4894,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.387173+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:51.387173+0000","last_peered":"2024-09-18T11:46:51.387173+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:51.387173+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39539958,"num_objects":4827,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4830,"num_write_kb":38628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":654239092,"num_objects":79868,"num_object_clones":0,"num_object_copies":319472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79895,"num_write_kb":639042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":15851520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181097984,"data_stored":1181097984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79922,"ondisk_log_size":79922,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631864,"kb_used_data":321520,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93739976,"statfs":{"total":96636764160,"available":95989735424,"internally_reserved":0,"allocated":329236480,"data_stored":329077502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":10.775},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":10.807}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":10.852},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.028}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":3.637},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":14.470000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614012,"kb_used_data":316660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93757828,"statfs":{"total":96636764160,"available":96008015872,"internally_reserved":0,"allocated":324259840,"data_stored":324100862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.262}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":2.0539999999999998},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":3.9969999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":1.2350000000000001},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.3939999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494168,"kb_used_data":196364,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93877672,"statfs":{"total":96636764160,"available":96130736128,"internally_reserved":0,"allocated":201076736,"data_stored":200917758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,4,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.375},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.2439999999999998},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":1.1499999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.55900000000000005},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":5.7569999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627744,"kb_used_data":317464,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93744096,"statfs":{"total":96636764160,"available":95993954304,"internally_reserved":0,"allocated":325083136,"data_stored":324924158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,7,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":5.1740000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.221}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.33900000000000002},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325193728,"data_stored":325193728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":201207808,"data_stored":201207808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":325087232,"data_stored":325087232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":329609216,"data_stored":329609216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:56.128 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:56.242 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 159 20 80879 80859 3.97208 2.32031 0.0487204 0.0393193 2024-09-18T11:46:56.694 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:56.694 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:56.724 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-18T11:46:55.880733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654239092,"num_objects":79868,"num_object_clones":0,"num_object_copies":319472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79895,"num_write_kb":639042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":15851520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79922,"ondisk_log_size":79922,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2367788,"kb_used_data":1152008,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":375119572,"statfs":{"total":386547056640,"available":384122441728,"internally_reserved":0,"allocated":1179656192,"data_stored":1179020280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,6,20,15,15,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31326208,"num_objects":3824,"num_object_clones":0,"num_object_copies":15296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3824,"num_write_kb":30592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005600"},"pg_stats":[{"pgid":"1.f","version":"31'5169","reported_seq":5209,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.882737+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:54.882737+0000","last_peered":"2024-09-18T11:46:54.882737+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:54.882737+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5169,"log_dups_size":0,"ondisk_log_size":5169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42344448,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":41352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5009","reported_seq":5071,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.382084+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:51.382084+0000","last_peered":"2024-09-18T11:46:51.382084+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:51.382084+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40998134,"num_objects":5005,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":40052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4895","reported_seq":4977,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.296569+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:51.296569+0000","last_peered":"2024-09-18T11:46:51.296569+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:51.296569+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40073851,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4894,"num_write_kb":39142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4936","reported_seq":4998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.233435+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:51.233435+0000","last_peered":"2024-09-18T11:46:51.233435+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:51.233435+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40400118,"num_objects":4932,"num_object_clones":0,"num_object_copies":19728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4934,"num_write_kb":39461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4991","reported_seq":5046,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.765465+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:54.765465+0000","last_peered":"2024-09-18T11:46:54.765465+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:54.765465+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40858870,"num_objects":4988,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4989,"num_write_kb":39909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5050","reported_seq":5093,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.815445+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:54.815445+0000","last_peered":"2024-09-18T11:46:54.815445+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:54.815445+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41369600,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":40400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5007","reported_seq":5072,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.463637+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:51.463637+0000","last_peered":"2024-09-18T11:46:51.463637+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:51.463637+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40965366,"num_objects":5001,"num_object_clones":0,"num_object_copies":20004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2867}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4890","reported_seq":4952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.406116+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:51.406116+0000","last_peered":"2024-09-18T11:46:51.406116+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:51.406116+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40015094,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5151","reported_seq":5209,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.829969+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:54.829969+0000","last_peered":"2024-09-18T11:46:54.829969+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:54.829969+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42125804,"num_objects":5143,"num_object_clones":0,"num_object_copies":20572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":41161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4819","reported_seq":4903,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.232238+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:51.232238+0000","last_peered":"2024-09-18T11:46:51.232238+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:51.232238+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458038,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4817,"num_write_kb":38534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2842}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5021","reported_seq":5078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.807609+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:54.807609+0000","last_peered":"2024-09-18T11:46:54.807609+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:54.807609+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41086833,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":40132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2953}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5134","reported_seq":5171,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.820321+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:54.820321+0000","last_peered":"2024-09-18T11:46:54.820321+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:54.820321+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5134,"log_dups_size":0,"ondisk_log_size":5134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42048123,"num_objects":5133,"num_object_clones":0,"num_object_copies":20532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5133,"num_write_kb":41063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3024}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5005","reported_seq":5080,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.301793+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:51.301793+0000","last_peered":"2024-09-18T11:46:51.301793+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:51.301793+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40991355,"num_objects":5004,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2950}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5043","reported_seq":5092,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.769246+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:54.769246+0000","last_peered":"2024-09-18T11:46:54.769246+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:54.769246+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41276662,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2881}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4970","reported_seq":8927,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.376400+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:51.376400+0000","last_peered":"2024-09-18T11:46:51.376400+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:51.376400+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40686838,"num_objects":4967,"num_object_clones":0,"num_object_copies":19868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":39741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":15851520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4832","reported_seq":4894,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:51.387173+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:51.387173+0000","last_peered":"2024-09-18T11:46:51.387173+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:51.387173+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39539958,"num_objects":4827,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4830,"num_write_kb":38628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":654239092,"num_objects":79868,"num_object_clones":0,"num_object_copies":319472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79895,"num_write_kb":639042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":15851520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181097984,"data_stored":1181097984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79922,"ondisk_log_size":79922,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631864,"kb_used_data":321520,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93739976,"statfs":{"total":96636764160,"available":95989735424,"internally_reserved":0,"allocated":329236480,"data_stored":329077502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":10.775},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":10.807}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":10.852},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.028}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":3.637},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":14.470000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614012,"kb_used_data":316660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93757828,"statfs":{"total":96636764160,"available":96008015872,"internally_reserved":0,"allocated":324259840,"data_stored":324100862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.262}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":2.0539999999999998},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":3.9969999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":1.2350000000000001},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.3939999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494168,"kb_used_data":196364,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93877672,"statfs":{"total":96636764160,"available":96130736128,"internally_reserved":0,"allocated":201076736,"data_stored":200917758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,4,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.375},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.2439999999999998},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":1.1499999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.55900000000000005},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":5.7569999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627744,"kb_used_data":317464,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93744096,"statfs":{"total":96636764160,"available":95993954304,"internally_reserved":0,"allocated":325083136,"data_stored":324924158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,7,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":5.1740000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":0.221}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.33900000000000002},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325193728,"data_stored":325193728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":201207808,"data_stored":201207808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":325087232,"data_stored":325087232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":329609216,"data_stored":329609216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:56.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:46:56.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5169", 'reported_seq': 5209, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:54.882737+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:54.882737+0000', 'last_peered': '2024-09-18T11:46:54.882737+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:54.882737+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5169, 'log_dups_size': 0, 'ondisk_log_size': 5169, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42344448, 'num_objects': 5169, 'num_object_clones': 0, 'num_object_copies': 20676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5169, 'num_write_kb': 41352, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3039}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:56.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:46:56.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5009", 'reported_seq': 5071, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:51.382084+0000', 'last_change': '2024-09-18T11:44:47.410196+0000', 'last_active': '2024-09-18T11:46:51.382084+0000', 'last_peered': '2024-09-18T11:46:51.382084+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:46:51.382084+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5009, 'log_dups_size': 0, 'ondisk_log_size': 5009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40998134, 'num_objects': 5005, 'num_object_clones': 0, 'num_object_copies': 20020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5007, 'num_write_kb': 40052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2903}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:56.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:46:56.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4895", 'reported_seq': 4977, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:51.296569+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:46:51.296569+0000', 'last_peered': '2024-09-18T11:46:51.296569+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:46:51.296569+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4895, 'log_dups_size': 0, 'ondisk_log_size': 4895, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40073851, 'num_objects': 4892, 'num_object_clones': 0, 'num_object_copies': 19568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4894, 'num_write_kb': 39142, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2838}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:46:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4936", 'reported_seq': 4998, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:51.233435+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:46:51.233435+0000', 'last_peered': '2024-09-18T11:46:51.233435+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:46:51.233435+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4936, 'log_dups_size': 0, 'ondisk_log_size': 4936, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40400118, 'num_objects': 4932, 'num_object_clones': 0, 'num_object_copies': 19728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4934, 'num_write_kb': 39461, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2818}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:46:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4991", 'reported_seq': 5046, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:54.765465+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:54.765465+0000', 'last_peered': '2024-09-18T11:46:54.765465+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:54.765465+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4991, 'log_dups_size': 0, 'ondisk_log_size': 4991, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40858870, 'num_objects': 4988, 'num_object_clones': 0, 'num_object_copies': 19952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4989, 'num_write_kb': 39909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2906}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:46:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5050", 'reported_seq': 5093, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:54.815445+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:54.815445+0000', 'last_peered': '2024-09-18T11:46:54.815445+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:54.815445+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5050, 'log_dups_size': 0, 'ondisk_log_size': 5050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41369600, 'num_objects': 5050, 'num_object_clones': 0, 'num_object_copies': 20200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5050, 'num_write_kb': 40400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2896}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:46:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5007", 'reported_seq': 5072, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:51.463637+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:46:51.463637+0000', 'last_peered': '2024-09-18T11:46:51.463637+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:46:51.463637+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5007, 'log_dups_size': 0, 'ondisk_log_size': 5007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40965366, 'num_objects': 5001, 'num_object_clones': 0, 'num_object_copies': 20004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5005, 'num_write_kb': 40020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2867}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:46:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4890", 'reported_seq': 4952, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:51.406116+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:46:51.406116+0000', 'last_peered': '2024-09-18T11:46:51.406116+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:46:51.406116+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4890, 'log_dups_size': 0, 'ondisk_log_size': 4890, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40015094, 'num_objects': 4885, 'num_object_clones': 0, 'num_object_copies': 19540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4888, 'num_write_kb': 39092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2851}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5151", 'reported_seq': 5209, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:54.829969+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:54.829969+0000', 'last_peered': '2024-09-18T11:46:54.829969+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:54.829969+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5151, 'log_dups_size': 0, 'ondisk_log_size': 5151, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42125804, 'num_objects': 5143, 'num_object_clones': 0, 'num_object_copies': 20572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5148, 'num_write_kb': 41161, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3008}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4819", 'reported_seq': 4903, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:51.232238+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:46:51.232238+0000', 'last_peered': '2024-09-18T11:46:51.232238+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:46:51.232238+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4819, 'log_dups_size': 0, 'ondisk_log_size': 4819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39458038, 'num_objects': 4817, 'num_object_clones': 0, 'num_object_copies': 19268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4817, 'num_write_kb': 38534, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2842}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5021", 'reported_seq': 5078, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:54.807609+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:54.807609+0000', 'last_peered': '2024-09-18T11:46:54.807609+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:54.807609+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5021, 'log_dups_size': 0, 'ondisk_log_size': 5021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41086833, 'num_objects': 5016, 'num_object_clones': 0, 'num_object_copies': 20064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5018, 'num_write_kb': 40132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2953}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5134", 'reported_seq': 5171, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:54.820321+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:54.820321+0000', 'last_peered': '2024-09-18T11:46:54.820321+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:54.820321+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5134, 'log_dups_size': 0, 'ondisk_log_size': 5134, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42048123, 'num_objects': 5133, 'num_object_clones': 0, 'num_object_copies': 20532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5133, 'num_write_kb': 41063, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5005", 'reported_seq': 5080, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:51.301793+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:46:51.301793+0000', 'last_peered': '2024-09-18T11:46:51.301793+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:46:51.301793+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5005, 'log_dups_size': 0, 'ondisk_log_size': 5005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40991355, 'num_objects': 5004, 'num_object_clones': 0, 'num_object_copies': 20016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5004, 'num_write_kb': 40031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2950}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:46:56.729 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5043", 'reported_seq': 5092, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:54.769246+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:54.769246+0000', 'last_peered': '2024-09-18T11:46:54.769246+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:54.769246+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5043, 'log_dups_size': 0, 'ondisk_log_size': 5043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41276662, 'num_objects': 5039, 'num_object_clones': 0, 'num_object_copies': 20156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5041, 'num_write_kb': 40324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:46:56.729 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:46:56.729 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4970", 'reported_seq': 8927, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:51.376400+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:51.376400+0000', 'last_peered': '2024-09-18T11:46:51.376400+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:51.376400+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4970, 'log_dups_size': 0, 'ondisk_log_size': 4970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40686838, 'num_objects': 4967, 'num_object_clones': 0, 'num_object_copies': 19868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 156, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4968, 'num_write_kb': 39741, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1939, 'num_bytes_recovered': 15851520, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4811}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:46:56.729 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:46:56.729 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4832", 'reported_seq': 4894, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:51.387173+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:46:51.387173+0000', 'last_peered': '2024-09-18T11:46:51.387173+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:46:51.387173+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 4832, 'log_dups_size': 0, 'ondisk_log_size': 4832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39539958, 'num_objects': 4827, 'num_object_clones': 0, 'num_object_copies': 19308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4830, 'num_write_kb': 38628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2814}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:46:56.729 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:46:57.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:46:57.242433+0000 min lat: 0.00272854 max lat: 2.42146 avg lat: 0.0394283 2024-09-18T11:46:57.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:46:57.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 160 20 81170 81150 3.96146 2.27344 0.0698157 0.0394283 2024-09-18T11:46:57.466 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:57.467 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31492/319472 objects degraded (9.858%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":215,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":79868,"data_bytes":654239092,"bytes_used":2424614912,"bytes_avail":384122441728,"bytes_total":386547056640,"degraded_objects":31492,"degraded_total":319472,"degraded_ratio":0.098575148995843145,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":129646,"recovering_keys_per_sec":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"write_bytes_sec":2609299,"read_op_per_sec":0,"write_op_per_sec":318},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:46:57.497 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:46:57.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:46:58.072 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:46:58.072 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:46:58.098 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-18T11:46:57.881677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661259636,"num_objects":80725,"num_object_clones":0,"num_object_copies":322900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80752,"num_write_kb":645898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2054,"num_bytes_recovered":16588800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80779,"ondisk_log_size":80779,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2372860,"kb_used_data":1165400,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375114500,"statfs":{"total":386547056640,"available":384117248000,"internally_reserved":0,"allocated":1193369600,"data_stored":1192733688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,6,19,9,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32710656,"num_objects":3993,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3993,"num_write_kb":31944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005468"},"pg_stats":[{"pgid":"1.f","version":"31'5169","reported_seq":5209,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.882737+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:54.882737+0000","last_peered":"2024-09-18T11:46:54.882737+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:54.882737+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5169,"log_dups_size":0,"ondisk_log_size":5169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42344448,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":41352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5102","reported_seq":5165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.356692+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:56.356692+0000","last_peered":"2024-09-18T11:46:56.356692+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:56.356692+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41759990,"num_objects":5098,"num_object_clones":0,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5100,"num_write_kb":40796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2996}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4991","reported_seq":5073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.377949+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:56.377949+0000","last_peered":"2024-09-18T11:46:56.377949+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:56.377949+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40860283,"num_objects":4988,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4990,"num_write_kb":39910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5020","reported_seq":5082,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.268648+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:56.268648+0000","last_peered":"2024-09-18T11:46:56.268648+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:56.268648+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41088246,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":40133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4991","reported_seq":5046,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.765465+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:54.765465+0000","last_peered":"2024-09-18T11:46:54.765465+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:54.765465+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40858870,"num_objects":4988,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4989,"num_write_kb":39909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5050","reported_seq":5093,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.815445+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:54.815445+0000","last_peered":"2024-09-18T11:46:54.815445+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:54.815445+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41369600,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":40400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5107","reported_seq":5172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.457191+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:56.457191+0000","last_peered":"2024-09-18T11:46:56.457191+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:56.457191+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5107,"log_dups_size":0,"ondisk_log_size":5107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41784566,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":40820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2967}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4982","reported_seq":5045,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.414593+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:56.414593+0000","last_peered":"2024-09-18T11:46:56.414593+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:56.414593+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768758,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":39828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5151","reported_seq":5209,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.829969+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:54.829969+0000","last_peered":"2024-09-18T11:46:54.829969+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:54.829969+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42125804,"num_objects":5143,"num_object_clones":0,"num_object_copies":20572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":41161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4913","reported_seq":4997,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.158932+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:56.158932+0000","last_peered":"2024-09-18T11:46:56.158932+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:56.158932+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40228086,"num_objects":4911,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4911,"num_write_kb":39286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5021","reported_seq":5078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.807609+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:54.807609+0000","last_peered":"2024-09-18T11:46:54.807609+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:54.807609+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41086833,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":40132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2953}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5134","reported_seq":5171,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.820321+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:54.820321+0000","last_peered":"2024-09-18T11:46:54.820321+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:54.820321+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5134,"log_dups_size":0,"ondisk_log_size":5134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42048123,"num_objects":5133,"num_object_clones":0,"num_object_copies":20532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5133,"num_write_kb":41063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3024}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5104","reported_seq":5180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.212064+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:56.212064+0000","last_peered":"2024-09-18T11:46:56.212064+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:56.212064+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5104,"log_dups_size":0,"ondisk_log_size":5104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41802363,"num_objects":5103,"num_object_clones":0,"num_object_copies":20412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5103,"num_write_kb":40823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3049}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5043","reported_seq":5092,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:54.769246+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:54.769246+0000","last_peered":"2024-09-18T11:46:54.769246+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:54.769246+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41276662,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2881}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5072","reported_seq":9209,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.329501+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:56.329501+0000","last_peered":"2024-09-18T11:46:56.329501+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:56.329501+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41522422,"num_objects":5069,"num_object_clones":0,"num_object_copies":20276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2029,"num_bytes_recovered":16588800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5003}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4929","reported_seq":4991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.468500+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:56.468500+0000","last_peered":"2024-09-18T11:46:56.468500+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:56.468500+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40334582,"num_objects":4924,"num_object_clones":0,"num_object_copies":19696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4927,"num_write_kb":39404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":661259636,"num_objects":80725,"num_object_clones":0,"num_object_copies":322900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80752,"num_write_kb":645898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2054,"num_bytes_recovered":16588800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1193897984,"data_stored":1193897984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80779,"ondisk_log_size":80779,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631864,"kb_used_data":321520,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93739976,"statfs":{"total":96636764160,"available":95989735424,"internally_reserved":0,"allocated":329236480,"data_stored":329077502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":10.775},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":10.807}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":10.852},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.028}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":3.637},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":14.470000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625436,"kb_used_data":323924,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93746404,"statfs":{"total":96636764160,"available":95996317696,"internally_reserved":0,"allocated":331698176,"data_stored":331539198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.6240000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":2.98},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":2.6619999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":2.6190000000000002},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.238}]}]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494168,"kb_used_data":196364,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93877672,"statfs":{"total":96636764160,"available":96130736128,"internally_reserved":0,"allocated":201076736,"data_stored":200917758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,4,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.375},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.2439999999999998},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":1.1499999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.55900000000000005},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":5.7569999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621392,"kb_used_data":323592,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93750448,"statfs":{"total":96636764160,"available":96000458752,"internally_reserved":0,"allocated":331358208,"data_stored":331199230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":4.7590000000000003},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":2.1080000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":8.548},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":5.4109999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.88100000000000001},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":2.1389999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331612160,"data_stored":331612160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":201207808,"data_stored":201207808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331468800,"data_stored":331468800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":329609216,"data_stored":329609216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:46:58.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 161 20 81462 81442 3.95103 2.28125 0.0593413 0.0395341 2024-09-18T11:46:59.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 162 20 81785 81765 3.94222 2.52344 0.0953308 0.0396217 2024-09-18T11:47:00.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 163 20 82102 82082 3.93322 2.47656 0.0756983 0.0397122 2024-09-18T11:47:01.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:01.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 164 20 82407 82387 3.92377 2.38281 0.0318866 0.0398033 2024-09-18T11:47:01.692 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:01.692 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:01.713 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-09-18T11:47:01.048709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666805620,"num_objects":81402,"num_object_clones":0,"num_object_copies":325608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81429,"num_write_kb":651314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2054,"num_bytes_recovered":16588800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81456,"ondisk_log_size":81456,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2377352,"kb_used_data":1178212,"kb_used_omap":14,"kb_used_meta":1199089,"kb_avail":375110008,"statfs":{"total":386547056640,"available":384112648192,"internally_reserved":0,"allocated":1206489088,"data_stored":1205853176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227867163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,3,8,25,6,7,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30949376,"num_objects":3778,"num_object_clones":0,"num_object_copies":15112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3778,"num_write_kb":30224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":1548288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.170485"},"pg_stats":[{"pgid":"1.f","version":"31'5269","reported_seq":5310,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.915955+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:59.915955+0000","last_peered":"2024-09-18T11:46:59.915955+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:59.915955+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43163648,"num_objects":5269,"num_object_clones":0,"num_object_copies":21076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":42152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5102","reported_seq":5165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.356692+0000","last_change":"2024-09-18T11:44:47.410196+0000","last_active":"2024-09-18T11:46:56.356692+0000","last_peered":"2024-09-18T11:46:56.356692+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:46:56.356692+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41759990,"num_objects":5098,"num_object_clones":0,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5100,"num_write_kb":40796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2996}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4991","reported_seq":5073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.377949+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:46:56.377949+0000","last_peered":"2024-09-18T11:46:56.377949+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:46:56.377949+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40860283,"num_objects":4988,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4990,"num_write_kb":39910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5020","reported_seq":5082,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.268648+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:46:56.268648+0000","last_peered":"2024-09-18T11:46:56.268648+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:46:56.268648+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41088246,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":40133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5110","reported_seq":5165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.908322+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:59.908322+0000","last_peered":"2024-09-18T11:46:59.908322+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:59.908322+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5110,"log_dups_size":0,"ondisk_log_size":5110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41833718,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":40861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5142","reported_seq":5185,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.810345+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:59.810345+0000","last_peered":"2024-09-18T11:46:59.810345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:59.810345+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5142,"log_dups_size":0,"ondisk_log_size":5142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42123264,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5142,"num_write_kb":41136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5107","reported_seq":5172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.457191+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:46:56.457191+0000","last_peered":"2024-09-18T11:46:56.457191+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:46:56.457191+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5107,"log_dups_size":0,"ondisk_log_size":5107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41784566,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":40820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2967}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4982","reported_seq":5045,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.414593+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:46:56.414593+0000","last_peered":"2024-09-18T11:46:56.414593+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:46:56.414593+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768758,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":39828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5241","reported_seq":5299,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.822372+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:59.822372+0000","last_peered":"2024-09-18T11:46:59.822372+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:59.822372+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5241,"log_dups_size":0,"ondisk_log_size":5241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42863084,"num_objects":5233,"num_object_clones":0,"num_object_copies":20932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5238,"num_write_kb":41881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4913","reported_seq":4997,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.158932+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:46:56.158932+0000","last_peered":"2024-09-18T11:46:56.158932+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:46:56.158932+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40228086,"num_objects":4911,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4911,"num_write_kb":39286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5115","reported_seq":5171,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.935353+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:59.935353+0000","last_peered":"2024-09-18T11:46:59.935353+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:59.935353+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41856881,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":40884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5231","reported_seq":5266,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.860631+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:59.860631+0000","last_peered":"2024-09-18T11:46:59.860631+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:59.860631+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42842747,"num_objects":5230,"num_object_clones":0,"num_object_copies":20920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5230,"num_write_kb":41839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5104","reported_seq":5180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.212064+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:46:56.212064+0000","last_peered":"2024-09-18T11:46:56.212064+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:46:56.212064+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5104,"log_dups_size":0,"ondisk_log_size":5104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41802363,"num_objects":5103,"num_object_clones":0,"num_object_copies":20412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5103,"num_write_kb":40823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3049}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5128","reported_seq":5177,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.909713+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:59.909713+0000","last_peered":"2024-09-18T11:46:59.909713+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:59.909713+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5128,"log_dups_size":0,"ondisk_log_size":5128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41972982,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":41004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5072","reported_seq":9209,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.329501+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:56.329501+0000","last_peered":"2024-09-18T11:46:56.329501+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:56.329501+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41522422,"num_objects":5069,"num_object_clones":0,"num_object_copies":20276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2029,"num_bytes_recovered":16588800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5003}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4929","reported_seq":4991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.468500+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:46:56.468500+0000","last_peered":"2024-09-18T11:46:56.468500+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:46:56.468500+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40334582,"num_objects":4924,"num_object_clones":0,"num_object_copies":19696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4927,"num_write_kb":39404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":666805620,"num_objects":81402,"num_object_clones":0,"num_object_copies":325608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81429,"num_write_kb":651314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2054,"num_bytes_recovered":16588800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1206898688,"data_stored":1206898688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81456,"ondisk_log_size":81456,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625528,"kb_used_data":327664,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93746312,"statfs":{"total":96636764160,"available":95996223488,"internally_reserved":0,"allocated":335527936,"data_stored":335368958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,0,3,12,2,5,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":2.375},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.3999999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":7.8970000000000002},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.376}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.7889999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625436,"kb_used_data":323924,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93746404,"statfs":{"total":96636764160,"available":95996317696,"internally_reserved":0,"allocated":331698176,"data_stored":331539198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:45:57 2024","interfaces":[{"interface":"back","average":{"1min":4.0880000000000001,"5min":2.0390000000000001,"15min":1.6970000000000001},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":3.0609999999999999,"5min":1.742,"15min":1.522},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":18.047000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.6240000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":2.98},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":2.6619999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":2.6190000000000002},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.238}]}]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504996,"kb_used_data":203032,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93866844,"statfs":{"total":96636764160,"available":96119648256,"internally_reserved":0,"allocated":207904768,"data_stored":207745790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":4.8520000000000003},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":10.898999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.9400000000000004},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":4.8819999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":3.476},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":4.968}]}]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621392,"kb_used_data":323592,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93750448,"statfs":{"total":96636764160,"available":96000458752,"internally_reserved":0,"allocated":331358208,"data_stored":331199230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":4.7590000000000003},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":2.1080000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":8.548},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":5.4109999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.88100000000000001},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":2.1389999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331612160,"data_stored":331612160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":207888384,"data_stored":207888384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331468800,"data_stored":331468800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":335929344,"data_stored":335929344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:01.715 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:02.205 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:02.206 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:02.230 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-18T11:47:02.067534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":673285492,"num_objects":82193,"num_object_clones":0,"num_object_copies":328772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82220,"num_write_kb":657642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2070,"num_bytes_recovered":16719872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82247,"ondisk_log_size":82247,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2396612,"kb_used_data":1189152,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375090748,"statfs":{"total":386547056640,"available":384092925952,"internally_reserved":0,"allocated":1217691648,"data_stored":1217055736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,0,10,29,11,9,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24576000,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3000,"num_write_kb":24000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.187601"},"pg_stats":[{"pgid":"1.f","version":"31'5269","reported_seq":5310,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.915955+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:59.915955+0000","last_peered":"2024-09-18T11:46:59.915955+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:59.915955+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43163648,"num_objects":5269,"num_object_clones":0,"num_object_copies":21076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":42152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'5198","reported_seq":5295,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.516273+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:01.516273+0000","last_peered":"2024-09-18T11:47:01.516273+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:01.516273+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546422,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5196,"num_write_kb":41564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'5090","reported_seq":5173,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.273753+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:01.273753+0000","last_peered":"2024-09-18T11:47:01.273753+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:01.273753+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41671291,"num_objects":5087,"num_object_clones":0,"num_object_copies":20348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":40702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3033}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'5132","reported_seq":5195,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.523619+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:01.523619+0000","last_peered":"2024-09-18T11:47:01.523619+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:01.523619+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42005750,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5130,"num_write_kb":41029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5110","reported_seq":5165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.908322+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:59.908322+0000","last_peered":"2024-09-18T11:46:59.908322+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:59.908322+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5110,"log_dups_size":0,"ondisk_log_size":5110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41833718,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":40861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5142","reported_seq":5185,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.810345+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:59.810345+0000","last_peered":"2024-09-18T11:46:59.810345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:59.810345+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5142,"log_dups_size":0,"ondisk_log_size":5142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42123264,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5142,"num_write_kb":41136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'5206","reported_seq":5272,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.463646+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:01.463646+0000","last_peered":"2024-09-18T11:47:01.463646+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:01.463646+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595574,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5204,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3066}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'5088","reported_seq":5152,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.420137+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:01.420137+0000","last_peered":"2024-09-18T11:47:01.420137+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:01.420137+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41637110,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":40676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3049}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5241","reported_seq":5299,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.822372+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:59.822372+0000","last_peered":"2024-09-18T11:46:59.822372+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:59.822372+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5241,"log_dups_size":0,"ondisk_log_size":5241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42863084,"num_objects":5233,"num_object_clones":0,"num_object_copies":20932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5238,"num_write_kb":41881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'5008","reported_seq":5093,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.332684+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:01.332684+0000","last_peered":"2024-09-18T11:47:01.332684+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:01.332684+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006326,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5006,"num_write_kb":40046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3031}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5115","reported_seq":5171,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.935353+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:59.935353+0000","last_peered":"2024-09-18T11:46:59.935353+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:59.935353+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41856881,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":40884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5231","reported_seq":5266,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.860631+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:59.860631+0000","last_peered":"2024-09-18T11:46:59.860631+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:59.860631+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42842747,"num_objects":5230,"num_object_clones":0,"num_object_copies":20920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5230,"num_write_kb":41839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'5196","reported_seq":5273,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.255657+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:01.255657+0000","last_peered":"2024-09-18T11:47:01.255657+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:01.255657+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42556027,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5195,"num_write_kb":41559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5128","reported_seq":5177,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.909713+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:59.909713+0000","last_peered":"2024-09-18T11:46:59.909713+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:59.909713+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5128,"log_dups_size":0,"ondisk_log_size":5128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41972982,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":41004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5072","reported_seq":9209,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.329501+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:56.329501+0000","last_peered":"2024-09-18T11:46:56.329501+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:56.329501+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41522422,"num_objects":5069,"num_object_clones":0,"num_object_copies":20276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2029,"num_bytes_recovered":16588800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5003}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'5021","reported_seq":5084,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.431953+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:01.431953+0000","last_peered":"2024-09-18T11:47:01.431953+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:01.431953+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41088246,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5019,"num_write_kb":40140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":673285492,"num_objects":82193,"num_object_clones":0,"num_object_copies":328772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82220,"num_write_kb":657642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2070,"num_bytes_recovered":16719872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1219342336,"data_stored":1219342336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82247,"ondisk_log_size":82247,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625528,"kb_used_data":327664,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93746312,"statfs":{"total":96636764160,"available":95996223488,"internally_reserved":0,"allocated":335527936,"data_stored":335368958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,0,3,12,2,5,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":2.375},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.3999999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":7.8970000000000002},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.376}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.7889999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634608,"kb_used_data":328936,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93737232,"statfs":{"total":96636764160,"available":95986925568,"internally_reserved":0,"allocated":336830464,"data_stored":336671486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":2.6699999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.6469999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":2.5800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":2.621},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.34699999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504996,"kb_used_data":203032,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93866844,"statfs":{"total":96636764160,"available":96119648256,"internally_reserved":0,"allocated":207904768,"data_stored":207745790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":4.8520000000000003},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":10.898999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.9400000000000004},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":4.8819999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":3.476},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":4.968}]}]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631480,"kb_used_data":329520,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93740360,"statfs":{"total":96636764160,"available":95990128640,"internally_reserved":0,"allocated":337428480,"data_stored":337269502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,9,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":3.956},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":6.5060000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":5.952},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":6.5439999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":4.04},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":1.1319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":337833984,"data_stored":337833984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":207888384,"data_stored":207888384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337690624,"data_stored":337690624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":335929344,"data_stored":335929344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:02.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:02.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5269", 'reported_seq': 5310, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:59.915955+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:46:59.915955+0000', 'last_peered': '2024-09-18T11:46:59.915955+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:46:59.915955+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5269, 'log_dups_size': 0, 'ondisk_log_size': 5269, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43163648, 'num_objects': 5269, 'num_object_clones': 0, 'num_object_copies': 21076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5269, 'num_write_kb': 42152, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3139}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'5198", 'reported_seq': 5295, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.516273+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:01.516273+0000', 'last_peered': '2024-09-18T11:47:01.516273+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:01.516273+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5198, 'log_dups_size': 0, 'ondisk_log_size': 5198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42546422, 'num_objects': 5194, 'num_object_clones': 0, 'num_object_copies': 20776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5196, 'num_write_kb': 41564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 131072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'5090", 'reported_seq': 5173, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.273753+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:01.273753+0000', 'last_peered': '2024-09-18T11:47:01.273753+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:01.273753+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5090, 'log_dups_size': 0, 'ondisk_log_size': 5090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41671291, 'num_objects': 5087, 'num_object_clones': 0, 'num_object_copies': 20348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5089, 'num_write_kb': 40702, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3033}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'5132", 'reported_seq': 5195, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.523619+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:01.523619+0000', 'last_peered': '2024-09-18T11:47:01.523619+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:01.523619+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5132, 'log_dups_size': 0, 'ondisk_log_size': 5132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42005750, 'num_objects': 5128, 'num_object_clones': 0, 'num_object_copies': 20512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5130, 'num_write_kb': 41029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3014}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5110", 'reported_seq': 5165, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:59.908322+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:46:59.908322+0000', 'last_peered': '2024-09-18T11:46:59.908322+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:46:59.908322+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5110, 'log_dups_size': 0, 'ondisk_log_size': 5110, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41833718, 'num_objects': 5107, 'num_object_clones': 0, 'num_object_copies': 20428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5108, 'num_write_kb': 40861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3025}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5142", 'reported_seq': 5185, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:59.810345+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:46:59.810345+0000', 'last_peered': '2024-09-18T11:46:59.810345+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:46:59.810345+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5142, 'log_dups_size': 0, 'ondisk_log_size': 5142, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42123264, 'num_objects': 5142, 'num_object_clones': 0, 'num_object_copies': 20568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5142, 'num_write_kb': 41136, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2988}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:02.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:02.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'5206", 'reported_seq': 5272, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.463646+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:01.463646+0000', 'last_peered': '2024-09-18T11:47:01.463646+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:01.463646+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5206, 'log_dups_size': 0, 'ondisk_log_size': 5206, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42595574, 'num_objects': 5200, 'num_object_clones': 0, 'num_object_copies': 20800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5204, 'num_write_kb': 41612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3066}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:02.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:02.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'5088", 'reported_seq': 5152, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.420137+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:01.420137+0000', 'last_peered': '2024-09-18T11:47:01.420137+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:01.420137+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5088, 'log_dups_size': 0, 'ondisk_log_size': 5088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41637110, 'num_objects': 5083, 'num_object_clones': 0, 'num_object_copies': 20332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5086, 'num_write_kb': 40676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3049}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:02.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:02.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5241", 'reported_seq': 5299, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:59.822372+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:46:59.822372+0000', 'last_peered': '2024-09-18T11:46:59.822372+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:46:59.822372+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5241, 'log_dups_size': 0, 'ondisk_log_size': 5241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42863084, 'num_objects': 5233, 'num_object_clones': 0, 'num_object_copies': 20932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5238, 'num_write_kb': 41881, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3098}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:02.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:02.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'5008", 'reported_seq': 5093, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.332684+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:01.332684+0000', 'last_peered': '2024-09-18T11:47:01.332684+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:01.332684+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5008, 'log_dups_size': 0, 'ondisk_log_size': 5008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41006326, 'num_objects': 5006, 'num_object_clones': 0, 'num_object_copies': 20024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5006, 'num_write_kb': 40046, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3031}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:02.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:02.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5115", 'reported_seq': 5171, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:59.935353+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:46:59.935353+0000', 'last_peered': '2024-09-18T11:46:59.935353+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:46:59.935353+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5115, 'log_dups_size': 0, 'ondisk_log_size': 5115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41856881, 'num_objects': 5110, 'num_object_clones': 0, 'num_object_copies': 20440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5112, 'num_write_kb': 40884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3047}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:02.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:02.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5231", 'reported_seq': 5266, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:59.860631+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:46:59.860631+0000', 'last_peered': '2024-09-18T11:46:59.860631+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:46:59.860631+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5231, 'log_dups_size': 0, 'ondisk_log_size': 5231, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42842747, 'num_objects': 5230, 'num_object_clones': 0, 'num_object_copies': 20920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5230, 'num_write_kb': 41839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:02.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:02.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'5196", 'reported_seq': 5273, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.255657+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:01.255657+0000', 'last_peered': '2024-09-18T11:47:01.255657+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:01.255657+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5196, 'log_dups_size': 0, 'ondisk_log_size': 5196, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42556027, 'num_objects': 5195, 'num_object_clones': 0, 'num_object_copies': 20780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5195, 'num_write_kb': 41559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:02.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:02.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5128", 'reported_seq': 5177, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:59.909713+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:46:59.909713+0000', 'last_peered': '2024-09-18T11:46:59.909713+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:46:59.909713+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5128, 'log_dups_size': 0, 'ondisk_log_size': 5128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41972982, 'num_objects': 5124, 'num_object_clones': 0, 'num_object_copies': 20496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5126, 'num_write_kb': 41004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:02.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-18T11:47:02.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5072", 'reported_seq': 9209, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:46:56.329501+0000', 'last_change': '2024-09-18T11:44:47.312117+0000', 'last_active': '2024-09-18T11:46:56.329501+0000', 'last_peered': '2024-09-18T11:46:56.329501+0000', 'last_clean': '2024-09-18T11:43:44.816829+0000', 'last_became_active': '2024-09-18T11:44:47.300191+0000', 'last_became_peered': '2024-09-18T11:44:47.300191+0000', 'last_unstale': '2024-09-18T11:46:56.329501+0000', 'last_undegraded': '2024-09-18T11:44:47.148727+0000', 'last_fullsized': '2024-09-18T11:44:47.122854+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5072, 'log_dups_size': 0, 'ondisk_log_size': 5072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41522422, 'num_objects': 5069, 'num_object_clones': 0, 'num_object_copies': 20276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 66, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5070, 'num_write_kb': 40557, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2029, 'num_bytes_recovered': 16588800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 5003}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:02.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:02.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'5021", 'reported_seq': 5084, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.431953+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:01.431953+0000', 'last_peered': '2024-09-18T11:47:01.431953+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:01.431953+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5021, 'log_dups_size': 0, 'ondisk_log_size': 5021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41088246, 'num_objects': 5016, 'num_object_clones': 0, 'num_object_copies': 20064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5019, 'num_write_kb': 40140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3003}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:02.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:02.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 165 20 82629 82609 3.9105 1.73438 0.0372837 0.0399377 2024-09-18T11:47:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:02.898 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31384/328772 objects degraded (9.546%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":220,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":82193,"data_bytes":673285492,"bytes_used":2454130688,"bytes_avail":384092925952,"bytes_total":386547056640,"degraded_objects":31384,"degraded_total":328772,"degraded_ratio":0.095458250702614572,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":86793,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":868352,"num_keys_recovered":0,"write_bytes_sec":2456432,"read_op_per_sec":0,"write_op_per_sec":299},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:02.930 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:02.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:03.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 166 20 82911 82891 3.90022 2.20312 0.0334018 0.0400501 2024-09-18T11:47:03.550 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:03.705 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:03.706 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-18T11:47:02.067534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":673285492,"num_objects":82193,"num_object_clones":0,"num_object_copies":328772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82220,"num_write_kb":657642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2070,"num_bytes_recovered":16719872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82247,"ondisk_log_size":82247,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2396612,"kb_used_data":1189152,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375090748,"statfs":{"total":386547056640,"available":384092925952,"internally_reserved":0,"allocated":1217691648,"data_stored":1217055736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,0,10,29,11,9,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24576000,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3000,"num_write_kb":24000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.187601"},"pg_stats":[{"pgid":"1.f","version":"31'5269","reported_seq":5310,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.915955+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:46:59.915955+0000","last_peered":"2024-09-18T11:46:59.915955+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:46:59.915955+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43163648,"num_objects":5269,"num_object_clones":0,"num_object_copies":21076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":42152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'5198","reported_seq":5295,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.516273+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:01.516273+0000","last_peered":"2024-09-18T11:47:01.516273+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:01.516273+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546422,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5196,"num_write_kb":41564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'5090","reported_seq":5173,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.273753+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:01.273753+0000","last_peered":"2024-09-18T11:47:01.273753+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:01.273753+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41671291,"num_objects":5087,"num_object_clones":0,"num_object_copies":20348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":40702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3033}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'5132","reported_seq":5195,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.523619+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:01.523619+0000","last_peered":"2024-09-18T11:47:01.523619+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:01.523619+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42005750,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5130,"num_write_kb":41029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5110","reported_seq":5165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.908322+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:46:59.908322+0000","last_peered":"2024-09-18T11:46:59.908322+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:46:59.908322+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5110,"log_dups_size":0,"ondisk_log_size":5110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41833718,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":40861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5142","reported_seq":5185,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.810345+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:46:59.810345+0000","last_peered":"2024-09-18T11:46:59.810345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:46:59.810345+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5142,"log_dups_size":0,"ondisk_log_size":5142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42123264,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5142,"num_write_kb":41136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'5206","reported_seq":5272,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.463646+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:01.463646+0000","last_peered":"2024-09-18T11:47:01.463646+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:01.463646+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595574,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5204,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3066}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'5088","reported_seq":5152,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.420137+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:01.420137+0000","last_peered":"2024-09-18T11:47:01.420137+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:01.420137+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41637110,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":40676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3049}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5241","reported_seq":5299,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.822372+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:46:59.822372+0000","last_peered":"2024-09-18T11:46:59.822372+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:46:59.822372+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5241,"log_dups_size":0,"ondisk_log_size":5241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42863084,"num_objects":5233,"num_object_clones":0,"num_object_copies":20932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5238,"num_write_kb":41881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'5008","reported_seq":5093,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.332684+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:01.332684+0000","last_peered":"2024-09-18T11:47:01.332684+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:01.332684+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006326,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5006,"num_write_kb":40046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3031}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5115","reported_seq":5171,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.935353+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:46:59.935353+0000","last_peered":"2024-09-18T11:46:59.935353+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:46:59.935353+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41856881,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":40884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5231","reported_seq":5266,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.860631+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:46:59.860631+0000","last_peered":"2024-09-18T11:46:59.860631+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:46:59.860631+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42842747,"num_objects":5230,"num_object_clones":0,"num_object_copies":20920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5230,"num_write_kb":41839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'5196","reported_seq":5273,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.255657+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:01.255657+0000","last_peered":"2024-09-18T11:47:01.255657+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:01.255657+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42556027,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5195,"num_write_kb":41559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5128","reported_seq":5177,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:59.909713+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:46:59.909713+0000","last_peered":"2024-09-18T11:46:59.909713+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:46:59.909713+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5128,"log_dups_size":0,"ondisk_log_size":5128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41972982,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":41004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5072","reported_seq":9209,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:46:56.329501+0000","last_change":"2024-09-18T11:44:47.312117+0000","last_active":"2024-09-18T11:46:56.329501+0000","last_peered":"2024-09-18T11:46:56.329501+0000","last_clean":"2024-09-18T11:43:44.816829+0000","last_became_active":"2024-09-18T11:44:47.300191+0000","last_became_peered":"2024-09-18T11:44:47.300191+0000","last_unstale":"2024-09-18T11:46:56.329501+0000","last_undegraded":"2024-09-18T11:44:47.148727+0000","last_fullsized":"2024-09-18T11:44:47.122854+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41522422,"num_objects":5069,"num_object_clones":0,"num_object_copies":20276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2029,"num_bytes_recovered":16588800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5003}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'5021","reported_seq":5084,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.431953+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:01.431953+0000","last_peered":"2024-09-18T11:47:01.431953+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:01.431953+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41088246,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5019,"num_write_kb":40140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":673285492,"num_objects":82193,"num_object_clones":0,"num_object_copies":328772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82220,"num_write_kb":657642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2070,"num_bytes_recovered":16719872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1219342336,"data_stored":1219342336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82247,"ondisk_log_size":82247,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625528,"kb_used_data":327664,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93746312,"statfs":{"total":96636764160,"available":95996223488,"internally_reserved":0,"allocated":335527936,"data_stored":335368958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,0,3,12,2,5,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":2.375},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.3999999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":7.8970000000000002},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.376}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.7889999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634608,"kb_used_data":328936,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93737232,"statfs":{"total":96636764160,"available":95986925568,"internally_reserved":0,"allocated":336830464,"data_stored":336671486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":2.6699999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.6469999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":2.5800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":2.621},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.34699999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504996,"kb_used_data":203032,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93866844,"statfs":{"total":96636764160,"available":96119648256,"internally_reserved":0,"allocated":207904768,"data_stored":207745790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":4.8520000000000003},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":10.898999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.9400000000000004},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":4.8819999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":3.476},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":4.968}]}]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631480,"kb_used_data":329520,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93740360,"statfs":{"total":96636764160,"available":95990128640,"internally_reserved":0,"allocated":337428480,"data_stored":337269502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,9,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":3.956},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":6.5060000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":5.952},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":6.5439999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":4.04},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":1.1319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":337833984,"data_stored":337833984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":207888384,"data_stored":207888384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337690624,"data_stored":337690624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":335929344,"data_stored":335929344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:04.249 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 167 20 83216 83196 3.89113 2.38281 0.0821854 0.0401453 2024-09-18T11:47:05.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 168 20 83491 83471 3.88076 2.14844 0.0649803 0.0402502 2024-09-18T11:47:06.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 169 20 83802 83782 3.87217 2.42969 0.0453319 0.0403383 2024-09-18T11:47:06.708 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:07.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 170 20 84100 84080 3.86306 2.32812 0.0969439 0.0404304 2024-09-18T11:47:07.247 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:07.247 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:07.268 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-18T11:47:05.885543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":679683444,"num_objects":82974,"num_object_clones":0,"num_object_copies":331896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83001,"num_write_kb":663890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2136,"num_bytes_recovered":17260544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83028,"ondisk_log_size":83028,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2420720,"kb_used_data":1200780,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375066640,"statfs":{"total":386547056640,"available":384068239360,"internally_reserved":0,"allocated":1229598720,"data_stored":1228946394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,16,10,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18423808,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":17992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003866"},"pg_stats":[{"pgid":"1.f","version":"33'5356","reported_seq":5399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.903660+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:04.903660+0000","last_peered":"2024-09-18T11:47:04.903660+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:04.903660+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43876352,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":42848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3226}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'5198","reported_seq":5295,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.516273+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:01.516273+0000","last_peered":"2024-09-18T11:47:01.516273+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:01.516273+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546422,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5196,"num_write_kb":41564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'5090","reported_seq":5173,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.273753+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:01.273753+0000","last_peered":"2024-09-18T11:47:01.273753+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:01.273753+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41671291,"num_objects":5087,"num_object_clones":0,"num_object_copies":20348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":40702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3033}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'5132","reported_seq":5195,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.523619+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:01.523619+0000","last_peered":"2024-09-18T11:47:01.523619+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:01.523619+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42005750,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5130,"num_write_kb":41029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5203","reported_seq":5260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.941177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:04.941177+0000","last_peered":"2024-09-18T11:47:04.941177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:04.941177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5203,"log_dups_size":0,"ondisk_log_size":5203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595574,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5201,"num_write_kb":41605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5234","reported_seq":5279,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.977617+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:04.977617+0000","last_peered":"2024-09-18T11:47:04.977617+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:04.977617+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5234,"log_dups_size":0,"ondisk_log_size":5234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42876928,"num_objects":5234,"num_object_clones":0,"num_object_copies":20936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":41872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'5206","reported_seq":5272,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.463646+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:01.463646+0000","last_peered":"2024-09-18T11:47:01.463646+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:01.463646+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595574,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5204,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3066}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'5088","reported_seq":5152,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.420137+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:01.420137+0000","last_peered":"2024-09-18T11:47:01.420137+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:01.420137+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41637110,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":40676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3049}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5308","reported_seq":5368,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.857887+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:04.857887+0000","last_peered":"2024-09-18T11:47:04.857887+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:04.857887+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5308,"log_dups_size":0,"ondisk_log_size":5308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43411948,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5305,"num_write_kb":42417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'5008","reported_seq":5093,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.332684+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:01.332684+0000","last_peered":"2024-09-18T11:47:01.332684+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:01.332684+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006326,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5006,"num_write_kb":40046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3031}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5208","reported_seq":5267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.932981+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:04.932981+0000","last_peered":"2024-09-18T11:47:04.932981+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:04.932981+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5208,"log_dups_size":0,"ondisk_log_size":5208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42618737,"num_objects":5203,"num_object_clones":0,"num_object_copies":20812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":41628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5311","reported_seq":5350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.969554+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:04.969554+0000","last_peered":"2024-09-18T11:47:04.969554+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:04.969554+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43498107,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5310,"num_write_kb":42479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3201}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'5196","reported_seq":5273,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.255657+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:01.255657+0000","last_peered":"2024-09-18T11:47:01.255657+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:01.255657+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42556027,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5195,"num_write_kb":41559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5236","reported_seq":5287,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.985628+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:04.985628+0000","last_peered":"2024-09-18T11:47:04.985628+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:04.985628+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42857718,"num_objects":5232,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":41868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3074}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5233","reported_seq":9518,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:05.066209+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:05.066209+0000","last_peered":"2024-09-18T11:47:05.066209+0000","last_clean":"2024-09-18T11:47:05.066209+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:05.066209+0000","last_undegraded":"2024-09-18T11:47:05.066209+0000","last_fullsized":"2024-09-18T11:47:05.066209+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5233,"log_dups_size":0,"ondisk_log_size":5233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42841334,"num_objects":5230,"num_object_clones":0,"num_object_copies":20920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5231,"num_write_kb":41845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"32'5021","reported_seq":5084,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.431953+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:01.431953+0000","last_peered":"2024-09-18T11:47:01.431953+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:01.431953+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41088246,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5019,"num_write_kb":40140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":679683444,"num_objects":82974,"num_object_clones":0,"num_object_copies":331896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83001,"num_write_kb":663890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2136,"num_bytes_recovered":17260544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1231171584,"data_stored":1231171584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83028,"ondisk_log_size":83028,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635508,"kb_used_data":333480,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93736332,"statfs":{"total":96636764160,"available":95986003968,"internally_reserved":0,"allocated":341483520,"data_stored":341316335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,4,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":8.9740000000000002},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":9.0009999999999994}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":4.8760000000000003},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.9459999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.9809999999999999},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":9.2300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634608,"kb_used_data":328936,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93737232,"statfs":{"total":96636764160,"available":95986925568,"internally_reserved":0,"allocated":336830464,"data_stored":336671486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":2.6699999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.6469999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":2.5800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":2.621},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.34699999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519124,"kb_used_data":208844,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93852716,"statfs":{"total":96636764160,"available":96105181184,"internally_reserved":0,"allocated":213856256,"data_stored":213689071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.224},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.20599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.23599999999999999},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.35499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631480,"kb_used_data":329520,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93740360,"statfs":{"total":96636764160,"available":95990128640,"internally_reserved":0,"allocated":337428480,"data_stored":337269502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,9,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":3.956},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":6.5060000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":5.952},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":6.5439999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":4.04},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":1.1319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":337833984,"data_stored":337833984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213987328,"data_stored":213987328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337690624,"data_stored":337690624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":341659648,"data_stored":341659648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:07.270 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:07.809 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:07.810 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:07.834 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-18T11:47:05.885543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":679683444,"num_objects":82974,"num_object_clones":0,"num_object_copies":331896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83001,"num_write_kb":663890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2136,"num_bytes_recovered":17260544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83028,"ondisk_log_size":83028,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2420720,"kb_used_data":1200780,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375066640,"statfs":{"total":386547056640,"available":384068239360,"internally_reserved":0,"allocated":1229598720,"data_stored":1228946394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,16,10,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18423808,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":17992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003866"},"pg_stats":[{"pgid":"1.f","version":"33'5356","reported_seq":5399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.903660+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:04.903660+0000","last_peered":"2024-09-18T11:47:04.903660+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:04.903660+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43876352,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":42848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3226}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'5198","reported_seq":5295,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.516273+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:01.516273+0000","last_peered":"2024-09-18T11:47:01.516273+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:01.516273+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546422,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5196,"num_write_kb":41564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'5090","reported_seq":5173,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.273753+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:01.273753+0000","last_peered":"2024-09-18T11:47:01.273753+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:01.273753+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41671291,"num_objects":5087,"num_object_clones":0,"num_object_copies":20348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":40702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3033}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'5132","reported_seq":5195,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.523619+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:01.523619+0000","last_peered":"2024-09-18T11:47:01.523619+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:01.523619+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42005750,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5130,"num_write_kb":41029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5203","reported_seq":5260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.941177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:04.941177+0000","last_peered":"2024-09-18T11:47:04.941177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:04.941177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5203,"log_dups_size":0,"ondisk_log_size":5203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595574,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5201,"num_write_kb":41605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5234","reported_seq":5279,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.977617+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:04.977617+0000","last_peered":"2024-09-18T11:47:04.977617+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:04.977617+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5234,"log_dups_size":0,"ondisk_log_size":5234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42876928,"num_objects":5234,"num_object_clones":0,"num_object_copies":20936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":41872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'5206","reported_seq":5272,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.463646+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:01.463646+0000","last_peered":"2024-09-18T11:47:01.463646+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:01.463646+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595574,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5204,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3066}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'5088","reported_seq":5152,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.420137+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:01.420137+0000","last_peered":"2024-09-18T11:47:01.420137+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:01.420137+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41637110,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":40676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3049}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5308","reported_seq":5368,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.857887+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:04.857887+0000","last_peered":"2024-09-18T11:47:04.857887+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:04.857887+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5308,"log_dups_size":0,"ondisk_log_size":5308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43411948,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5305,"num_write_kb":42417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'5008","reported_seq":5093,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.332684+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:01.332684+0000","last_peered":"2024-09-18T11:47:01.332684+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:01.332684+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006326,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5006,"num_write_kb":40046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3031}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5208","reported_seq":5267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.932981+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:04.932981+0000","last_peered":"2024-09-18T11:47:04.932981+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:04.932981+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5208,"log_dups_size":0,"ondisk_log_size":5208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42618737,"num_objects":5203,"num_object_clones":0,"num_object_copies":20812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":41628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5311","reported_seq":5350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.969554+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:04.969554+0000","last_peered":"2024-09-18T11:47:04.969554+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:04.969554+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43498107,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5310,"num_write_kb":42479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3201}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'5196","reported_seq":5273,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.255657+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:01.255657+0000","last_peered":"2024-09-18T11:47:01.255657+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:01.255657+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42556027,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5195,"num_write_kb":41559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5236","reported_seq":5287,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.985628+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:04.985628+0000","last_peered":"2024-09-18T11:47:04.985628+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:04.985628+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42857718,"num_objects":5232,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":41868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3074}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5233","reported_seq":9518,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:05.066209+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:05.066209+0000","last_peered":"2024-09-18T11:47:05.066209+0000","last_clean":"2024-09-18T11:47:05.066209+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:05.066209+0000","last_undegraded":"2024-09-18T11:47:05.066209+0000","last_fullsized":"2024-09-18T11:47:05.066209+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5233,"log_dups_size":0,"ondisk_log_size":5233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42841334,"num_objects":5230,"num_object_clones":0,"num_object_copies":20920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5231,"num_write_kb":41845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"32'5021","reported_seq":5084,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:01.431953+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:01.431953+0000","last_peered":"2024-09-18T11:47:01.431953+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:01.431953+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41088246,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5019,"num_write_kb":40140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":679683444,"num_objects":82974,"num_object_clones":0,"num_object_copies":331896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83001,"num_write_kb":663890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2136,"num_bytes_recovered":17260544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1231171584,"data_stored":1231171584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83028,"ondisk_log_size":83028,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635508,"kb_used_data":333480,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93736332,"statfs":{"total":96636764160,"available":95986003968,"internally_reserved":0,"allocated":341483520,"data_stored":341316335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,4,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":8.9740000000000002},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":9.0009999999999994}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":4.8760000000000003},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.9459999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.9809999999999999},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":9.2300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634608,"kb_used_data":328936,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93737232,"statfs":{"total":96636764160,"available":95986925568,"internally_reserved":0,"allocated":336830464,"data_stored":336671486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":2.6699999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.6469999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":2.5800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":2.621},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.34699999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519124,"kb_used_data":208844,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93852716,"statfs":{"total":96636764160,"available":96105181184,"internally_reserved":0,"allocated":213856256,"data_stored":213689071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.224},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.20599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.23599999999999999},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.35499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631480,"kb_used_data":329520,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93740360,"statfs":{"total":96636764160,"available":95990128640,"internally_reserved":0,"allocated":337428480,"data_stored":337269502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,9,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":3.956},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":6.5060000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:01 2024","interfaces":[{"interface":"back","average":{"1min":4.4829999999999997,"5min":1.5700000000000001,"15min":1.0840000000000001},"min":{"1min":0.25600000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":5.952},{"interface":"front","average":{"1min":3.9910000000000001,"5min":1.7470000000000001,"15min":1.373},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":12.256,"5min":12.256,"15min":12.256},"last":6.5439999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":4.04},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":1.1319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":337833984,"data_stored":337833984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213987328,"data_stored":213987328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337690624,"data_stored":337690624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":341659648,"data_stored":341659648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:07.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:07.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5356", 'reported_seq': 5399, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:04.903660+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:47:04.903660+0000', 'last_peered': '2024-09-18T11:47:04.903660+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:47:04.903660+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5356, 'log_dups_size': 0, 'ondisk_log_size': 5356, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43876352, 'num_objects': 5356, 'num_object_clones': 0, 'num_object_copies': 21424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5356, 'num_write_kb': 42848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3226}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:07.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:07.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'5198", 'reported_seq': 5295, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.516273+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:01.516273+0000', 'last_peered': '2024-09-18T11:47:01.516273+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:01.516273+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5198, 'log_dups_size': 0, 'ondisk_log_size': 5198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42546422, 'num_objects': 5194, 'num_object_clones': 0, 'num_object_copies': 20776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5196, 'num_write_kb': 41564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 131072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:07.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:07.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'5090", 'reported_seq': 5173, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.273753+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:01.273753+0000', 'last_peered': '2024-09-18T11:47:01.273753+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:01.273753+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5090, 'log_dups_size': 0, 'ondisk_log_size': 5090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41671291, 'num_objects': 5087, 'num_object_clones': 0, 'num_object_copies': 20348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5089, 'num_write_kb': 40702, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3033}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:07.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:07.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'5132", 'reported_seq': 5195, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.523619+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:01.523619+0000', 'last_peered': '2024-09-18T11:47:01.523619+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:01.523619+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5132, 'log_dups_size': 0, 'ondisk_log_size': 5132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42005750, 'num_objects': 5128, 'num_object_clones': 0, 'num_object_copies': 20512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5130, 'num_write_kb': 41029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3014}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:07.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:07.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5203", 'reported_seq': 5260, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:04.941177+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:47:04.941177+0000', 'last_peered': '2024-09-18T11:47:04.941177+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:47:04.941177+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5203, 'log_dups_size': 0, 'ondisk_log_size': 5203, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42595574, 'num_objects': 5200, 'num_object_clones': 0, 'num_object_copies': 20800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5201, 'num_write_kb': 41605, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3118}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:07.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:07.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5234", 'reported_seq': 5279, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:04.977617+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:47:04.977617+0000', 'last_peered': '2024-09-18T11:47:04.977617+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:47:04.977617+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5234, 'log_dups_size': 0, 'ondisk_log_size': 5234, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42876928, 'num_objects': 5234, 'num_object_clones': 0, 'num_object_copies': 20936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5234, 'num_write_kb': 41872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3080}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:07.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:07.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'5206", 'reported_seq': 5272, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.463646+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:01.463646+0000', 'last_peered': '2024-09-18T11:47:01.463646+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:01.463646+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5206, 'log_dups_size': 0, 'ondisk_log_size': 5206, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42595574, 'num_objects': 5200, 'num_object_clones': 0, 'num_object_copies': 20800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5204, 'num_write_kb': 41612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3066}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:07.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:07.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'5088", 'reported_seq': 5152, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.420137+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:01.420137+0000', 'last_peered': '2024-09-18T11:47:01.420137+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:01.420137+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5088, 'log_dups_size': 0, 'ondisk_log_size': 5088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41637110, 'num_objects': 5083, 'num_object_clones': 0, 'num_object_copies': 20332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5086, 'num_write_kb': 40676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3049}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:07.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:07.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5308", 'reported_seq': 5368, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:04.857887+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:47:04.857887+0000', 'last_peered': '2024-09-18T11:47:04.857887+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:47:04.857887+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5308, 'log_dups_size': 0, 'ondisk_log_size': 5308, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43411948, 'num_objects': 5300, 'num_object_clones': 0, 'num_object_copies': 21200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5305, 'num_write_kb': 42417, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3165}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:07.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:07.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'5008", 'reported_seq': 5093, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.332684+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:01.332684+0000', 'last_peered': '2024-09-18T11:47:01.332684+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:01.332684+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5008, 'log_dups_size': 0, 'ondisk_log_size': 5008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41006326, 'num_objects': 5006, 'num_object_clones': 0, 'num_object_copies': 20024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5006, 'num_write_kb': 40046, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3031}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:07.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:07.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5208", 'reported_seq': 5267, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:04.932981+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:47:04.932981+0000', 'last_peered': '2024-09-18T11:47:04.932981+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:47:04.932981+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5208, 'log_dups_size': 0, 'ondisk_log_size': 5208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42618737, 'num_objects': 5203, 'num_object_clones': 0, 'num_object_copies': 20812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5203, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5205, 'num_write_kb': 41628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3140}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:07.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:07.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5311", 'reported_seq': 5350, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:04.969554+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:47:04.969554+0000', 'last_peered': '2024-09-18T11:47:04.969554+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:47:04.969554+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5311, 'log_dups_size': 0, 'ondisk_log_size': 5311, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43498107, 'num_objects': 5310, 'num_object_clones': 0, 'num_object_copies': 21240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5310, 'num_write_kb': 42479, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:07.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:07.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'5196", 'reported_seq': 5273, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.255657+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:01.255657+0000', 'last_peered': '2024-09-18T11:47:01.255657+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:01.255657+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5196, 'log_dups_size': 0, 'ondisk_log_size': 5196, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42556027, 'num_objects': 5195, 'num_object_clones': 0, 'num_object_copies': 20780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5195, 'num_write_kb': 41559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:07.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:07.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5236", 'reported_seq': 5287, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:04.985628+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:47:04.985628+0000', 'last_peered': '2024-09-18T11:47:04.985628+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:47:04.985628+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5236, 'log_dups_size': 0, 'ondisk_log_size': 5236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42857718, 'num_objects': 5232, 'num_object_clones': 0, 'num_object_copies': 20928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5234, 'num_write_kb': 41868, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:07.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:07.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'5021", 'reported_seq': 5084, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:01.431953+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:01.431953+0000', 'last_peered': '2024-09-18T11:47:01.431953+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:01.431953+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5021, 'log_dups_size': 0, 'ondisk_log_size': 5021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41088246, 'num_objects': 5016, 'num_object_clones': 0, 'num_object_copies': 20064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5019, 'num_write_kb': 40140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3003}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:07.839 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:08.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 171 20 84383 84363 3.8534 2.21094 0.0516244 0.0405333 2024-09-18T11:47:08.475 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:08.475 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31237/334676 objects degraded (9.334%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":226,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":83669,"data_bytes":685376884,"bytes_used":2498760704,"bytes_avail":384048295936,"bytes_total":386547056640,"degraded_objects":31237,"degraded_total":334676,"degraded_ratio":0.093335046432968,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":166838,"recovering_keys_per_sec":0,"num_objects_recovered":163,"num_bytes_recovered":1335296,"num_keys_recovered":0,"write_bytes_sec":3013334,"read_op_per_sec":0,"write_op_per_sec":367},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:08.497 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:08.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:09.070 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:09.071 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:09.112 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-18T11:47:07.886327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":685376884,"num_objects":83669,"num_object_clones":0,"num_object_copies":334676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83696,"num_write_kb":669450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":17924096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83723,"ondisk_log_size":83723,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2440196,"kb_used_data":1211936,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375047164,"statfs":{"total":386547056640,"available":384048295936,"internally_reserved":0,"allocated":1241022464,"data_stored":1240353724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,8,5,10,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24117248,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2944,"num_write_kb":23552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":1335296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003508"},"pg_stats":[{"pgid":"1.f","version":"33'5356","reported_seq":5399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.903660+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:04.903660+0000","last_peered":"2024-09-18T11:47:04.903660+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:04.903660+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43876352,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":42848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3226}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5275","reported_seq":5533,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:06.495579+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:06.495579+0000","last_peered":"2024-09-18T11:47:06.495579+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:06.495579+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5275,"log_dups_size":0,"ondisk_log_size":5275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43177206,"num_objects":5271,"num_object_clones":0,"num_object_copies":21084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":42180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5180","reported_seq":5264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:06.392273+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:06.392273+0000","last_peered":"2024-09-18T11:47:06.392273+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:06.392273+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42408571,"num_objects":5177,"num_object_clones":0,"num_object_copies":20708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5179,"num_write_kb":41422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3123}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5227","reported_seq":5291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:06.469641+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:06.469641+0000","last_peered":"2024-09-18T11:47:06.469641+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:06.469641+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42783990,"num_objects":5223,"num_object_clones":0,"num_object_copies":20892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5225,"num_write_kb":41789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5203","reported_seq":5260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.941177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:04.941177+0000","last_peered":"2024-09-18T11:47:04.941177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:04.941177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5203,"log_dups_size":0,"ondisk_log_size":5203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595574,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5201,"num_write_kb":41605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5234","reported_seq":5279,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.977617+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:04.977617+0000","last_peered":"2024-09-18T11:47:04.977617+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:04.977617+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5234,"log_dups_size":0,"ondisk_log_size":5234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42876928,"num_objects":5234,"num_object_clones":0,"num_object_copies":20936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":41872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5286","reported_seq":5353,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:06.460247+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:06.460247+0000","last_peered":"2024-09-18T11:47:06.460247+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:06.460247+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43250934,"num_objects":5280,"num_object_clones":0,"num_object_copies":21120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5284,"num_write_kb":42252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3146}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5176","reported_seq":5241,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:06.396151+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:06.396151+0000","last_peered":"2024-09-18T11:47:06.396151+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:06.396151+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5176,"log_dups_size":0,"ondisk_log_size":5176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42358006,"num_objects":5171,"num_object_clones":0,"num_object_copies":20684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5174,"num_write_kb":41380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5308","reported_seq":5368,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.857887+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:04.857887+0000","last_peered":"2024-09-18T11:47:04.857887+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:04.857887+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5308,"log_dups_size":0,"ondisk_log_size":5308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43411948,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5305,"num_write_kb":42417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5092","reported_seq":5176,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:06.272201+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:06.272201+0000","last_peered":"2024-09-18T11:47:06.272201+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:06.272201+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5092,"log_dups_size":0,"ondisk_log_size":5092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41694454,"num_objects":5090,"num_object_clones":0,"num_object_copies":20360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5090,"num_write_kb":40718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5208","reported_seq":5267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.932981+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:04.932981+0000","last_peered":"2024-09-18T11:47:04.932981+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:04.932981+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5208,"log_dups_size":0,"ondisk_log_size":5208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42618737,"num_objects":5203,"num_object_clones":0,"num_object_copies":20812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":41628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5311","reported_seq":5350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.969554+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:04.969554+0000","last_peered":"2024-09-18T11:47:04.969554+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:04.969554+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43498107,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5310,"num_write_kb":42479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3201}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5291","reported_seq":5369,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:06.421592+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:06.421592+0000","last_peered":"2024-09-18T11:47:06.421592+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:06.421592+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5291,"log_dups_size":0,"ondisk_log_size":5291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43334267,"num_objects":5290,"num_object_clones":0,"num_object_copies":21160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5290,"num_write_kb":42319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3236}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5236","reported_seq":5287,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:04.985628+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:04.985628+0000","last_peered":"2024-09-18T11:47:04.985628+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:04.985628+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42857718,"num_objects":5232,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":41868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3074}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5233","reported_seq":9518,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:05.066209+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:05.066209+0000","last_peered":"2024-09-18T11:47:05.066209+0000","last_clean":"2024-09-18T11:47:05.066209+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:05.066209+0000","last_undegraded":"2024-09-18T11:47:05.066209+0000","last_fullsized":"2024-09-18T11:47:05.066209+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5233,"log_dups_size":0,"ondisk_log_size":5233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42841334,"num_objects":5230,"num_object_clones":0,"num_object_copies":20920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5231,"num_write_kb":41845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5107","reported_seq":5171,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:06.441628+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:06.441628+0000","last_peered":"2024-09-18T11:47:06.441628+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:06.441628+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5107,"log_dups_size":0,"ondisk_log_size":5107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41792758,"num_objects":5102,"num_object_clones":0,"num_object_copies":20408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":40828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":685376884,"num_objects":83669,"num_object_clones":0,"num_object_copies":334676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83696,"num_write_kb":669450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":17924096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1242763264,"data_stored":1242763264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83723,"ondisk_log_size":83723,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635508,"kb_used_data":333480,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93736332,"statfs":{"total":96636764160,"available":95986003968,"internally_reserved":0,"allocated":341483520,"data_stored":341316335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,4,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":8.9740000000000002},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":9.0009999999999994}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":4.8760000000000003},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.9459999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.9809999999999999},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":9.2300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644408,"kb_used_data":334576,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93727432,"statfs":{"total":96636764160,"available":95976890368,"internally_reserved":0,"allocated":342605824,"data_stored":342438639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":0.221},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":4.6440000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":15.538},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":5.3419999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:46:09 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.7309999999999999,"15min":2.5070000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":0.27100000000000002},{"interface":"front","average":{"1min":4.9009999999999998,"5min":2.3399999999999999,"15min":1.913},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.673999999999999,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.874}]}]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519124,"kb_used_data":208844,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93852716,"statfs":{"total":96636764160,"available":96105181184,"internally_reserved":0,"allocated":213856256,"data_stored":213689071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.224},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.20599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.23599999999999999},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.35499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641156,"kb_used_data":335036,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93730684,"statfs":{"total":96636764160,"available":95980220416,"internally_reserved":0,"allocated":343076864,"data_stored":342909679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,1,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":4.5910000000000002},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":4.6310000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":7.6929999999999996},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.7469999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":0.40400000000000003},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":3.121}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":343638016,"data_stored":343638016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213987328,"data_stored":213987328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":343478272,"data_stored":343478272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":341659648,"data_stored":341659648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:09.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 172 20 84663 84643 3.84372 2.1875 0.0482606 0.0406376 2024-09-18T11:47:10.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 173 20 84981 84961 3.83586 2.48438 0.0397458 0.0407193 2024-09-18T11:47:11.246 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 174 20 85294 85274 3.82785 2.44531 0.0364388 0.0408058 2024-09-18T11:47:12.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:12.246 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 175 20 85583 85563 3.81888 2.25781 0.0507935 0.0408991 2024-09-18T11:47:12.701 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:12.702 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:12.724 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-18T11:47:11.887816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":697787764,"num_objects":85184,"num_object_clones":0,"num_object_copies":340736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85211,"num_write_kb":681570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":18366464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85238,"ondisk_log_size":85238,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2464356,"kb_used_data":1236100,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375023004,"statfs":{"total":386547056640,"available":384023556096,"internally_reserved":0,"allocated":1265766400,"data_stored":1265097660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,21,3,8,12,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24502272,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2991,"num_write_kb":23928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1646592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004093"},"pg_stats":[{"pgid":"1.f","version":"33'5455","reported_seq":5498,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.959069+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:09.959069+0000","last_peered":"2024-09-18T11:47:09.959069+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:09.959069+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44687360,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5455,"num_write_kb":43640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5375","reported_seq":5743,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.553499+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:11.553499+0000","last_peered":"2024-09-18T11:47:11.553499+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:11.553499+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5375,"log_dups_size":0,"ondisk_log_size":5375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43996406,"num_objects":5371,"num_object_clones":0,"num_object_copies":21484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5373,"num_write_kb":42980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5284","reported_seq":5368,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.351382+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:11.351382+0000","last_peered":"2024-09-18T11:47:11.351382+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:11.351382+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5284,"log_dups_size":0,"ondisk_log_size":5284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43260539,"num_objects":5281,"num_object_clones":0,"num_object_copies":21124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5283,"num_write_kb":42254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5306","reported_seq":5370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.494798+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:11.494798+0000","last_peered":"2024-09-18T11:47:11.494798+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:11.494798+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43431158,"num_objects":5302,"num_object_clones":0,"num_object_copies":21208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5304,"num_write_kb":42421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5306","reported_seq":5363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:10.007479+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:10.007479+0000","last_peered":"2024-09-18T11:47:10.007479+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:10.007479+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43439350,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5304,"num_write_kb":42429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5320","reported_seq":5365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.917997+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:09.917997+0000","last_peered":"2024-09-18T11:47:09.917997+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:09.917997+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5320,"log_dups_size":0,"ondisk_log_size":5320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43581440,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5320,"num_write_kb":42560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5373","reported_seq":5440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.557138+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:11.557138+0000","last_peered":"2024-09-18T11:47:11.557138+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:11.557138+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5373,"log_dups_size":0,"ondisk_log_size":5373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43963638,"num_objects":5367,"num_object_clones":0,"num_object_copies":21468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5371,"num_write_kb":42948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3233}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5269","reported_seq":5333,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.497179+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:11.497179+0000","last_peered":"2024-09-18T11:47:11.497179+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:11.497179+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43119862,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5267,"num_write_kb":42124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5408","reported_seq":5468,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.734284+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:09.734284+0000","last_peered":"2024-09-18T11:47:09.734284+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:09.734284+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5408,"log_dups_size":0,"ondisk_log_size":5408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44231148,"num_objects":5400,"num_object_clones":0,"num_object_copies":21600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5405,"num_write_kb":43217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5180","reported_seq":5266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.325841+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:11.325841+0000","last_peered":"2024-09-18T11:47:11.325841+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:11.325841+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42415350,"num_objects":5178,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5178,"num_write_kb":41422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5304","reported_seq":5363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.984833+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:09.984833+0000","last_peered":"2024-09-18T11:47:09.984833+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:09.984833+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43405169,"num_objects":5299,"num_object_clones":0,"num_object_copies":21196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":42396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5402","reported_seq":5439,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.961986+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:09.961986+0000","last_peered":"2024-09-18T11:47:09.961986+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:09.961986+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44243579,"num_objects":5401,"num_object_clones":0,"num_object_copies":21604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5401,"num_write_kb":43207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3292}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5382","reported_seq":5460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.387006+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:11.387006+0000","last_peered":"2024-09-18T11:47:11.387006+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:11.387006+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079739,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":43047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3327}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5331","reported_seq":5381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:10.016166+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:10.016166+0000","last_peered":"2024-09-18T11:47:10.016166+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:10.016166+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43635958,"num_objects":5327,"num_object_clones":0,"num_object_copies":21308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":42628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5333","reported_seq":9618,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:10.134598+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:10.134598+0000","last_peered":"2024-09-18T11:47:10.134598+0000","last_clean":"2024-09-18T11:47:10.134598+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:10.134598+0000","last_undegraded":"2024-09-18T11:47:10.134598+0000","last_fullsized":"2024-09-18T11:47:10.134598+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5333,"log_dups_size":0,"ondisk_log_size":5333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43660534,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5331,"num_write_kb":42645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5210","reported_seq":5274,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.506936+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:11.506936+0000","last_peered":"2024-09-18T11:47:11.506936+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:11.506936+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5210,"log_dups_size":0,"ondisk_log_size":5210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42636534,"num_objects":5205,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5208,"num_write_kb":41652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":697787764,"num_objects":85184,"num_object_clones":0,"num_object_copies":340736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85211,"num_write_kb":681570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":18366464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267531776,"data_stored":1267531776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85238,"ondisk_log_size":85238,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645536,"kb_used_data":339352,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93726304,"statfs":{"total":96636764160,"available":95975735296,"internally_reserved":0,"allocated":347496448,"data_stored":347329263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,2,4,2,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":1.3580000000000001},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.377}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":1.3340000000000001},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.6660000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.25},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":2.7810000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637860,"kb_used_data":340508,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93733980,"statfs":{"total":96636764160,"available":95983595520,"internally_reserved":0,"allocated":348680192,"data_stored":348513007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,0,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":9.5340000000000007},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.665}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.9340000000000002},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":9.4969999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":2.6930000000000001},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":7.6779999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529580,"kb_used_data":215140,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93842260,"statfs":{"total":96636764160,"available":96094474240,"internally_reserved":0,"allocated":220303360,"data_stored":220136175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.40000000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":2.859},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.7830000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.372},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.94499999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651380,"kb_used_data":341100,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93720460,"statfs":{"total":96636764160,"available":95969751040,"internally_reserved":0,"allocated":349286400,"data_stored":349119215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":5.9660000000000002},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":8.0709999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":3.7330000000000001},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.9859999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":5.992},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":349777920,"data_stored":349777920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":220340224,"data_stored":220340224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349589504,"data_stored":349589504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347824128,"data_stored":347824128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:12.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:13.246 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 176 20 85861 85841 3.80952 2.17188 0.169115 0.0409986 2024-09-18T11:47:13.307 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:13.307 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:13.327 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-18T11:47:11.887816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":697787764,"num_objects":85184,"num_object_clones":0,"num_object_copies":340736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85211,"num_write_kb":681570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":18366464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85238,"ondisk_log_size":85238,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2464356,"kb_used_data":1236100,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375023004,"statfs":{"total":386547056640,"available":384023556096,"internally_reserved":0,"allocated":1265766400,"data_stored":1265097660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,21,3,8,12,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24502272,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2991,"num_write_kb":23928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1646592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004093"},"pg_stats":[{"pgid":"1.f","version":"33'5455","reported_seq":5498,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.959069+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:09.959069+0000","last_peered":"2024-09-18T11:47:09.959069+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:09.959069+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44687360,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5455,"num_write_kb":43640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5375","reported_seq":5743,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.553499+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:11.553499+0000","last_peered":"2024-09-18T11:47:11.553499+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:11.553499+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5375,"log_dups_size":0,"ondisk_log_size":5375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43996406,"num_objects":5371,"num_object_clones":0,"num_object_copies":21484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5373,"num_write_kb":42980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5284","reported_seq":5368,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.351382+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:11.351382+0000","last_peered":"2024-09-18T11:47:11.351382+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:11.351382+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5284,"log_dups_size":0,"ondisk_log_size":5284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43260539,"num_objects":5281,"num_object_clones":0,"num_object_copies":21124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5283,"num_write_kb":42254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5306","reported_seq":5370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.494798+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:11.494798+0000","last_peered":"2024-09-18T11:47:11.494798+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:11.494798+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43431158,"num_objects":5302,"num_object_clones":0,"num_object_copies":21208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5304,"num_write_kb":42421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5306","reported_seq":5363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:10.007479+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:10.007479+0000","last_peered":"2024-09-18T11:47:10.007479+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:10.007479+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43439350,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5304,"num_write_kb":42429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5320","reported_seq":5365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.917997+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:09.917997+0000","last_peered":"2024-09-18T11:47:09.917997+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:09.917997+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5320,"log_dups_size":0,"ondisk_log_size":5320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43581440,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5320,"num_write_kb":42560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5373","reported_seq":5440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.557138+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:11.557138+0000","last_peered":"2024-09-18T11:47:11.557138+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:11.557138+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5373,"log_dups_size":0,"ondisk_log_size":5373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43963638,"num_objects":5367,"num_object_clones":0,"num_object_copies":21468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5371,"num_write_kb":42948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3233}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5269","reported_seq":5333,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.497179+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:11.497179+0000","last_peered":"2024-09-18T11:47:11.497179+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:11.497179+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43119862,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5267,"num_write_kb":42124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5408","reported_seq":5468,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.734284+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:09.734284+0000","last_peered":"2024-09-18T11:47:09.734284+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:09.734284+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5408,"log_dups_size":0,"ondisk_log_size":5408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44231148,"num_objects":5400,"num_object_clones":0,"num_object_copies":21600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5405,"num_write_kb":43217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5180","reported_seq":5266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.325841+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:11.325841+0000","last_peered":"2024-09-18T11:47:11.325841+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:11.325841+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42415350,"num_objects":5178,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5178,"num_write_kb":41422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5304","reported_seq":5363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.984833+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:09.984833+0000","last_peered":"2024-09-18T11:47:09.984833+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:09.984833+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43405169,"num_objects":5299,"num_object_clones":0,"num_object_copies":21196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":42396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5402","reported_seq":5439,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.961986+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:09.961986+0000","last_peered":"2024-09-18T11:47:09.961986+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:09.961986+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44243579,"num_objects":5401,"num_object_clones":0,"num_object_copies":21604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5401,"num_write_kb":43207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3292}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5382","reported_seq":5460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.387006+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:11.387006+0000","last_peered":"2024-09-18T11:47:11.387006+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:11.387006+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079739,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":43047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3327}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5331","reported_seq":5381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:10.016166+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:10.016166+0000","last_peered":"2024-09-18T11:47:10.016166+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:10.016166+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43635958,"num_objects":5327,"num_object_clones":0,"num_object_copies":21308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":42628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5333","reported_seq":9618,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:10.134598+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:10.134598+0000","last_peered":"2024-09-18T11:47:10.134598+0000","last_clean":"2024-09-18T11:47:10.134598+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:10.134598+0000","last_undegraded":"2024-09-18T11:47:10.134598+0000","last_fullsized":"2024-09-18T11:47:10.134598+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5333,"log_dups_size":0,"ondisk_log_size":5333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43660534,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5331,"num_write_kb":42645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5210","reported_seq":5274,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.506936+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:11.506936+0000","last_peered":"2024-09-18T11:47:11.506936+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:11.506936+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5210,"log_dups_size":0,"ondisk_log_size":5210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42636534,"num_objects":5205,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5208,"num_write_kb":41652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":697787764,"num_objects":85184,"num_object_clones":0,"num_object_copies":340736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85211,"num_write_kb":681570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":18366464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267531776,"data_stored":1267531776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85238,"ondisk_log_size":85238,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645536,"kb_used_data":339352,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93726304,"statfs":{"total":96636764160,"available":95975735296,"internally_reserved":0,"allocated":347496448,"data_stored":347329263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,2,4,2,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":1.3580000000000001},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.377}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":1.3340000000000001},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.6660000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.25},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":2.7810000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637860,"kb_used_data":340508,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93733980,"statfs":{"total":96636764160,"available":95983595520,"internally_reserved":0,"allocated":348680192,"data_stored":348513007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,0,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":9.5340000000000007},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.665}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.9340000000000002},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":9.4969999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":2.6930000000000001},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":7.6779999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529580,"kb_used_data":215140,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93842260,"statfs":{"total":96636764160,"available":96094474240,"internally_reserved":0,"allocated":220303360,"data_stored":220136175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.40000000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":2.859},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.7830000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.372},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.94499999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651380,"kb_used_data":341100,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93720460,"statfs":{"total":96636764160,"available":95969751040,"internally_reserved":0,"allocated":349286400,"data_stored":349119215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":5.9660000000000002},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":8.0709999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":3.7330000000000001},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.9859999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":5.992},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":349777920,"data_stored":349777920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":220340224,"data_stored":220340224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349589504,"data_stored":349589504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347824128,"data_stored":347824128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:13.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:13.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5455", 'reported_seq': 5498, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:09.959069+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:47:09.959069+0000', 'last_peered': '2024-09-18T11:47:09.959069+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:47:09.959069+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5455, 'log_dups_size': 0, 'ondisk_log_size': 5455, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44687360, 'num_objects': 5455, 'num_object_clones': 0, 'num_object_copies': 21820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5455, 'num_write_kb': 43640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3325}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:13.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:13.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5375", 'reported_seq': 5743, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:11.553499+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:11.553499+0000', 'last_peered': '2024-09-18T11:47:11.553499+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:11.553499+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5375, 'log_dups_size': 0, 'ondisk_log_size': 5375, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43996406, 'num_objects': 5371, 'num_object_clones': 0, 'num_object_copies': 21484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1951, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5373, 'num_write_kb': 42980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 153, 'num_bytes_recovered': 1236992, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3420}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:13.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:13.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5284", 'reported_seq': 5368, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:11.351382+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:11.351382+0000', 'last_peered': '2024-09-18T11:47:11.351382+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:11.351382+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5284, 'log_dups_size': 0, 'ondisk_log_size': 5284, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43260539, 'num_objects': 5281, 'num_object_clones': 0, 'num_object_copies': 21124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5283, 'num_write_kb': 42254, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3227}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:13.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5306", 'reported_seq': 5370, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:11.494798+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:11.494798+0000', 'last_peered': '2024-09-18T11:47:11.494798+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:11.494798+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5306, 'log_dups_size': 0, 'ondisk_log_size': 5306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43431158, 'num_objects': 5302, 'num_object_clones': 0, 'num_object_copies': 21208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5302, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5304, 'num_write_kb': 42421, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5306", 'reported_seq': 5363, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:10.007479+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:47:10.007479+0000', 'last_peered': '2024-09-18T11:47:10.007479+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:47:10.007479+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5306, 'log_dups_size': 0, 'ondisk_log_size': 5306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43439350, 'num_objects': 5303, 'num_object_clones': 0, 'num_object_copies': 21212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5304, 'num_write_kb': 42429, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3221}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5320", 'reported_seq': 5365, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:09.917997+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:47:09.917997+0000', 'last_peered': '2024-09-18T11:47:09.917997+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:47:09.917997+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5320, 'log_dups_size': 0, 'ondisk_log_size': 5320, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43581440, 'num_objects': 5320, 'num_object_clones': 0, 'num_object_copies': 21280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5320, 'num_write_kb': 42560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3166}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5373", 'reported_seq': 5440, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:11.557138+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:11.557138+0000', 'last_peered': '2024-09-18T11:47:11.557138+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:11.557138+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5373, 'log_dups_size': 0, 'ondisk_log_size': 5373, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43963638, 'num_objects': 5367, 'num_object_clones': 0, 'num_object_copies': 21468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5371, 'num_write_kb': 42948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3233}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5269", 'reported_seq': 5333, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:11.497179+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:11.497179+0000', 'last_peered': '2024-09-18T11:47:11.497179+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:11.497179+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5269, 'log_dups_size': 0, 'ondisk_log_size': 5269, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43119862, 'num_objects': 5264, 'num_object_clones': 0, 'num_object_copies': 21056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5267, 'num_write_kb': 42124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:13.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5408", 'reported_seq': 5468, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:09.734284+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:47:09.734284+0000', 'last_peered': '2024-09-18T11:47:09.734284+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:47:09.734284+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5408, 'log_dups_size': 0, 'ondisk_log_size': 5408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44231148, 'num_objects': 5400, 'num_object_clones': 0, 'num_object_copies': 21600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5405, 'num_write_kb': 43217, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3265}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:13.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:13.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5180", 'reported_seq': 5266, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:11.325841+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:11.325841+0000', 'last_peered': '2024-09-18T11:47:11.325841+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:11.325841+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5180, 'log_dups_size': 0, 'ondisk_log_size': 5180, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42415350, 'num_objects': 5178, 'num_object_clones': 0, 'num_object_copies': 20712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5178, 'num_write_kb': 41422, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3203}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:13.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:13.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5304", 'reported_seq': 5363, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:09.984833+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:47:09.984833+0000', 'last_peered': '2024-09-18T11:47:09.984833+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:47:09.984833+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5304, 'log_dups_size': 0, 'ondisk_log_size': 5304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43405169, 'num_objects': 5299, 'num_object_clones': 0, 'num_object_copies': 21196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5301, 'num_write_kb': 42396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3236}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:13.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:13.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5402", 'reported_seq': 5439, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:09.961986+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:47:09.961986+0000', 'last_peered': '2024-09-18T11:47:09.961986+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:47:09.961986+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5402, 'log_dups_size': 0, 'ondisk_log_size': 5402, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44243579, 'num_objects': 5401, 'num_object_clones': 0, 'num_object_copies': 21604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5401, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5401, 'num_write_kb': 43207, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3292}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:13.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:13.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5382", 'reported_seq': 5460, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:11.387006+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:11.387006+0000', 'last_peered': '2024-09-18T11:47:11.387006+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:11.387006+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5382, 'log_dups_size': 0, 'ondisk_log_size': 5382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44079739, 'num_objects': 5381, 'num_object_clones': 0, 'num_object_copies': 21524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5381, 'num_write_kb': 43047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3327}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:13.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:13.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5331", 'reported_seq': 5381, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:10.016166+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:47:10.016166+0000', 'last_peered': '2024-09-18T11:47:10.016166+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:47:10.016166+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5331, 'log_dups_size': 0, 'ondisk_log_size': 5331, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43635958, 'num_objects': 5327, 'num_object_clones': 0, 'num_object_copies': 21308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5329, 'num_write_kb': 42628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3169}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:13.331 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:13.331 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5210", 'reported_seq': 5274, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:11.506936+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:11.506936+0000', 'last_peered': '2024-09-18T11:47:11.506936+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:11.506936+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5210, 'log_dups_size': 0, 'ondisk_log_size': 5210, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42636534, 'num_objects': 5205, 'num_object_clones': 0, 'num_object_copies': 20820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5208, 'num_write_kb': 41652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3192}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:13.331 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:14.017 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:14.017 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31183/340736 objects degraded (9.152%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":232,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":85184,"data_bytes":697787764,"bytes_used":2523500544,"bytes_avail":384023556096,"bytes_total":386547056640,"degraded_objects":31183,"degraded_total":340736,"degraded_ratio":0.091516599361382414,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":164591,"recovering_keys_per_sec":0,"num_objects_recovered":201,"num_bytes_recovered":1646592,"num_keys_recovered":0,"write_bytes_sec":2449224,"read_op_per_sec":0,"write_op_per_sec":298},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:14.037 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:14.038 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:14.246 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 177 20 86141 86121 3.80036 2.1875 0.0612205 0.0411006 2024-09-18T11:47:14.584 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:14.584 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:14.608 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-18T11:47:13.889053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":697787764,"num_objects":85184,"num_object_clones":0,"num_object_copies":340736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85211,"num_write_kb":681570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":18366464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85238,"ondisk_log_size":85238,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2464356,"kb_used_data":1236100,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375023004,"statfs":{"total":386547056640,"available":384023556096,"internally_reserved":0,"allocated":1265766400,"data_stored":1265097660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,21,3,8,12,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24502272,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2991,"num_write_kb":23928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1646592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.821518"},"pg_stats":[{"pgid":"1.f","version":"33'5455","reported_seq":5498,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.959069+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:09.959069+0000","last_peered":"2024-09-18T11:47:09.959069+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:09.959069+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44687360,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5455,"num_write_kb":43640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5375","reported_seq":5743,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.553499+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:11.553499+0000","last_peered":"2024-09-18T11:47:11.553499+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:11.553499+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5375,"log_dups_size":0,"ondisk_log_size":5375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43996406,"num_objects":5371,"num_object_clones":0,"num_object_copies":21484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5373,"num_write_kb":42980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5284","reported_seq":5368,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.351382+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:11.351382+0000","last_peered":"2024-09-18T11:47:11.351382+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:11.351382+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5284,"log_dups_size":0,"ondisk_log_size":5284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43260539,"num_objects":5281,"num_object_clones":0,"num_object_copies":21124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5283,"num_write_kb":42254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5306","reported_seq":5370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.494798+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:11.494798+0000","last_peered":"2024-09-18T11:47:11.494798+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:11.494798+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43431158,"num_objects":5302,"num_object_clones":0,"num_object_copies":21208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5304,"num_write_kb":42421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5306","reported_seq":5363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:10.007479+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:10.007479+0000","last_peered":"2024-09-18T11:47:10.007479+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:10.007479+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43439350,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5304,"num_write_kb":42429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5320","reported_seq":5365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.917997+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:09.917997+0000","last_peered":"2024-09-18T11:47:09.917997+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:09.917997+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5320,"log_dups_size":0,"ondisk_log_size":5320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43581440,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5320,"num_write_kb":42560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5373","reported_seq":5440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.557138+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:11.557138+0000","last_peered":"2024-09-18T11:47:11.557138+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:11.557138+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5373,"log_dups_size":0,"ondisk_log_size":5373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43963638,"num_objects":5367,"num_object_clones":0,"num_object_copies":21468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5371,"num_write_kb":42948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3233}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5269","reported_seq":5333,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.497179+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:11.497179+0000","last_peered":"2024-09-18T11:47:11.497179+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:11.497179+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43119862,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5267,"num_write_kb":42124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5408","reported_seq":5468,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.734284+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:09.734284+0000","last_peered":"2024-09-18T11:47:09.734284+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:09.734284+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5408,"log_dups_size":0,"ondisk_log_size":5408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44231148,"num_objects":5400,"num_object_clones":0,"num_object_copies":21600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5405,"num_write_kb":43217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5180","reported_seq":5266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.325841+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:11.325841+0000","last_peered":"2024-09-18T11:47:11.325841+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:11.325841+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42415350,"num_objects":5178,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5178,"num_write_kb":41422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5304","reported_seq":5363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.984833+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:09.984833+0000","last_peered":"2024-09-18T11:47:09.984833+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:09.984833+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43405169,"num_objects":5299,"num_object_clones":0,"num_object_copies":21196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":42396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5402","reported_seq":5439,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:09.961986+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:09.961986+0000","last_peered":"2024-09-18T11:47:09.961986+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:09.961986+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44243579,"num_objects":5401,"num_object_clones":0,"num_object_copies":21604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5401,"num_write_kb":43207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3292}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5382","reported_seq":5460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.387006+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:11.387006+0000","last_peered":"2024-09-18T11:47:11.387006+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:11.387006+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079739,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":43047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3327}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5331","reported_seq":5381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:10.016166+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:10.016166+0000","last_peered":"2024-09-18T11:47:10.016166+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:10.016166+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43635958,"num_objects":5327,"num_object_clones":0,"num_object_copies":21308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":42628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5333","reported_seq":9618,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:10.134598+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:10.134598+0000","last_peered":"2024-09-18T11:47:10.134598+0000","last_clean":"2024-09-18T11:47:10.134598+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:10.134598+0000","last_undegraded":"2024-09-18T11:47:10.134598+0000","last_fullsized":"2024-09-18T11:47:10.134598+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5333,"log_dups_size":0,"ondisk_log_size":5333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43660534,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5331,"num_write_kb":42645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5210","reported_seq":5274,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:11.506936+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:11.506936+0000","last_peered":"2024-09-18T11:47:11.506936+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:11.506936+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5210,"log_dups_size":0,"ondisk_log_size":5210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42636534,"num_objects":5205,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5208,"num_write_kb":41652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":697787764,"num_objects":85184,"num_object_clones":0,"num_object_copies":340736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85211,"num_write_kb":681570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":18366464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267531776,"data_stored":1267531776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85238,"ondisk_log_size":85238,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645536,"kb_used_data":339352,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93726304,"statfs":{"total":96636764160,"available":95975735296,"internally_reserved":0,"allocated":347496448,"data_stored":347329263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,2,4,2,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.8170000000000002,"5min":3.0049999999999999,"15min":2.8690000000000002},"min":{"1min":0.254,"5min":0.155,"15min":0.155},"max":{"1min":12.866,"5min":12.866,"15min":12.866},"last":1.3580000000000001},{"interface":"front","average":{"1min":5.7699999999999996,"5min":3.294,"15min":2.8809999999999998},"min":{"1min":0.33500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":31.675000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.377}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":1.3340000000000001},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.6660000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:46:11 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.5149999999999999,"15min":1.163},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.25},{"interface":"front","average":{"1min":4.9459999999999997,"5min":3.0619999999999998,"15min":2.7480000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":2.7810000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637860,"kb_used_data":340508,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93733980,"statfs":{"total":96636764160,"available":95983595520,"internally_reserved":0,"allocated":348680192,"data_stored":348513007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,0,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":9.5340000000000007},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.665}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.9340000000000002},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":9.4969999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":2.6930000000000001},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":7.6779999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529580,"kb_used_data":215140,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93842260,"statfs":{"total":96636764160,"available":96094474240,"internally_reserved":0,"allocated":220303360,"data_stored":220136175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.40000000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":2.859},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.7830000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.372},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.94499999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651380,"kb_used_data":341100,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93720460,"statfs":{"total":96636764160,"available":95969751040,"internally_reserved":0,"allocated":349286400,"data_stored":349119215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":5.9660000000000002},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":8.0709999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":3.7330000000000001},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.9859999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":2.431,"15min":2.3839999999999999},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.4059999999999997,"5min":12.266,"15min":12.266},"last":5.992},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.0419999999999998,"15min":1.694},"min":{"1min":0.43099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":349777920,"data_stored":349777920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":220340224,"data_stored":220340224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349589504,"data_stored":349589504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347824128,"data_stored":347824128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:15.246 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 178 20 86439 86419 3.79209 2.32812 0.0434572 0.0411928 2024-09-18T11:47:16.249 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 179 20 86755 86735 3.7847 2.46875 0.0554316 0.0412734 2024-09-18T11:47:17.247 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:47:17.246031+0000 min lat: 0.00272854 max lat: 2.42146 avg lat: 0.0413589 2024-09-18T11:47:17.247 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:47:17.247 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 180 20 87059 87039 3.77686 2.375 0.0637584 0.0413589 2024-09-18T11:47:17.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:18.134 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:18.134 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:18.162 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-18T11:47:17.892192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":709772660,"num_objects":86647,"num_object_clones":0,"num_object_copies":346588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86674,"num_write_kb":693274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2318,"num_bytes_recovered":18751488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86701,"ondisk_log_size":86701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2471308,"kb_used_data":1259692,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375016052,"statfs":{"total":386547056640,"available":384016437248,"internally_reserved":0,"allocated":1289924608,"data_stored":1289255868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,22,7,10,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30089216,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3673,"num_write_kb":29384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006648"},"pg_stats":[{"pgid":"1.f","version":"33'5555","reported_seq":5598,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.092083+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:15.092083+0000","last_peered":"2024-09-18T11:47:15.092083+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:15.092083+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45506560,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5555,"num_write_kb":44440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5476","reported_seq":5938,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.563188+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:16.563188+0000","last_peered":"2024-09-18T11:47:16.563188+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:16.563188+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44823798,"num_objects":5472,"num_object_clones":0,"num_object_copies":21888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3568}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5384","reported_seq":5468,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.368228+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:16.368228+0000","last_peered":"2024-09-18T11:47:16.368228+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:16.368228+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079739,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5383,"num_write_kb":43054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3327}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5401","reported_seq":5465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.429424+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:16.429424+0000","last_peered":"2024-09-18T11:47:16.429424+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:16.429424+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44209398,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5393","reported_seq":5450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:14.965318+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:14.965318+0000","last_peered":"2024-09-18T11:47:14.965318+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:14.965318+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44152054,"num_objects":5390,"num_object_clones":0,"num_object_copies":21560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5398","reported_seq":5443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.029273+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:15.029273+0000","last_peered":"2024-09-18T11:47:15.029273+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:15.029273+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44220416,"num_objects":5398,"num_object_clones":0,"num_object_copies":21592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5398,"num_write_kb":43184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5491","reported_seq":5558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.597404+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:16.597404+0000","last_peered":"2024-09-18T11:47:16.597404+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:16.597404+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44930294,"num_objects":5485,"num_object_clones":0,"num_object_copies":21940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5489,"num_write_kb":43892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5363","reported_seq":5428,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.574295+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:16.574295+0000","last_peered":"2024-09-18T11:47:16.574295+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:16.574295+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5363,"log_dups_size":0,"ondisk_log_size":5363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43889910,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5361,"num_write_kb":42876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5486","reported_seq":5546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.056283+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:15.056283+0000","last_peered":"2024-09-18T11:47:15.056283+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:15.056283+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5486,"log_dups_size":0,"ondisk_log_size":5486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44870124,"num_objects":5478,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":43841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3343}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5272","reported_seq":5358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.352955+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:16.352955+0000","last_peered":"2024-09-18T11:47:16.352955+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:16.352955+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5272,"log_dups_size":0,"ondisk_log_size":5272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169014,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5270,"num_write_kb":42158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3295}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5400","reported_seq":5457,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.019196+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:15.019196+0000","last_peered":"2024-09-18T11:47:15.019196+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:15.019196+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5400,"log_dups_size":0,"ondisk_log_size":5400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44191601,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":43164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5480","reported_seq":5519,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:14.981855+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:14.981855+0000","last_peered":"2024-09-18T11:47:14.981855+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:14.981855+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44882555,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":43831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3370}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5476","reported_seq":5554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.449908+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:16.449908+0000","last_peered":"2024-09-18T11:47:16.449908+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:16.449908+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44849787,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5475,"num_write_kb":43799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5414","reported_seq":5465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.045244+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:15.045244+0000","last_peered":"2024-09-18T11:47:15.045244+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:15.045244+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5414,"log_dups_size":0,"ondisk_log_size":5414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44315894,"num_objects":5410,"num_object_clones":0,"num_object_copies":21640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5412,"num_write_kb":43292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3252}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5422","reported_seq":9706,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:15.116018+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:15.116018+0000","last_peered":"2024-09-18T11:47:15.116018+0000","last_clean":"2024-09-18T11:47:15.116018+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:15.116018+0000","last_undegraded":"2024-09-18T11:47:15.116018+0000","last_fullsized":"2024-09-18T11:47:15.116018+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389622,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5420,"num_write_kb":43357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5290","reported_seq":5354,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.399787+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:16.399787+0000","last_peered":"2024-09-18T11:47:16.399787+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:16.399787+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43291894,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":42292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":709772660,"num_objects":86647,"num_object_clones":0,"num_object_copies":346588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86674,"num_write_kb":693274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2318,"num_bytes_recovered":18751488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1291812864,"data_stored":1291812864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86701,"ondisk_log_size":86701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655496,"kb_used_data":345152,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93716344,"statfs":{"total":96636764160,"available":95965536256,"internally_reserved":0,"allocated":353435648,"data_stored":353268463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,13,2,2,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":7.5380000000000003},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.5650000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":11.385999999999999},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":11.757}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":11.487},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":3.0950000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":647796,"kb_used_data":346284,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93724044,"statfs":{"total":96636764160,"available":95973421056,"internally_reserved":0,"allocated":354594816,"data_stored":354427631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,8,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":13.643000000000001},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":5.6959999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":4.7770000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":13.603999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":5.7229999999999999},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.252}]}]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523200,"kb_used_data":221240,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93848640,"statfs":{"total":96636764160,"available":96101007360,"internally_reserved":0,"allocated":226549760,"data_stored":226382575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.40000000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":2.859},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.7830000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.372},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.94499999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644816,"kb_used_data":347016,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93727024,"statfs":{"total":96636764160,"available":95976472576,"internally_reserved":0,"allocated":355344384,"data_stored":355177199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.8740000000000001},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":10.762}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":9.7170000000000005},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":10.701000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":2.9039999999999999},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":10.736000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":355815424,"data_stored":355815424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226525184,"data_stored":226525184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":355602432,"data_stored":355602432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":353869824,"data_stored":353869824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:18.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:18.246 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 181 20 87344 87324 3.7683 2.22656 0.0419836 0.0414518 2024-09-18T11:47:18.718 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:18.718 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:18.745 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-18T11:47:17.892192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":709772660,"num_objects":86647,"num_object_clones":0,"num_object_copies":346588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86674,"num_write_kb":693274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2318,"num_bytes_recovered":18751488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86701,"ondisk_log_size":86701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2471308,"kb_used_data":1259692,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375016052,"statfs":{"total":386547056640,"available":384016437248,"internally_reserved":0,"allocated":1289924608,"data_stored":1289255868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,22,7,10,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30089216,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3673,"num_write_kb":29384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006648"},"pg_stats":[{"pgid":"1.f","version":"33'5555","reported_seq":5598,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.092083+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:15.092083+0000","last_peered":"2024-09-18T11:47:15.092083+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:15.092083+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45506560,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5555,"num_write_kb":44440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5476","reported_seq":5938,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.563188+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:16.563188+0000","last_peered":"2024-09-18T11:47:16.563188+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:16.563188+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44823798,"num_objects":5472,"num_object_clones":0,"num_object_copies":21888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3568}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5384","reported_seq":5468,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.368228+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:16.368228+0000","last_peered":"2024-09-18T11:47:16.368228+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:16.368228+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079739,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5383,"num_write_kb":43054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3327}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5401","reported_seq":5465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.429424+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:16.429424+0000","last_peered":"2024-09-18T11:47:16.429424+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:16.429424+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44209398,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5393","reported_seq":5450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:14.965318+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:14.965318+0000","last_peered":"2024-09-18T11:47:14.965318+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:14.965318+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44152054,"num_objects":5390,"num_object_clones":0,"num_object_copies":21560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5398","reported_seq":5443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.029273+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:15.029273+0000","last_peered":"2024-09-18T11:47:15.029273+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:15.029273+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44220416,"num_objects":5398,"num_object_clones":0,"num_object_copies":21592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5398,"num_write_kb":43184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5491","reported_seq":5558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.597404+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:16.597404+0000","last_peered":"2024-09-18T11:47:16.597404+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:16.597404+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44930294,"num_objects":5485,"num_object_clones":0,"num_object_copies":21940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5489,"num_write_kb":43892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5363","reported_seq":5428,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.574295+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:16.574295+0000","last_peered":"2024-09-18T11:47:16.574295+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:16.574295+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5363,"log_dups_size":0,"ondisk_log_size":5363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43889910,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5361,"num_write_kb":42876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5486","reported_seq":5546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.056283+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:15.056283+0000","last_peered":"2024-09-18T11:47:15.056283+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:15.056283+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5486,"log_dups_size":0,"ondisk_log_size":5486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44870124,"num_objects":5478,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":43841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3343}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5272","reported_seq":5358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.352955+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:16.352955+0000","last_peered":"2024-09-18T11:47:16.352955+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:16.352955+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5272,"log_dups_size":0,"ondisk_log_size":5272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169014,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5270,"num_write_kb":42158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3295}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5400","reported_seq":5457,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.019196+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:15.019196+0000","last_peered":"2024-09-18T11:47:15.019196+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:15.019196+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5400,"log_dups_size":0,"ondisk_log_size":5400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44191601,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":43164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5480","reported_seq":5519,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:14.981855+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:14.981855+0000","last_peered":"2024-09-18T11:47:14.981855+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:14.981855+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44882555,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":43831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3370}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5476","reported_seq":5554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.449908+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:16.449908+0000","last_peered":"2024-09-18T11:47:16.449908+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:16.449908+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44849787,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5475,"num_write_kb":43799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5414","reported_seq":5465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.045244+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:15.045244+0000","last_peered":"2024-09-18T11:47:15.045244+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:15.045244+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5414,"log_dups_size":0,"ondisk_log_size":5414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44315894,"num_objects":5410,"num_object_clones":0,"num_object_copies":21640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5412,"num_write_kb":43292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3252}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5422","reported_seq":9706,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:15.116018+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:15.116018+0000","last_peered":"2024-09-18T11:47:15.116018+0000","last_clean":"2024-09-18T11:47:15.116018+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:15.116018+0000","last_undegraded":"2024-09-18T11:47:15.116018+0000","last_fullsized":"2024-09-18T11:47:15.116018+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389622,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5420,"num_write_kb":43357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5290","reported_seq":5354,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.399787+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:16.399787+0000","last_peered":"2024-09-18T11:47:16.399787+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:16.399787+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43291894,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":42292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":709772660,"num_objects":86647,"num_object_clones":0,"num_object_copies":346588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86674,"num_write_kb":693274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2318,"num_bytes_recovered":18751488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1291812864,"data_stored":1291812864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86701,"ondisk_log_size":86701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655496,"kb_used_data":345152,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93716344,"statfs":{"total":96636764160,"available":95965536256,"internally_reserved":0,"allocated":353435648,"data_stored":353268463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,13,2,2,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":7.5380000000000003},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.5650000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":11.385999999999999},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":11.757}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":11.487},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":3.0950000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":647796,"kb_used_data":346284,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93724044,"statfs":{"total":96636764160,"available":95973421056,"internally_reserved":0,"allocated":354594816,"data_stored":354427631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,8,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":13.643000000000001},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":5.6959999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":4.7770000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":13.603999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":5.7229999999999999},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.252}]}]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523200,"kb_used_data":221240,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93848640,"statfs":{"total":96636764160,"available":96101007360,"internally_reserved":0,"allocated":226549760,"data_stored":226382575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.40000000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":2.859},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.7830000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.372},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.94499999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644816,"kb_used_data":347016,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93727024,"statfs":{"total":96636764160,"available":95976472576,"internally_reserved":0,"allocated":355344384,"data_stored":355177199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.8740000000000001},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":10.762}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":9.7170000000000005},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":10.701000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":2.9039999999999999},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":10.736000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":355815424,"data_stored":355815424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226525184,"data_stored":226525184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":355602432,"data_stored":355602432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":353869824,"data_stored":353869824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:18.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:18.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5555", 'reported_seq': 5598, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:15.092083+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:47:15.092083+0000', 'last_peered': '2024-09-18T11:47:15.092083+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:47:15.092083+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5555, 'log_dups_size': 0, 'ondisk_log_size': 5555, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45506560, 'num_objects': 5555, 'num_object_clones': 0, 'num_object_copies': 22220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5555, 'num_write_kb': 44440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3425}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:18.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:18.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5476", 'reported_seq': 5938, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:16.563188+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:16.563188+0000', 'last_peered': '2024-09-18T11:47:16.563188+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:16.563188+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5476, 'log_dups_size': 0, 'ondisk_log_size': 5476, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44823798, 'num_objects': 5472, 'num_object_clones': 0, 'num_object_copies': 21888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5474, 'num_write_kb': 43788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 200, 'num_bytes_recovered': 1622016, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3568}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:18.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:18.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5384", 'reported_seq': 5468, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:16.368228+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:16.368228+0000', 'last_peered': '2024-09-18T11:47:16.368228+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:16.368228+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5384, 'log_dups_size': 0, 'ondisk_log_size': 5384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44079739, 'num_objects': 5381, 'num_object_clones': 0, 'num_object_copies': 21524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5383, 'num_write_kb': 43054, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3327}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:18.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:18.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5401", 'reported_seq': 5465, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:16.429424+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:16.429424+0000', 'last_peered': '2024-09-18T11:47:16.429424+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:16.429424+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5401, 'log_dups_size': 0, 'ondisk_log_size': 5401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44209398, 'num_objects': 5397, 'num_object_clones': 0, 'num_object_copies': 21588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5399, 'num_write_kb': 43181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3283}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:18.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:18.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5393", 'reported_seq': 5450, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:14.965318+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:47:14.965318+0000', 'last_peered': '2024-09-18T11:47:14.965318+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:47:14.965318+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5393, 'log_dups_size': 0, 'ondisk_log_size': 5393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44152054, 'num_objects': 5390, 'num_object_clones': 0, 'num_object_copies': 21560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5391, 'num_write_kb': 43125, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3308}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:18.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5398", 'reported_seq': 5443, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:15.029273+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:47:15.029273+0000', 'last_peered': '2024-09-18T11:47:15.029273+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:47:15.029273+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5398, 'log_dups_size': 0, 'ondisk_log_size': 5398, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44220416, 'num_objects': 5398, 'num_object_clones': 0, 'num_object_copies': 21592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5398, 'num_write_kb': 43184, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3244}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5491", 'reported_seq': 5558, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:16.597404+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:16.597404+0000', 'last_peered': '2024-09-18T11:47:16.597404+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:16.597404+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5491, 'log_dups_size': 0, 'ondisk_log_size': 5491, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44930294, 'num_objects': 5485, 'num_object_clones': 0, 'num_object_copies': 21940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5489, 'num_write_kb': 43892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5363", 'reported_seq': 5428, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:16.574295+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:16.574295+0000', 'last_peered': '2024-09-18T11:47:16.574295+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:16.574295+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5363, 'log_dups_size': 0, 'ondisk_log_size': 5363, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43889910, 'num_objects': 5358, 'num_object_clones': 0, 'num_object_copies': 21432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5361, 'num_write_kb': 42876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3324}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5486", 'reported_seq': 5546, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:15.056283+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:47:15.056283+0000', 'last_peered': '2024-09-18T11:47:15.056283+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:47:15.056283+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5486, 'log_dups_size': 0, 'ondisk_log_size': 5486, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44870124, 'num_objects': 5478, 'num_object_clones': 0, 'num_object_copies': 21912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5483, 'num_write_kb': 43841, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3343}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5272", 'reported_seq': 5358, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:16.352955+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:16.352955+0000', 'last_peered': '2024-09-18T11:47:16.352955+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:16.352955+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5272, 'log_dups_size': 0, 'ondisk_log_size': 5272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43169014, 'num_objects': 5270, 'num_object_clones': 0, 'num_object_copies': 21080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5270, 'num_write_kb': 42158, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3295}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:18.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5400", 'reported_seq': 5457, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:15.019196+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:47:15.019196+0000', 'last_peered': '2024-09-18T11:47:15.019196+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:47:15.019196+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5400, 'log_dups_size': 0, 'ondisk_log_size': 5400, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44191601, 'num_objects': 5395, 'num_object_clones': 0, 'num_object_copies': 21580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5397, 'num_write_kb': 43164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3332}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:18.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:18.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5480", 'reported_seq': 5519, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:14.981855+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:47:14.981855+0000', 'last_peered': '2024-09-18T11:47:14.981855+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:47:14.981855+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5480, 'log_dups_size': 0, 'ondisk_log_size': 5480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44882555, 'num_objects': 5479, 'num_object_clones': 0, 'num_object_copies': 21916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5479, 'num_write_kb': 43831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3370}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:18.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:18.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5476", 'reported_seq': 5554, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:16.449908+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:16.449908+0000', 'last_peered': '2024-09-18T11:47:16.449908+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:16.449908+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5476, 'log_dups_size': 0, 'ondisk_log_size': 5476, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44849787, 'num_objects': 5475, 'num_object_clones': 0, 'num_object_copies': 21900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5475, 'num_write_kb': 43799, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:18.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:18.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5414", 'reported_seq': 5465, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:15.045244+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:47:15.045244+0000', 'last_peered': '2024-09-18T11:47:15.045244+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:47:15.045244+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5414, 'log_dups_size': 0, 'ondisk_log_size': 5414, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44315894, 'num_objects': 5410, 'num_object_clones': 0, 'num_object_copies': 21640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5412, 'num_write_kb': 43292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:18.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:18.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5290", 'reported_seq': 5354, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:16.399787+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:16.399787+0000', 'last_peered': '2024-09-18T11:47:16.399787+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:16.399787+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5290, 'log_dups_size': 0, 'ondisk_log_size': 5290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43291894, 'num_objects': 5285, 'num_object_clones': 0, 'num_object_copies': 21140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5285, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5288, 'num_write_kb': 42292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:18.750 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:19.255 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 182 20 87622 87602 3.75947 2.17188 0.0788016 0.0415489 2024-09-18T11:47:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:19.393 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31136/346588 objects degraded (8.984%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":237,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":86647,"data_bytes":709772660,"bytes_used":2530619392,"bytes_avail":384016437248,"bytes_total":386547056640,"degraded_objects":31136,"degraded_total":346588,"degraded_ratio":0.089835770424827169,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":124176,"recovering_keys_per_sec":0,"num_objects_recovered":182,"num_bytes_recovered":1490944,"num_keys_recovered":0,"write_bytes_sec":2506046,"read_op_per_sec":0,"write_op_per_sec":305},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:19.410 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:19.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:19.989 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:19.990 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:20.016 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-18T11:47:19.893441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":709772660,"num_objects":86647,"num_object_clones":0,"num_object_copies":346588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86674,"num_write_kb":693274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2318,"num_bytes_recovered":18751488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86701,"ondisk_log_size":86701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2471308,"kb_used_data":1259692,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375016052,"statfs":{"total":386547056640,"available":384016437248,"internally_reserved":0,"allocated":1289924608,"data_stored":1289255868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,22,7,10,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24395776,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2978,"num_write_kb":23824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":827392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007114"},"pg_stats":[{"pgid":"1.f","version":"33'5555","reported_seq":5598,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.092083+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:15.092083+0000","last_peered":"2024-09-18T11:47:15.092083+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:15.092083+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45506560,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5555,"num_write_kb":44440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5476","reported_seq":5938,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.563188+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:16.563188+0000","last_peered":"2024-09-18T11:47:16.563188+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:16.563188+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44823798,"num_objects":5472,"num_object_clones":0,"num_object_copies":21888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3568}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5384","reported_seq":5468,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.368228+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:16.368228+0000","last_peered":"2024-09-18T11:47:16.368228+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:16.368228+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079739,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5383,"num_write_kb":43054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3327}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5401","reported_seq":5465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.429424+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:16.429424+0000","last_peered":"2024-09-18T11:47:16.429424+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:16.429424+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44209398,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5393","reported_seq":5450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:14.965318+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:14.965318+0000","last_peered":"2024-09-18T11:47:14.965318+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:14.965318+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44152054,"num_objects":5390,"num_object_clones":0,"num_object_copies":21560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5398","reported_seq":5443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.029273+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:15.029273+0000","last_peered":"2024-09-18T11:47:15.029273+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:15.029273+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44220416,"num_objects":5398,"num_object_clones":0,"num_object_copies":21592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5398,"num_write_kb":43184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5491","reported_seq":5558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.597404+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:16.597404+0000","last_peered":"2024-09-18T11:47:16.597404+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:16.597404+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44930294,"num_objects":5485,"num_object_clones":0,"num_object_copies":21940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5489,"num_write_kb":43892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5363","reported_seq":5428,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.574295+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:16.574295+0000","last_peered":"2024-09-18T11:47:16.574295+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:16.574295+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5363,"log_dups_size":0,"ondisk_log_size":5363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43889910,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5361,"num_write_kb":42876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5486","reported_seq":5546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.056283+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:15.056283+0000","last_peered":"2024-09-18T11:47:15.056283+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:15.056283+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5486,"log_dups_size":0,"ondisk_log_size":5486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44870124,"num_objects":5478,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":43841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3343}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5272","reported_seq":5358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.352955+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:16.352955+0000","last_peered":"2024-09-18T11:47:16.352955+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:16.352955+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5272,"log_dups_size":0,"ondisk_log_size":5272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169014,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5270,"num_write_kb":42158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3295}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5400","reported_seq":5457,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.019196+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:15.019196+0000","last_peered":"2024-09-18T11:47:15.019196+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:15.019196+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5400,"log_dups_size":0,"ondisk_log_size":5400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44191601,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":43164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5480","reported_seq":5519,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:14.981855+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:14.981855+0000","last_peered":"2024-09-18T11:47:14.981855+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:14.981855+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44882555,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":43831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3370}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5476","reported_seq":5554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.449908+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:16.449908+0000","last_peered":"2024-09-18T11:47:16.449908+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:16.449908+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44849787,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5475,"num_write_kb":43799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5414","reported_seq":5465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:15.045244+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:15.045244+0000","last_peered":"2024-09-18T11:47:15.045244+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:15.045244+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5414,"log_dups_size":0,"ondisk_log_size":5414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44315894,"num_objects":5410,"num_object_clones":0,"num_object_copies":21640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5412,"num_write_kb":43292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3252}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5422","reported_seq":9706,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:15.116018+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:15.116018+0000","last_peered":"2024-09-18T11:47:15.116018+0000","last_clean":"2024-09-18T11:47:15.116018+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:15.116018+0000","last_undegraded":"2024-09-18T11:47:15.116018+0000","last_fullsized":"2024-09-18T11:47:15.116018+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389622,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5420,"num_write_kb":43357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5290","reported_seq":5354,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:16.399787+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:16.399787+0000","last_peered":"2024-09-18T11:47:16.399787+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:16.399787+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43291894,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":42292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":709772660,"num_objects":86647,"num_object_clones":0,"num_object_copies":346588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86674,"num_write_kb":693274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2318,"num_bytes_recovered":18751488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1291812864,"data_stored":1291812864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86701,"ondisk_log_size":86701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655496,"kb_used_data":345152,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93716344,"statfs":{"total":96636764160,"available":95965536256,"internally_reserved":0,"allocated":353435648,"data_stored":353268463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,13,2,2,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":7.5380000000000003},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.5650000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":11.385999999999999},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":11.757}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":11.487},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":3.0950000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":647796,"kb_used_data":346284,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93724044,"statfs":{"total":96636764160,"available":95973421056,"internally_reserved":0,"allocated":354594816,"data_stored":354427631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,8,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":13.643000000000001},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":5.6959999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":4.7770000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":13.603999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":5.7229999999999999},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.252}]}]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523200,"kb_used_data":221240,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93848640,"statfs":{"total":96636764160,"available":96101007360,"internally_reserved":0,"allocated":226549760,"data_stored":226382575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.40000000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":2.859},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.7830000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.372},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.94499999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644816,"kb_used_data":347016,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93727024,"statfs":{"total":96636764160,"available":95976472576,"internally_reserved":0,"allocated":355344384,"data_stored":355177199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.8740000000000001},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":10.762}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":9.7170000000000005},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":10.701000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":2.9039999999999999},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":10.736000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":355815424,"data_stored":355815424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226525184,"data_stored":226525184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":355602432,"data_stored":355602432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":353869824,"data_stored":353869824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:20.249 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 183 20 87910 87890 3.75122 2.25 0.0590672 0.0416436 2024-09-18T11:47:21.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 184 20 88228 88208 3.74433 2.48438 0.0233987 0.0417164 2024-09-18T11:47:22.252 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 185 20 88533 88513 3.73692 2.38281 0.114511 0.0418002 2024-09-18T11:47:23.019 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:23.252 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 186 20 88829 88809 3.72926 2.3125 0.0520695 0.0418818 2024-09-18T11:47:23.540 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:23.540 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:23.561 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-18T11:47:21.894199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":721741172,"num_objects":88108,"num_object_clones":0,"num_object_copies":352432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88135,"num_write_kb":704962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2368,"num_bytes_recovered":19161088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88162,"ondisk_log_size":88162,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2495292,"kb_used_data":1283676,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374992068,"statfs":{"total":386547056640,"available":383991877632,"internally_reserved":0,"allocated":1314484224,"data_stored":1313815484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,5,12,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36364288,"num_objects":4439,"num_object_clones":0,"num_object_copies":17756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4439,"num_write_kb":35512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007236"},"pg_stats":[{"pgid":"1.f","version":"33'5647","reported_seq":5690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:19.961210+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:19.961210+0000","last_peered":"2024-09-18T11:47:19.961210+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:19.961210+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46260224,"num_objects":5647,"num_object_clones":0,"num_object_copies":22588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5647,"num_write_kb":45176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5561","reported_seq":6123,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.597782+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:21.597782+0000","last_peered":"2024-09-18T11:47:21.597782+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:21.597782+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5561,"log_dups_size":0,"ondisk_log_size":5561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45520118,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3703}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5485","reported_seq":5569,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.427075+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:21.427075+0000","last_peered":"2024-09-18T11:47:21.427075+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:21.427075+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907131,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5484,"num_write_kb":43862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3428}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5480","reported_seq":5544,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.494590+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:21.494590+0000","last_peered":"2024-09-18T11:47:21.494590+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:21.494590+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44856566,"num_objects":5476,"num_object_clones":0,"num_object_copies":21904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5478,"num_write_kb":43813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5483","reported_seq":5540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.067782+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:20.067782+0000","last_peered":"2024-09-18T11:47:20.067782+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:20.067782+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44889334,"num_objects":5480,"num_object_clones":0,"num_object_copies":21920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":43845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5483","reported_seq":5528,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:19.999154+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:19.999154+0000","last_peered":"2024-09-18T11:47:19.999154+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:19.999154+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44916736,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":43864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5587","reported_seq":5654,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.528727+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:21.528727+0000","last_peered":"2024-09-18T11:47:21.528727+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:21.528727+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45716726,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5585,"num_write_kb":44660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3447}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5455","reported_seq":5520,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.565050+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:21.565050+0000","last_peered":"2024-09-18T11:47:21.565050+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:21.565050+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44643574,"num_objects":5450,"num_object_clones":0,"num_object_copies":21800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5453,"num_write_kb":43612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5596","reported_seq":5656,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.068517+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:20.068517+0000","last_peered":"2024-09-18T11:47:20.068517+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:20.068517+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45771244,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5593,"num_write_kb":44721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5364","reported_seq":5450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.367728+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:21.367728+0000","last_peered":"2024-09-18T11:47:21.367728+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:21.367728+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5364,"log_dups_size":0,"ondisk_log_size":5364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43922678,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":42894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3387}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5480","reported_seq":5539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.051854+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:20.051854+0000","last_peered":"2024-09-18T11:47:20.051854+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:20.051854+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44846961,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5477,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5576","reported_seq":5615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:19.944150+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:19.944150+0000","last_peered":"2024-09-18T11:47:19.944150+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:19.944150+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45668987,"num_objects":5575,"num_object_clones":0,"num_object_copies":22300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5575,"num_write_kb":44599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3466}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5565","reported_seq":5643,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.439723+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:21.439723+0000","last_peered":"2024-09-18T11:47:21.439723+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:21.439723+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45578875,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5564,"num_write_kb":44511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3510}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5512","reported_seq":5563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.019242+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:20.019242+0000","last_peered":"2024-09-18T11:47:20.019242+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:20.019242+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45118710,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5510,"num_write_kb":44076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5506","reported_seq":9791,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:20.070788+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:20.070788+0000","last_peered":"2024-09-18T11:47:20.070788+0000","last_clean":"2024-09-18T11:47:20.070788+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:20.070788+0000","last_undegraded":"2024-09-18T11:47:20.070788+0000","last_fullsized":"2024-09-18T11:47:20.070788+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45077750,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5504,"num_write_kb":44029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5382","reported_seq":5446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.617482+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:21.617482+0000","last_peered":"2024-09-18T11:47:21.617482+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:21.617482+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44045558,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5380,"num_write_kb":43028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":721741172,"num_objects":88108,"num_object_clones":0,"num_object_copies":352432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88135,"num_write_kb":704962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2368,"num_bytes_recovered":19161088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1316167680,"data_stored":1316167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88162,"ondisk_log_size":88162,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649016,"kb_used_data":351152,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93722824,"statfs":{"total":96636764160,"available":95972171776,"internally_reserved":0,"allocated":359579648,"data_stored":359412463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":3.0099999999999998},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.0409999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":5.9379999999999997},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.1180000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.54400000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":657908,"kb_used_data":352236,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93713932,"statfs":{"total":96636764160,"available":95963066368,"internally_reserved":0,"allocated":360689664,"data_stored":360522479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":0.51000000000000001},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":0.53400000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.55300000000000005},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.45400000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533424,"kb_used_data":227304,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93838416,"statfs":{"total":96636764160,"available":96090537984,"internally_reserved":0,"allocated":232759296,"data_stored":232592111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,7,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":5.1619999999999999},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":16.056999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":9.891},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.1970000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":5.4249999999999998},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":5.8470000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654944,"kb_used_data":352984,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93716896,"statfs":{"total":96636764160,"available":95966101504,"internally_reserved":0,"allocated":361455616,"data_stored":361288431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":14.872999999999999},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":11.914},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.4240000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":4.3869999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361848832,"data_stored":361848832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":232783872,"data_stored":232783872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361648128,"data_stored":361648128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359886848,"data_stored":359886848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:23.563 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:24.083 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:24.084 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:24.107 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-18T11:47:23.894836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":721741172,"num_objects":88108,"num_object_clones":0,"num_object_copies":352432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88135,"num_write_kb":704962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2368,"num_bytes_recovered":19161088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88162,"ondisk_log_size":88162,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2495292,"kb_used_data":1283676,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374992068,"statfs":{"total":386547056640,"available":383991877632,"internally_reserved":0,"allocated":1314484224,"data_stored":1313815484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,5,12,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23953408,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":23392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007019"},"pg_stats":[{"pgid":"1.f","version":"33'5647","reported_seq":5690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:19.961210+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:19.961210+0000","last_peered":"2024-09-18T11:47:19.961210+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:19.961210+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46260224,"num_objects":5647,"num_object_clones":0,"num_object_copies":22588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5647,"num_write_kb":45176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5561","reported_seq":6123,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.597782+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:21.597782+0000","last_peered":"2024-09-18T11:47:21.597782+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:21.597782+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5561,"log_dups_size":0,"ondisk_log_size":5561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45520118,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3703}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5485","reported_seq":5569,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.427075+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:21.427075+0000","last_peered":"2024-09-18T11:47:21.427075+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:21.427075+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907131,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5484,"num_write_kb":43862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3428}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5480","reported_seq":5544,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.494590+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:21.494590+0000","last_peered":"2024-09-18T11:47:21.494590+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:21.494590+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44856566,"num_objects":5476,"num_object_clones":0,"num_object_copies":21904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5478,"num_write_kb":43813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5483","reported_seq":5540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.067782+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:20.067782+0000","last_peered":"2024-09-18T11:47:20.067782+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:20.067782+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44889334,"num_objects":5480,"num_object_clones":0,"num_object_copies":21920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":43845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5483","reported_seq":5528,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:19.999154+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:19.999154+0000","last_peered":"2024-09-18T11:47:19.999154+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:19.999154+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44916736,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":43864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5587","reported_seq":5654,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.528727+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:21.528727+0000","last_peered":"2024-09-18T11:47:21.528727+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:21.528727+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45716726,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5585,"num_write_kb":44660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3447}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5455","reported_seq":5520,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.565050+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:21.565050+0000","last_peered":"2024-09-18T11:47:21.565050+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:21.565050+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44643574,"num_objects":5450,"num_object_clones":0,"num_object_copies":21800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5453,"num_write_kb":43612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5596","reported_seq":5656,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.068517+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:20.068517+0000","last_peered":"2024-09-18T11:47:20.068517+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:20.068517+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45771244,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5593,"num_write_kb":44721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5364","reported_seq":5450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.367728+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:21.367728+0000","last_peered":"2024-09-18T11:47:21.367728+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:21.367728+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5364,"log_dups_size":0,"ondisk_log_size":5364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43922678,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":42894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3387}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5480","reported_seq":5539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.051854+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:20.051854+0000","last_peered":"2024-09-18T11:47:20.051854+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:20.051854+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44846961,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5477,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5576","reported_seq":5615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:19.944150+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:19.944150+0000","last_peered":"2024-09-18T11:47:19.944150+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:19.944150+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45668987,"num_objects":5575,"num_object_clones":0,"num_object_copies":22300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5575,"num_write_kb":44599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3466}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5565","reported_seq":5643,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.439723+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:21.439723+0000","last_peered":"2024-09-18T11:47:21.439723+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:21.439723+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45578875,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5564,"num_write_kb":44511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3510}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5512","reported_seq":5563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.019242+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:20.019242+0000","last_peered":"2024-09-18T11:47:20.019242+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:20.019242+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45118710,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5510,"num_write_kb":44076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5506","reported_seq":9791,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:20.070788+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:20.070788+0000","last_peered":"2024-09-18T11:47:20.070788+0000","last_clean":"2024-09-18T11:47:20.070788+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:20.070788+0000","last_undegraded":"2024-09-18T11:47:20.070788+0000","last_fullsized":"2024-09-18T11:47:20.070788+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45077750,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5504,"num_write_kb":44029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5382","reported_seq":5446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.617482+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:21.617482+0000","last_peered":"2024-09-18T11:47:21.617482+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:21.617482+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44045558,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5380,"num_write_kb":43028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":721741172,"num_objects":88108,"num_object_clones":0,"num_object_copies":352432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88135,"num_write_kb":704962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2368,"num_bytes_recovered":19161088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1316167680,"data_stored":1316167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88162,"ondisk_log_size":88162,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649016,"kb_used_data":351152,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93722824,"statfs":{"total":96636764160,"available":95972171776,"internally_reserved":0,"allocated":359579648,"data_stored":359412463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":3.0099999999999998},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.0409999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":5.9379999999999997},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.1180000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.54400000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":657908,"kb_used_data":352236,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93713932,"statfs":{"total":96636764160,"available":95963066368,"internally_reserved":0,"allocated":360689664,"data_stored":360522479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":0.51000000000000001},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":0.53400000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.55300000000000005},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.45400000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533424,"kb_used_data":227304,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93838416,"statfs":{"total":96636764160,"available":96090537984,"internally_reserved":0,"allocated":232759296,"data_stored":232592111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,7,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":5.1619999999999999},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":16.056999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":9.891},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.1970000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":5.4249999999999998},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":5.8470000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654944,"kb_used_data":352984,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93716896,"statfs":{"total":96636764160,"available":95966101504,"internally_reserved":0,"allocated":361455616,"data_stored":361288431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":14.872999999999999},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":11.914},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.4240000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":4.3869999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361848832,"data_stored":361848832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":232783872,"data_stored":232783872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361648128,"data_stored":361648128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359886848,"data_stored":359886848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:24.110 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:24.110 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5647", 'reported_seq': 5690, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:19.961210+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:47:19.961210+0000', 'last_peered': '2024-09-18T11:47:19.961210+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:47:19.961210+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5647, 'log_dups_size': 0, 'ondisk_log_size': 5647, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46260224, 'num_objects': 5647, 'num_object_clones': 0, 'num_object_copies': 22588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5647, 'num_write_kb': 45176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3517}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:24.110 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:24.110 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5561", 'reported_seq': 6123, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:21.597782+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:21.597782+0000', 'last_peered': '2024-09-18T11:47:21.597782+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:21.597782+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5561, 'log_dups_size': 0, 'ondisk_log_size': 5561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45520118, 'num_objects': 5557, 'num_object_clones': 0, 'num_object_copies': 22228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1854, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5559, 'num_write_kb': 44468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 250, 'num_bytes_recovered': 2031616, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3703}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:24.111 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:24.111 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5485", 'reported_seq': 5569, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:21.427075+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:21.427075+0000', 'last_peered': '2024-09-18T11:47:21.427075+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:21.427075+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5485, 'log_dups_size': 0, 'ondisk_log_size': 5485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44907131, 'num_objects': 5482, 'num_object_clones': 0, 'num_object_copies': 21928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5484, 'num_write_kb': 43862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3428}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:24.111 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:24.111 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5480", 'reported_seq': 5544, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:21.494590+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:21.494590+0000', 'last_peered': '2024-09-18T11:47:21.494590+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:21.494590+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5480, 'log_dups_size': 0, 'ondisk_log_size': 5480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44856566, 'num_objects': 5476, 'num_object_clones': 0, 'num_object_copies': 21904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5478, 'num_write_kb': 43813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3362}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:24.111 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:24.112 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5483", 'reported_seq': 5540, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:20.067782+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:47:20.067782+0000', 'last_peered': '2024-09-18T11:47:20.067782+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:47:20.067782+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5483, 'log_dups_size': 0, 'ondisk_log_size': 5483, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44889334, 'num_objects': 5480, 'num_object_clones': 0, 'num_object_copies': 21920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5481, 'num_write_kb': 43845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3398}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:24.112 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:24.112 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5483", 'reported_seq': 5528, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:19.999154+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:47:19.999154+0000', 'last_peered': '2024-09-18T11:47:19.999154+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:47:19.999154+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5483, 'log_dups_size': 0, 'ondisk_log_size': 5483, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44916736, 'num_objects': 5483, 'num_object_clones': 0, 'num_object_copies': 21932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5483, 'num_write_kb': 43864, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3329}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:24.112 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:24.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5587", 'reported_seq': 5654, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:21.528727+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:21.528727+0000', 'last_peered': '2024-09-18T11:47:21.528727+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:21.528727+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5587, 'log_dups_size': 0, 'ondisk_log_size': 5587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45716726, 'num_objects': 5581, 'num_object_clones': 0, 'num_object_copies': 22324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5585, 'num_write_kb': 44660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3447}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:24.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:24.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5455", 'reported_seq': 5520, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:21.565050+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:21.565050+0000', 'last_peered': '2024-09-18T11:47:21.565050+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:21.565050+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5455, 'log_dups_size': 0, 'ondisk_log_size': 5455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44643574, 'num_objects': 5450, 'num_object_clones': 0, 'num_object_copies': 21800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5453, 'num_write_kb': 43612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3416}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:24.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:24.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5596", 'reported_seq': 5656, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:20.068517+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:47:20.068517+0000', 'last_peered': '2024-09-18T11:47:20.068517+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:47:20.068517+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5596, 'log_dups_size': 0, 'ondisk_log_size': 5596, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45771244, 'num_objects': 5588, 'num_object_clones': 0, 'num_object_copies': 22352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5593, 'num_write_kb': 44721, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3453}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:24.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:24.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5364", 'reported_seq': 5450, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:21.367728+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:21.367728+0000', 'last_peered': '2024-09-18T11:47:21.367728+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:21.367728+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5364, 'log_dups_size': 0, 'ondisk_log_size': 5364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43922678, 'num_objects': 5362, 'num_object_clones': 0, 'num_object_copies': 21448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5362, 'num_write_kb': 42894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3387}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:24.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:24.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5480", 'reported_seq': 5539, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:20.051854+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:47:20.051854+0000', 'last_peered': '2024-09-18T11:47:20.051854+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:47:20.051854+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5480, 'log_dups_size': 0, 'ondisk_log_size': 5480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44846961, 'num_objects': 5475, 'num_object_clones': 0, 'num_object_copies': 21900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5477, 'num_write_kb': 43804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3412}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:24.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:24.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5576", 'reported_seq': 5615, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:19.944150+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:47:19.944150+0000', 'last_peered': '2024-09-18T11:47:19.944150+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:47:19.944150+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5576, 'log_dups_size': 0, 'ondisk_log_size': 5576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45668987, 'num_objects': 5575, 'num_object_clones': 0, 'num_object_copies': 22300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5575, 'num_write_kb': 44599, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3466}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:24.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:24.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5565", 'reported_seq': 5643, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:21.439723+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:21.439723+0000', 'last_peered': '2024-09-18T11:47:21.439723+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:21.439723+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5565, 'log_dups_size': 0, 'ondisk_log_size': 5565, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45578875, 'num_objects': 5564, 'num_object_clones': 0, 'num_object_copies': 22256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5564, 'num_write_kb': 44511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3510}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:24.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:24.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5512", 'reported_seq': 5563, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:20.019242+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:47:20.019242+0000', 'last_peered': '2024-09-18T11:47:20.019242+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:47:20.019242+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5512, 'log_dups_size': 0, 'ondisk_log_size': 5512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45118710, 'num_objects': 5508, 'num_object_clones': 0, 'num_object_copies': 22032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5510, 'num_write_kb': 44076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3350}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:24.116 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:24.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5382", 'reported_seq': 5446, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:21.617482+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:21.617482+0000', 'last_peered': '2024-09-18T11:47:21.617482+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:21.617482+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5382, 'log_dups_size': 0, 'ondisk_log_size': 5382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44045558, 'num_objects': 5377, 'num_object_clones': 0, 'num_object_copies': 21508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5380, 'num_write_kb': 43028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3364}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:24.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:24.252 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 187 20 89086 89066 3.72006 2.00781 0.149691 0.0419862 2024-09-18T11:47:24.755 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:24.755 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31086/352432 objects degraded (8.820%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":242,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88108,"data_bytes":721741172,"bytes_used":2555179008,"bytes_avail":383991877632,"bytes_total":386547056640,"degraded_objects":31086,"degraded_total":352432,"degraded_ratio":0.088204249330367274,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":66179,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"write_bytes_sec":1994950,"read_op_per_sec":0,"write_op_per_sec":243},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:24.779 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:24.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:25.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 188 20 89359 89339 3.71161 2.13281 0.0713016 0.0420875 2024-09-18T11:47:25.368 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:25.368 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:25.400 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-18T11:47:23.894836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":721741172,"num_objects":88108,"num_object_clones":0,"num_object_copies":352432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88135,"num_write_kb":704962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2368,"num_bytes_recovered":19161088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88162,"ondisk_log_size":88162,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2495292,"kb_used_data":1283676,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374992068,"statfs":{"total":386547056640,"available":383991877632,"internally_reserved":0,"allocated":1314484224,"data_stored":1313815484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,5,12,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23953408,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":23392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007019"},"pg_stats":[{"pgid":"1.f","version":"33'5647","reported_seq":5690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:19.961210+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:19.961210+0000","last_peered":"2024-09-18T11:47:19.961210+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:19.961210+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46260224,"num_objects":5647,"num_object_clones":0,"num_object_copies":22588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5647,"num_write_kb":45176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5561","reported_seq":6123,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.597782+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:21.597782+0000","last_peered":"2024-09-18T11:47:21.597782+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:21.597782+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5561,"log_dups_size":0,"ondisk_log_size":5561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45520118,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3703}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5485","reported_seq":5569,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.427075+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:21.427075+0000","last_peered":"2024-09-18T11:47:21.427075+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:21.427075+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907131,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5484,"num_write_kb":43862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3428}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5480","reported_seq":5544,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.494590+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:21.494590+0000","last_peered":"2024-09-18T11:47:21.494590+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:21.494590+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44856566,"num_objects":5476,"num_object_clones":0,"num_object_copies":21904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5478,"num_write_kb":43813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5483","reported_seq":5540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.067782+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:20.067782+0000","last_peered":"2024-09-18T11:47:20.067782+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:20.067782+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44889334,"num_objects":5480,"num_object_clones":0,"num_object_copies":21920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":43845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5483","reported_seq":5528,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:19.999154+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:19.999154+0000","last_peered":"2024-09-18T11:47:19.999154+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:19.999154+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44916736,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":43864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5587","reported_seq":5654,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.528727+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:21.528727+0000","last_peered":"2024-09-18T11:47:21.528727+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:21.528727+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45716726,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5585,"num_write_kb":44660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3447}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5455","reported_seq":5520,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.565050+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:21.565050+0000","last_peered":"2024-09-18T11:47:21.565050+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:21.565050+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44643574,"num_objects":5450,"num_object_clones":0,"num_object_copies":21800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5453,"num_write_kb":43612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5596","reported_seq":5656,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.068517+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:20.068517+0000","last_peered":"2024-09-18T11:47:20.068517+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:20.068517+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45771244,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5593,"num_write_kb":44721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5364","reported_seq":5450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.367728+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:21.367728+0000","last_peered":"2024-09-18T11:47:21.367728+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:21.367728+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5364,"log_dups_size":0,"ondisk_log_size":5364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43922678,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":42894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3387}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5480","reported_seq":5539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.051854+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:20.051854+0000","last_peered":"2024-09-18T11:47:20.051854+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:20.051854+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44846961,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5477,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5576","reported_seq":5615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:19.944150+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:19.944150+0000","last_peered":"2024-09-18T11:47:19.944150+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:19.944150+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45668987,"num_objects":5575,"num_object_clones":0,"num_object_copies":22300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5575,"num_write_kb":44599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3466}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5565","reported_seq":5643,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.439723+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:21.439723+0000","last_peered":"2024-09-18T11:47:21.439723+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:21.439723+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45578875,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5564,"num_write_kb":44511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3510}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5512","reported_seq":5563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:20.019242+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:20.019242+0000","last_peered":"2024-09-18T11:47:20.019242+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:20.019242+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45118710,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5510,"num_write_kb":44076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5506","reported_seq":9791,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:20.070788+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:20.070788+0000","last_peered":"2024-09-18T11:47:20.070788+0000","last_clean":"2024-09-18T11:47:20.070788+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:20.070788+0000","last_undegraded":"2024-09-18T11:47:20.070788+0000","last_fullsized":"2024-09-18T11:47:20.070788+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45077750,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5504,"num_write_kb":44029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5382","reported_seq":5446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:21.617482+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:21.617482+0000","last_peered":"2024-09-18T11:47:21.617482+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:21.617482+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44045558,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5380,"num_write_kb":43028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":721741172,"num_objects":88108,"num_object_clones":0,"num_object_copies":352432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88135,"num_write_kb":704962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2368,"num_bytes_recovered":19161088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1316167680,"data_stored":1316167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88162,"ondisk_log_size":88162,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649016,"kb_used_data":351152,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93722824,"statfs":{"total":96636764160,"available":95972171776,"internally_reserved":0,"allocated":359579648,"data_stored":359412463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":3.0099999999999998},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.0409999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":5.9379999999999997},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.1180000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.54400000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":657908,"kb_used_data":352236,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93713932,"statfs":{"total":96636764160,"available":95963066368,"internally_reserved":0,"allocated":360689664,"data_stored":360522479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":0.51000000000000001},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":0.53400000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.55300000000000005},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.45400000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533424,"kb_used_data":227304,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93838416,"statfs":{"total":96636764160,"available":96090537984,"internally_reserved":0,"allocated":232759296,"data_stored":232592111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,7,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":5.1619999999999999},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":16.056999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":9.891},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.1970000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":5.4249999999999998},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":5.8470000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654944,"kb_used_data":352984,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93716896,"statfs":{"total":96636764160,"available":95966101504,"internally_reserved":0,"allocated":361455616,"data_stored":361288431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":14.872999999999999},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":11.914},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.4240000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":4.3869999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361848832,"data_stored":361848832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":232783872,"data_stored":232783872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361648128,"data_stored":361648128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359886848,"data_stored":359886848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:26.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 189 20 89658 89638 3.70434 2.33594 0.0343709 0.0421706 2024-09-18T11:47:27.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 190 20 89958 89938 3.69717 2.34375 0.0653474 0.0422496 2024-09-18T11:47:28.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 191 20 90260 90240 3.69017 2.35938 0.100889 0.0423302 2024-09-18T11:47:28.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:28.953 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:28.954 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:29.000 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-18T11:47:27.896245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":733562228,"num_objects":89551,"num_object_clones":0,"num_object_copies":358204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89578,"num_write_kb":716506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2415,"num_bytes_recovered":19546112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89605,"ondisk_log_size":89605,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2535288,"kb_used_data":1307032,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":374952072,"statfs":{"total":386547056640,"available":383950921728,"internally_reserved":0,"allocated":1338400768,"data_stored":1337732028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,14,2,9,13,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30130176,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3678,"num_write_kb":29424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004873"},"pg_stats":[{"pgid":"1.f","version":"33'5731","reported_seq":5774,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:24.813831+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:24.813831+0000","last_peered":"2024-09-18T11:47:24.813831+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:24.813831+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5731,"log_dups_size":0,"ondisk_log_size":5731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46948352,"num_objects":5731,"num_object_clones":0,"num_object_copies":22924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":45848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5648","reported_seq":6304,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.571829+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:26.571829+0000","last_peered":"2024-09-18T11:47:26.571829+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:26.571829+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232822,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5646,"num_write_kb":45164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2416640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5553","reported_seq":5637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.393044+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:26.393044+0000","last_peered":"2024-09-18T11:47:26.393044+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:26.393044+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5553,"log_dups_size":0,"ondisk_log_size":5553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45464187,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5552,"num_write_kb":44406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5579","reported_seq":5643,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.575315+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:26.575315+0000","last_peered":"2024-09-18T11:47:26.575315+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:26.575315+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45667574,"num_objects":5575,"num_object_clones":0,"num_object_copies":22300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5577,"num_write_kb":44605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5565","reported_seq":5621,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.040733+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:25.040733+0000","last_peered":"2024-09-18T11:47:25.040733+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:25.040733+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45561078,"num_objects":5562,"num_object_clones":0,"num_object_copies":22248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5563,"num_write_kb":44501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5570","reported_seq":5615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.036278+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:25.036278+0000","last_peered":"2024-09-18T11:47:25.036278+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:25.036278+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45629440,"num_objects":5570,"num_object_clones":0,"num_object_copies":22280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5570,"num_write_kb":44560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5680","reported_seq":5747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.616663+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:26.616663+0000","last_peered":"2024-09-18T11:47:26.616663+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:26.616663+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46478582,"num_objects":5674,"num_object_clones":0,"num_object_copies":22696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":45404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5561","reported_seq":5626,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.561943+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:26.561943+0000","last_peered":"2024-09-18T11:47:26.561943+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:26.561943+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5561,"log_dups_size":0,"ondisk_log_size":5561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45511926,"num_objects":5556,"num_object_clones":0,"num_object_copies":22224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5697","reported_seq":5757,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.068171+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:25.068171+0000","last_peered":"2024-09-18T11:47:25.068171+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:25.068171+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5697,"log_dups_size":0,"ondisk_log_size":5697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46598636,"num_objects":5689,"num_object_clones":0,"num_object_copies":22756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5694,"num_write_kb":45529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3554}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5464","reported_seq":5550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.368227+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:26.368227+0000","last_peered":"2024-09-18T11:47:26.368227+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:26.368227+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5464,"log_dups_size":0,"ondisk_log_size":5464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44741878,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5462,"num_write_kb":43694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3487}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5588","reported_seq":5646,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:24.996249+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:24.996249+0000","last_peered":"2024-09-18T11:47:24.996249+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:24.996249+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5588,"log_dups_size":0,"ondisk_log_size":5588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45731697,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5585,"num_write_kb":44668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5666","reported_seq":5705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.049066+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:25.049066+0000","last_peered":"2024-09-18T11:47:25.049066+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:25.049066+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5666,"log_dups_size":0,"ondisk_log_size":5666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46406267,"num_objects":5665,"num_object_clones":0,"num_object_copies":22660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":45319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3556}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5632","reported_seq":5710,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.328617+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:26.328617+0000","last_peered":"2024-09-18T11:47:26.328617+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:26.328617+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5632,"log_dups_size":0,"ondisk_log_size":5632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46127739,"num_objects":5631,"num_object_clones":0,"num_object_copies":22524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5631,"num_write_kb":45047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5585","reported_seq":5636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.036441+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:25.036441+0000","last_peered":"2024-09-18T11:47:25.036441+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:25.036441+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5585,"log_dups_size":0,"ondisk_log_size":5585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45716726,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5583,"num_write_kb":44660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5598","reported_seq":9883,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:25.167974+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:25.167974+0000","last_peered":"2024-09-18T11:47:25.167974+0000","last_clean":"2024-09-18T11:47:25.167974+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:25.167974+0000","last_undegraded":"2024-09-18T11:47:25.167974+0000","last_fullsized":"2024-09-18T11:47:25.167974+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45831414,"num_objects":5595,"num_object_clones":0,"num_object_copies":22380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":44765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5488","reported_seq":5552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.580211+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:26.580211+0000","last_peered":"2024-09-18T11:47:26.580211+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:26.580211+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5488,"log_dups_size":0,"ondisk_log_size":5488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44913910,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5486,"num_write_kb":43876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3470}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":733562228,"num_objects":89551,"num_object_clones":0,"num_object_copies":358204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89578,"num_write_kb":716506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2415,"num_bytes_recovered":19546112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1339998208,"data_stored":1339998208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89605,"ondisk_log_size":89605,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":658952,"kb_used_data":356928,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93712888,"statfs":{"total":96636764160,"available":95961997312,"internally_reserved":0,"allocated":365494272,"data_stored":365327087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,1,3,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":21.539999999999999},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":21.562000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":21.501000000000001},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":21.132000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":21.192},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":21.213000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667820,"kb_used_data":357988,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93704020,"statfs":{"total":96636764160,"available":95952916480,"internally_reserved":0,"allocated":366579712,"data_stored":366412527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":6.4729999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.0779999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":16.919},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":7.3799999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":3.1080000000000001},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":3.2919999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543696,"kb_used_data":233416,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93828144,"statfs":{"total":96636764160,"available":96080019456,"internally_reserved":0,"allocated":239017984,"data_stored":238850799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":3.544},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":8.4670000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":9.8309999999999995},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.29899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":15.903},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":25.440000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664820,"kb_used_data":358700,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93707020,"statfs":{"total":96636764160,"available":95955988480,"internally_reserved":0,"allocated":367308800,"data_stored":367141615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,0,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":6.0869999999999997},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":7.1909999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.883}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":13.831}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367681536,"data_stored":367681536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238960640,"data_stored":238960640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367489024,"data_stored":367489024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":365867008,"data_stored":365867008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:29.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:29.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 192 20 90540 90520 3.68234 2.1875 0.0687631 0.0424217 2024-09-18T11:47:29.567 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:29.568 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:29.592 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-18T11:47:27.896245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":733562228,"num_objects":89551,"num_object_clones":0,"num_object_copies":358204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89578,"num_write_kb":716506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2415,"num_bytes_recovered":19546112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89605,"ondisk_log_size":89605,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2535288,"kb_used_data":1307032,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":374952072,"statfs":{"total":386547056640,"available":383950921728,"internally_reserved":0,"allocated":1338400768,"data_stored":1337732028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,14,2,9,13,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30130176,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3678,"num_write_kb":29424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004873"},"pg_stats":[{"pgid":"1.f","version":"33'5731","reported_seq":5774,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:24.813831+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:24.813831+0000","last_peered":"2024-09-18T11:47:24.813831+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:24.813831+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5731,"log_dups_size":0,"ondisk_log_size":5731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46948352,"num_objects":5731,"num_object_clones":0,"num_object_copies":22924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":45848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5648","reported_seq":6304,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.571829+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:26.571829+0000","last_peered":"2024-09-18T11:47:26.571829+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:26.571829+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232822,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5646,"num_write_kb":45164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2416640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5553","reported_seq":5637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.393044+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:26.393044+0000","last_peered":"2024-09-18T11:47:26.393044+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:26.393044+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5553,"log_dups_size":0,"ondisk_log_size":5553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45464187,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5552,"num_write_kb":44406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5579","reported_seq":5643,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.575315+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:26.575315+0000","last_peered":"2024-09-18T11:47:26.575315+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:26.575315+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45667574,"num_objects":5575,"num_object_clones":0,"num_object_copies":22300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5577,"num_write_kb":44605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5565","reported_seq":5621,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.040733+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:25.040733+0000","last_peered":"2024-09-18T11:47:25.040733+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:25.040733+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45561078,"num_objects":5562,"num_object_clones":0,"num_object_copies":22248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5563,"num_write_kb":44501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5570","reported_seq":5615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.036278+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:25.036278+0000","last_peered":"2024-09-18T11:47:25.036278+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:25.036278+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45629440,"num_objects":5570,"num_object_clones":0,"num_object_copies":22280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5570,"num_write_kb":44560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5680","reported_seq":5747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.616663+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:26.616663+0000","last_peered":"2024-09-18T11:47:26.616663+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:26.616663+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46478582,"num_objects":5674,"num_object_clones":0,"num_object_copies":22696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":45404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5561","reported_seq":5626,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.561943+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:26.561943+0000","last_peered":"2024-09-18T11:47:26.561943+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:26.561943+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5561,"log_dups_size":0,"ondisk_log_size":5561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45511926,"num_objects":5556,"num_object_clones":0,"num_object_copies":22224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5697","reported_seq":5757,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.068171+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:25.068171+0000","last_peered":"2024-09-18T11:47:25.068171+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:25.068171+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5697,"log_dups_size":0,"ondisk_log_size":5697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46598636,"num_objects":5689,"num_object_clones":0,"num_object_copies":22756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5694,"num_write_kb":45529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3554}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5464","reported_seq":5550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.368227+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:26.368227+0000","last_peered":"2024-09-18T11:47:26.368227+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:26.368227+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5464,"log_dups_size":0,"ondisk_log_size":5464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44741878,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5462,"num_write_kb":43694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3487}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5588","reported_seq":5646,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:24.996249+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:24.996249+0000","last_peered":"2024-09-18T11:47:24.996249+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:24.996249+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5588,"log_dups_size":0,"ondisk_log_size":5588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45731697,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5585,"num_write_kb":44668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5666","reported_seq":5705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.049066+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:25.049066+0000","last_peered":"2024-09-18T11:47:25.049066+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:25.049066+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5666,"log_dups_size":0,"ondisk_log_size":5666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46406267,"num_objects":5665,"num_object_clones":0,"num_object_copies":22660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":45319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3556}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5632","reported_seq":5710,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.328617+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:26.328617+0000","last_peered":"2024-09-18T11:47:26.328617+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:26.328617+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5632,"log_dups_size":0,"ondisk_log_size":5632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46127739,"num_objects":5631,"num_object_clones":0,"num_object_copies":22524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5631,"num_write_kb":45047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5585","reported_seq":5636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.036441+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:25.036441+0000","last_peered":"2024-09-18T11:47:25.036441+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:25.036441+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5585,"log_dups_size":0,"ondisk_log_size":5585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45716726,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5583,"num_write_kb":44660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5598","reported_seq":9883,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:25.167974+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:25.167974+0000","last_peered":"2024-09-18T11:47:25.167974+0000","last_clean":"2024-09-18T11:47:25.167974+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:25.167974+0000","last_undegraded":"2024-09-18T11:47:25.167974+0000","last_fullsized":"2024-09-18T11:47:25.167974+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45831414,"num_objects":5595,"num_object_clones":0,"num_object_copies":22380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":44765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5488","reported_seq":5552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.580211+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:26.580211+0000","last_peered":"2024-09-18T11:47:26.580211+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:26.580211+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5488,"log_dups_size":0,"ondisk_log_size":5488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44913910,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5486,"num_write_kb":43876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3470}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":733562228,"num_objects":89551,"num_object_clones":0,"num_object_copies":358204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89578,"num_write_kb":716506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2415,"num_bytes_recovered":19546112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1339998208,"data_stored":1339998208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89605,"ondisk_log_size":89605,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":658952,"kb_used_data":356928,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93712888,"statfs":{"total":96636764160,"available":95961997312,"internally_reserved":0,"allocated":365494272,"data_stored":365327087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,1,3,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":21.539999999999999},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":21.562000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":21.501000000000001},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":21.132000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":21.192},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":21.213000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667820,"kb_used_data":357988,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93704020,"statfs":{"total":96636764160,"available":95952916480,"internally_reserved":0,"allocated":366579712,"data_stored":366412527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":6.4729999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.0779999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":16.919},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":7.3799999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":3.1080000000000001},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":3.2919999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543696,"kb_used_data":233416,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93828144,"statfs":{"total":96636764160,"available":96080019456,"internally_reserved":0,"allocated":239017984,"data_stored":238850799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":3.544},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":8.4670000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":9.8309999999999995},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.29899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":15.903},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":25.440000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664820,"kb_used_data":358700,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93707020,"statfs":{"total":96636764160,"available":95955988480,"internally_reserved":0,"allocated":367308800,"data_stored":367141615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,0,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":6.0869999999999997},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":7.1909999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.883}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":13.831}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367681536,"data_stored":367681536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238960640,"data_stored":238960640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367489024,"data_stored":367489024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":365867008,"data_stored":365867008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:29.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:29.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5731", 'reported_seq': 5774, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:24.813831+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:47:24.813831+0000', 'last_peered': '2024-09-18T11:47:24.813831+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:47:24.813831+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5731, 'log_dups_size': 0, 'ondisk_log_size': 5731, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46948352, 'num_objects': 5731, 'num_object_clones': 0, 'num_object_copies': 22924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5731, 'num_write_kb': 45848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3601}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:29.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:29.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5648", 'reported_seq': 6304, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:26.571829+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:26.571829+0000', 'last_peered': '2024-09-18T11:47:26.571829+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:26.571829+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5648, 'log_dups_size': 0, 'ondisk_log_size': 5648, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46232822, 'num_objects': 5644, 'num_object_clones': 0, 'num_object_copies': 22576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1807, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5646, 'num_write_kb': 45164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 297, 'num_bytes_recovered': 2416640, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:29.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:29.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5553", 'reported_seq': 5637, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:26.393044+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:26.393044+0000', 'last_peered': '2024-09-18T11:47:26.393044+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:26.393044+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5553, 'log_dups_size': 0, 'ondisk_log_size': 5553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45464187, 'num_objects': 5550, 'num_object_clones': 0, 'num_object_copies': 22200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5552, 'num_write_kb': 44406, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3496}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:29.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:29.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5579", 'reported_seq': 5643, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:26.575315+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:26.575315+0000', 'last_peered': '2024-09-18T11:47:26.575315+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:26.575315+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5579, 'log_dups_size': 0, 'ondisk_log_size': 5579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45667574, 'num_objects': 5575, 'num_object_clones': 0, 'num_object_copies': 22300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5577, 'num_write_kb': 44605, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3461}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:29.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:29.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5565", 'reported_seq': 5621, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:25.040733+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:47:25.040733+0000', 'last_peered': '2024-09-18T11:47:25.040733+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:47:25.040733+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5565, 'log_dups_size': 0, 'ondisk_log_size': 5565, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45561078, 'num_objects': 5562, 'num_object_clones': 0, 'num_object_copies': 22248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5563, 'num_write_kb': 44501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3480}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:29.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5570", 'reported_seq': 5615, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:25.036278+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:47:25.036278+0000', 'last_peered': '2024-09-18T11:47:25.036278+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:47:25.036278+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5570, 'log_dups_size': 0, 'ondisk_log_size': 5570, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45629440, 'num_objects': 5570, 'num_object_clones': 0, 'num_object_copies': 22280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5570, 'num_write_kb': 44560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3416}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5680", 'reported_seq': 5747, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:26.616663+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:26.616663+0000', 'last_peered': '2024-09-18T11:47:26.616663+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:26.616663+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5680, 'log_dups_size': 0, 'ondisk_log_size': 5680, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46478582, 'num_objects': 5674, 'num_object_clones': 0, 'num_object_copies': 22696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5678, 'num_write_kb': 45404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3540}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5561", 'reported_seq': 5626, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:26.561943+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:26.561943+0000', 'last_peered': '2024-09-18T11:47:26.561943+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:26.561943+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5561, 'log_dups_size': 0, 'ondisk_log_size': 5561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45511926, 'num_objects': 5556, 'num_object_clones': 0, 'num_object_copies': 22224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5559, 'num_write_kb': 44460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5697", 'reported_seq': 5757, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:25.068171+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:47:25.068171+0000', 'last_peered': '2024-09-18T11:47:25.068171+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:47:25.068171+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5697, 'log_dups_size': 0, 'ondisk_log_size': 5697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46598636, 'num_objects': 5689, 'num_object_clones': 0, 'num_object_copies': 22756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5694, 'num_write_kb': 45529, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3554}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5464", 'reported_seq': 5550, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:26.368227+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:26.368227+0000', 'last_peered': '2024-09-18T11:47:26.368227+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:26.368227+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5464, 'log_dups_size': 0, 'ondisk_log_size': 5464, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44741878, 'num_objects': 5462, 'num_object_clones': 0, 'num_object_copies': 21848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5462, 'num_write_kb': 43694, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3487}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:29.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5588", 'reported_seq': 5646, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:24.996249+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:47:24.996249+0000', 'last_peered': '2024-09-18T11:47:24.996249+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:47:24.996249+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5588, 'log_dups_size': 0, 'ondisk_log_size': 5588, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45731697, 'num_objects': 5583, 'num_object_clones': 0, 'num_object_copies': 22332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5585, 'num_write_kb': 44668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3520}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:29.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:29.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5666", 'reported_seq': 5705, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:25.049066+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:47:25.049066+0000', 'last_peered': '2024-09-18T11:47:25.049066+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:47:25.049066+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5666, 'log_dups_size': 0, 'ondisk_log_size': 5666, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46406267, 'num_objects': 5665, 'num_object_clones': 0, 'num_object_copies': 22660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5665, 'num_write_kb': 45319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3556}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:29.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:29.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5632", 'reported_seq': 5710, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:26.328617+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:26.328617+0000', 'last_peered': '2024-09-18T11:47:26.328617+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:26.328617+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5632, 'log_dups_size': 0, 'ondisk_log_size': 5632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46127739, 'num_objects': 5631, 'num_object_clones': 0, 'num_object_copies': 22524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5631, 'num_write_kb': 45047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3577}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:29.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:29.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5585", 'reported_seq': 5636, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:25.036441+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:47:25.036441+0000', 'last_peered': '2024-09-18T11:47:25.036441+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:47:25.036441+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5585, 'log_dups_size': 0, 'ondisk_log_size': 5585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45716726, 'num_objects': 5581, 'num_object_clones': 0, 'num_object_copies': 22324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5583, 'num_write_kb': 44660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:29.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:29.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5488", 'reported_seq': 5552, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:26.580211+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:26.580211+0000', 'last_peered': '2024-09-18T11:47:26.580211+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:26.580211+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5488, 'log_dups_size': 0, 'ondisk_log_size': 5488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44913910, 'num_objects': 5483, 'num_object_clones': 0, 'num_object_copies': 21932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5486, 'num_write_kb': 43876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3470}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:29.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:30.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 193 20 90805 90785 3.67399 2.07031 0.0654431 0.042518 2024-09-18T11:47:30.297 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:30.297 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31039/358204 objects degraded (8.665%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":248,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89551,"data_bytes":733562228,"bytes_used":2596134912,"bytes_avail":383950921728,"bytes_total":386547056640,"degraded_objects":31039,"degraded_total":358204,"degraded_ratio":0.086651740349074829,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":66193,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"write_bytes_sec":1981696,"read_op_per_sec":0,"write_op_per_sec":241},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:30.327 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:30.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:30.899 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:30.900 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:30.940 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-18T11:47:29.896841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":733562228,"num_objects":89551,"num_object_clones":0,"num_object_copies":358204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89578,"num_write_kb":716506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2415,"num_bytes_recovered":19546112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89605,"ondisk_log_size":89605,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2535288,"kb_used_data":1307032,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":374952072,"statfs":{"total":386547056640,"available":383950921728,"internally_reserved":0,"allocated":1338400768,"data_stored":1337732028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,14,2,9,13,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23789568,"num_objects":2904,"num_object_clones":0,"num_object_copies":11616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2904,"num_write_kb":23232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004648"},"pg_stats":[{"pgid":"1.f","version":"33'5731","reported_seq":5774,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:24.813831+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:24.813831+0000","last_peered":"2024-09-18T11:47:24.813831+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:24.813831+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5731,"log_dups_size":0,"ondisk_log_size":5731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46948352,"num_objects":5731,"num_object_clones":0,"num_object_copies":22924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":45848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5648","reported_seq":6304,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.571829+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:26.571829+0000","last_peered":"2024-09-18T11:47:26.571829+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:26.571829+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232822,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5646,"num_write_kb":45164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2416640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5553","reported_seq":5637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.393044+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:26.393044+0000","last_peered":"2024-09-18T11:47:26.393044+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:26.393044+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5553,"log_dups_size":0,"ondisk_log_size":5553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45464187,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5552,"num_write_kb":44406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5579","reported_seq":5643,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.575315+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:26.575315+0000","last_peered":"2024-09-18T11:47:26.575315+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:26.575315+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45667574,"num_objects":5575,"num_object_clones":0,"num_object_copies":22300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5577,"num_write_kb":44605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5565","reported_seq":5621,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.040733+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:25.040733+0000","last_peered":"2024-09-18T11:47:25.040733+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:25.040733+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45561078,"num_objects":5562,"num_object_clones":0,"num_object_copies":22248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5563,"num_write_kb":44501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5570","reported_seq":5615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.036278+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:25.036278+0000","last_peered":"2024-09-18T11:47:25.036278+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:25.036278+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45629440,"num_objects":5570,"num_object_clones":0,"num_object_copies":22280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5570,"num_write_kb":44560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5680","reported_seq":5747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.616663+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:26.616663+0000","last_peered":"2024-09-18T11:47:26.616663+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:26.616663+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46478582,"num_objects":5674,"num_object_clones":0,"num_object_copies":22696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":45404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5561","reported_seq":5626,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.561943+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:26.561943+0000","last_peered":"2024-09-18T11:47:26.561943+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:26.561943+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5561,"log_dups_size":0,"ondisk_log_size":5561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45511926,"num_objects":5556,"num_object_clones":0,"num_object_copies":22224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5697","reported_seq":5757,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.068171+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:25.068171+0000","last_peered":"2024-09-18T11:47:25.068171+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:25.068171+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5697,"log_dups_size":0,"ondisk_log_size":5697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46598636,"num_objects":5689,"num_object_clones":0,"num_object_copies":22756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5694,"num_write_kb":45529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3554}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5464","reported_seq":5550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.368227+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:26.368227+0000","last_peered":"2024-09-18T11:47:26.368227+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:26.368227+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5464,"log_dups_size":0,"ondisk_log_size":5464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44741878,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5462,"num_write_kb":43694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3487}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5588","reported_seq":5646,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:24.996249+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:24.996249+0000","last_peered":"2024-09-18T11:47:24.996249+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:24.996249+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5588,"log_dups_size":0,"ondisk_log_size":5588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45731697,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5585,"num_write_kb":44668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5666","reported_seq":5705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.049066+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:25.049066+0000","last_peered":"2024-09-18T11:47:25.049066+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:25.049066+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5666,"log_dups_size":0,"ondisk_log_size":5666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46406267,"num_objects":5665,"num_object_clones":0,"num_object_copies":22660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":45319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3556}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5632","reported_seq":5710,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.328617+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:26.328617+0000","last_peered":"2024-09-18T11:47:26.328617+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:26.328617+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5632,"log_dups_size":0,"ondisk_log_size":5632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46127739,"num_objects":5631,"num_object_clones":0,"num_object_copies":22524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5631,"num_write_kb":45047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5585","reported_seq":5636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:25.036441+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:25.036441+0000","last_peered":"2024-09-18T11:47:25.036441+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:25.036441+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5585,"log_dups_size":0,"ondisk_log_size":5585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45716726,"num_objects":5581,"num_object_clones":0,"num_object_copies":22324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5583,"num_write_kb":44660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5598","reported_seq":9883,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:25.167974+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:25.167974+0000","last_peered":"2024-09-18T11:47:25.167974+0000","last_clean":"2024-09-18T11:47:25.167974+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:25.167974+0000","last_undegraded":"2024-09-18T11:47:25.167974+0000","last_fullsized":"2024-09-18T11:47:25.167974+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45831414,"num_objects":5595,"num_object_clones":0,"num_object_copies":22380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":44765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5488","reported_seq":5552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:26.580211+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:26.580211+0000","last_peered":"2024-09-18T11:47:26.580211+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:26.580211+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5488,"log_dups_size":0,"ondisk_log_size":5488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44913910,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5486,"num_write_kb":43876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3470}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":733562228,"num_objects":89551,"num_object_clones":0,"num_object_copies":358204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89578,"num_write_kb":716506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2415,"num_bytes_recovered":19546112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1339998208,"data_stored":1339998208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89605,"ondisk_log_size":89605,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":658952,"kb_used_data":356928,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93712888,"statfs":{"total":96636764160,"available":95961997312,"internally_reserved":0,"allocated":365494272,"data_stored":365327087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,1,3,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":21.539999999999999},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":21.562000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":21.501000000000001},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":21.132000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":21.192},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":21.213000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667820,"kb_used_data":357988,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93704020,"statfs":{"total":96636764160,"available":95952916480,"internally_reserved":0,"allocated":366579712,"data_stored":366412527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":6.4729999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.0779999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":16.919},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":7.3799999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":3.1080000000000001},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":3.2919999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543696,"kb_used_data":233416,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93828144,"statfs":{"total":96636764160,"available":96080019456,"internally_reserved":0,"allocated":239017984,"data_stored":238850799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":3.544},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":8.4670000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":9.8309999999999995},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.29899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":15.903},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":25.440000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664820,"kb_used_data":358700,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93707020,"statfs":{"total":96636764160,"available":95955988480,"internally_reserved":0,"allocated":367308800,"data_stored":367141615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,0,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":6.0869999999999997},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":7.1909999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.883}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":13.831}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367681536,"data_stored":367681536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238960640,"data_stored":238960640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367489024,"data_stored":367489024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":365867008,"data_stored":365867008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:31.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 194 20 91078 91058 3.66605 2.13281 0.0538332 0.04261 2024-09-18T11:47:32.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 195 20 91375 91355 3.65915 2.32031 0.116507 0.0426884 2024-09-18T11:47:33.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 196 20 91670 91650 3.65224 2.30469 0.0772881 0.0427662 2024-09-18T11:47:33.942 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:34.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 197 20 91965 91945 3.6454 2.30469 0.111038 0.0428455 2024-09-18T11:47:34.490 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:34.491 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:34.520 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-18T11:47:33.900765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":745424244,"num_objects":90999,"num_object_clones":0,"num_object_copies":363996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91026,"num_write_kb":728090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":19939328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91053,"ondisk_log_size":91053,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2558592,"kb_used_data":1330336,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":374928768,"statfs":{"total":386547056640,"available":383927058432,"internally_reserved":0,"allocated":1362264064,"data_stored":1361595324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,6,7,9,8,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23683072,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2891,"num_write_kb":23128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006566"},"pg_stats":[{"pgid":"1.f","version":"33'5825","reported_seq":5868,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.002487+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:30.002487+0000","last_peered":"2024-09-18T11:47:30.002487+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:30.002487+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5825,"log_dups_size":0,"ondisk_log_size":5825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47718400,"num_objects":5825,"num_object_clones":0,"num_object_copies":23300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5825,"num_write_kb":46600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5746","reported_seq":6498,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.601363+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:31.601363+0000","last_peered":"2024-09-18T11:47:31.601363+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:31.601363+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47035638,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":45948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":2809856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5640","reported_seq":5724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.247324+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:31.247324+0000","last_peered":"2024-09-18T11:47:31.247324+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:31.247324+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5640,"log_dups_size":0,"ondisk_log_size":5640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46176891,"num_objects":5637,"num_object_clones":0,"num_object_copies":22548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":45102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3583}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5660","reported_seq":5724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.578778+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:31.578778+0000","last_peered":"2024-09-18T11:47:31.578778+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:31.578778+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331126,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5654","reported_seq":5711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.065219+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:30.065219+0000","last_peered":"2024-09-18T11:47:30.065219+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:30.065219+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46290166,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":45213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5651","reported_seq":5696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.010974+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:30.010974+0000","last_peered":"2024-09-18T11:47:30.010974+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:30.010974+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46292992,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5651,"num_write_kb":45208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5772","reported_seq":5839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.632967+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:31.632967+0000","last_peered":"2024-09-18T11:47:31.632967+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:31.632967+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232246,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3632}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5675","reported_seq":5738,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.555274+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:31.555274+0000","last_peered":"2024-09-18T11:47:31.555274+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:31.555274+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46445814,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5673,"num_write_kb":45372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5783","reported_seq":5843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.005882+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:30.005882+0000","last_peered":"2024-09-18T11:47:30.005882+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:30.005882+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5783,"log_dups_size":0,"ondisk_log_size":5783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47303148,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5780,"num_write_kb":46217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3640}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5556","reported_seq":5642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.406187+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:31.406187+0000","last_peered":"2024-09-18T11:47:31.406187+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:31.406187+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45495542,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":44430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3579}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5672","reported_seq":5730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.100556+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:30.100556+0000","last_peered":"2024-09-18T11:47:30.100556+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:30.100556+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46419825,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":45340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5762","reported_seq":5801,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.004757+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:30.004757+0000","last_peered":"2024-09-18T11:47:30.004757+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:30.004757+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47192699,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5761,"num_write_kb":46087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3652}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5714","reported_seq":5791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.408474+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:31.408474+0000","last_peered":"2024-09-18T11:47:31.408474+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:31.408474+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5714,"log_dups_size":0,"ondisk_log_size":5714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46799483,"num_objects":5713,"num_object_clones":0,"num_object_copies":22852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5713,"num_write_kb":45703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3659}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5688","reported_seq":5739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.099962+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:30.099962+0000","last_peered":"2024-09-18T11:47:30.099962+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:30.099962+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5688,"log_dups_size":0,"ondisk_log_size":5688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46560502,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5686,"num_write_kb":45484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5670","reported_seq":9954,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:30.206038+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:30.206038+0000","last_peered":"2024-09-18T11:47:30.206038+0000","last_clean":"2024-09-18T11:47:30.206038+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:30.206038+0000","last_undegraded":"2024-09-18T11:47:30.206038+0000","last_fullsized":"2024-09-18T11:47:30.206038+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5670,"log_dups_size":0,"ondisk_log_size":5670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421238,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5668,"num_write_kb":45341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5585","reported_seq":5649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.631709+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:31.631709+0000","last_peered":"2024-09-18T11:47:31.631709+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:31.631709+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5585,"log_dups_size":0,"ondisk_log_size":5585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708534,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5583,"num_write_kb":44652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":745424244,"num_objects":90999,"num_object_clones":0,"num_object_copies":363996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91026,"num_write_kb":728090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":19939328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1363775488,"data_stored":1363775488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91053,"ondisk_log_size":91053,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668880,"kb_used_data":362696,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93702960,"statfs":{"total":96636764160,"available":95951831040,"internally_reserved":0,"allocated":371400704,"data_stored":371233519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,3,3,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":10.956},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":10.978}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":10.920999999999999},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.7779999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.2060000000000004},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":7.3890000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661148,"kb_used_data":363796,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93710692,"statfs":{"total":96636764160,"available":95959748608,"internally_reserved":0,"allocated":372527104,"data_stored":372359919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":1.5449999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.43099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":2.5179999999999998},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":0.59599999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.45900000000000002},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.829}]}]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553868,"kb_used_data":239428,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93817972,"statfs":{"total":96636764160,"available":96069603328,"internally_reserved":0,"allocated":245174272,"data_stored":245007087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":3.4390000000000001},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":3.4980000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":10.619999999999999},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.4670000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.34399999999999997},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":2.8780000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674696,"kb_used_data":364416,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93697144,"statfs":{"total":96636764160,"available":95945875456,"internally_reserved":0,"allocated":373161984,"data_stored":372994799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,2,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.5350000000000001},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":12.351000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":4.9109999999999996},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.4990000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":2.5640000000000001},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":6.3390000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373526528,"data_stored":373526528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245100544,"data_stored":245100544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":373317632,"data_stored":373317632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":371830784,"data_stored":371830784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:34.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:35.070 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:35.093 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-18T11:47:33.900765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":745424244,"num_objects":90999,"num_object_clones":0,"num_object_copies":363996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91026,"num_write_kb":728090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":19939328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91053,"ondisk_log_size":91053,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2558592,"kb_used_data":1330336,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":374928768,"statfs":{"total":386547056640,"available":383927058432,"internally_reserved":0,"allocated":1362264064,"data_stored":1361595324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,6,7,9,8,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23683072,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2891,"num_write_kb":23128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006566"},"pg_stats":[{"pgid":"1.f","version":"33'5825","reported_seq":5868,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.002487+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:30.002487+0000","last_peered":"2024-09-18T11:47:30.002487+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:30.002487+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5825,"log_dups_size":0,"ondisk_log_size":5825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47718400,"num_objects":5825,"num_object_clones":0,"num_object_copies":23300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5825,"num_write_kb":46600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5746","reported_seq":6498,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.601363+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:31.601363+0000","last_peered":"2024-09-18T11:47:31.601363+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:31.601363+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47035638,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":45948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":2809856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5640","reported_seq":5724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.247324+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:31.247324+0000","last_peered":"2024-09-18T11:47:31.247324+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:31.247324+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5640,"log_dups_size":0,"ondisk_log_size":5640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46176891,"num_objects":5637,"num_object_clones":0,"num_object_copies":22548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":45102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3583}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5660","reported_seq":5724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.578778+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:31.578778+0000","last_peered":"2024-09-18T11:47:31.578778+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:31.578778+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331126,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5654","reported_seq":5711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.065219+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:30.065219+0000","last_peered":"2024-09-18T11:47:30.065219+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:30.065219+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46290166,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":45213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5651","reported_seq":5696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.010974+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:30.010974+0000","last_peered":"2024-09-18T11:47:30.010974+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:30.010974+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46292992,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5651,"num_write_kb":45208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5772","reported_seq":5839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.632967+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:31.632967+0000","last_peered":"2024-09-18T11:47:31.632967+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:31.632967+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232246,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3632}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5675","reported_seq":5738,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.555274+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:31.555274+0000","last_peered":"2024-09-18T11:47:31.555274+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:31.555274+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46445814,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5673,"num_write_kb":45372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5783","reported_seq":5843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.005882+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:30.005882+0000","last_peered":"2024-09-18T11:47:30.005882+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:30.005882+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5783,"log_dups_size":0,"ondisk_log_size":5783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47303148,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5780,"num_write_kb":46217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3640}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5556","reported_seq":5642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.406187+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:31.406187+0000","last_peered":"2024-09-18T11:47:31.406187+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:31.406187+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45495542,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":44430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3579}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5672","reported_seq":5730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.100556+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:30.100556+0000","last_peered":"2024-09-18T11:47:30.100556+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:30.100556+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46419825,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":45340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5762","reported_seq":5801,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.004757+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:30.004757+0000","last_peered":"2024-09-18T11:47:30.004757+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:30.004757+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47192699,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5761,"num_write_kb":46087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3652}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5714","reported_seq":5791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.408474+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:31.408474+0000","last_peered":"2024-09-18T11:47:31.408474+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:31.408474+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5714,"log_dups_size":0,"ondisk_log_size":5714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46799483,"num_objects":5713,"num_object_clones":0,"num_object_copies":22852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5713,"num_write_kb":45703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3659}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5688","reported_seq":5739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:30.099962+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:30.099962+0000","last_peered":"2024-09-18T11:47:30.099962+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:30.099962+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5688,"log_dups_size":0,"ondisk_log_size":5688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46560502,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5686,"num_write_kb":45484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5670","reported_seq":9954,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:30.206038+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:30.206038+0000","last_peered":"2024-09-18T11:47:30.206038+0000","last_clean":"2024-09-18T11:47:30.206038+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:30.206038+0000","last_undegraded":"2024-09-18T11:47:30.206038+0000","last_fullsized":"2024-09-18T11:47:30.206038+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5670,"log_dups_size":0,"ondisk_log_size":5670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421238,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5668,"num_write_kb":45341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5585","reported_seq":5649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.631709+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:31.631709+0000","last_peered":"2024-09-18T11:47:31.631709+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:31.631709+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5585,"log_dups_size":0,"ondisk_log_size":5585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708534,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5583,"num_write_kb":44652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":745424244,"num_objects":90999,"num_object_clones":0,"num_object_copies":363996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91026,"num_write_kb":728090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":19939328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1363775488,"data_stored":1363775488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91053,"ondisk_log_size":91053,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668880,"kb_used_data":362696,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93702960,"statfs":{"total":96636764160,"available":95951831040,"internally_reserved":0,"allocated":371400704,"data_stored":371233519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,3,3,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":10.956},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":10.978}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":10.920999999999999},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":4.7779999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.2060000000000004},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":7.3890000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661148,"kb_used_data":363796,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93710692,"statfs":{"total":96636764160,"available":95959748608,"internally_reserved":0,"allocated":372527104,"data_stored":372359919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":1.5449999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.43099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":2.5179999999999998},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":0.59599999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.45900000000000002},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.829}]}]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553868,"kb_used_data":239428,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93817972,"statfs":{"total":96636764160,"available":96069603328,"internally_reserved":0,"allocated":245174272,"data_stored":245007087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":3.4390000000000001},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":3.4980000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":10.619999999999999},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.4670000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":0.34399999999999997},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":2.8780000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674696,"kb_used_data":364416,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93697144,"statfs":{"total":96636764160,"available":95945875456,"internally_reserved":0,"allocated":373161984,"data_stored":372994799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,2,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.5350000000000001},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":12.351000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":4.9109999999999996},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.4990000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":2.5640000000000001},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":6.3390000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373526528,"data_stored":373526528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245100544,"data_stored":245100544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":373317632,"data_stored":373317632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":371830784,"data_stored":371830784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:35.094 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:35.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5825", 'reported_seq': 5868, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:30.002487+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:47:30.002487+0000', 'last_peered': '2024-09-18T11:47:30.002487+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:47:30.002487+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5825, 'log_dups_size': 0, 'ondisk_log_size': 5825, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47718400, 'num_objects': 5825, 'num_object_clones': 0, 'num_object_copies': 23300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5825, 'num_write_kb': 46600, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3695}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:35.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:35.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5746", 'reported_seq': 6498, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:31.601363+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:31.601363+0000', 'last_peered': '2024-09-18T11:47:31.601363+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:31.601363+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5746, 'log_dups_size': 0, 'ondisk_log_size': 5746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47035638, 'num_objects': 5742, 'num_object_clones': 0, 'num_object_copies': 22968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1759, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5744, 'num_write_kb': 45948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 345, 'num_bytes_recovered': 2809856, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3983}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:35.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:35.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5640", 'reported_seq': 5724, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:31.247324+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:31.247324+0000', 'last_peered': '2024-09-18T11:47:31.247324+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:31.247324+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5640, 'log_dups_size': 0, 'ondisk_log_size': 5640, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46176891, 'num_objects': 5637, 'num_object_clones': 0, 'num_object_copies': 22548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5639, 'num_write_kb': 45102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3583}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:35.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:35.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5660", 'reported_seq': 5724, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:31.578778+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:31.578778+0000', 'last_peered': '2024-09-18T11:47:31.578778+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:31.578778+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5660, 'log_dups_size': 0, 'ondisk_log_size': 5660, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46331126, 'num_objects': 5656, 'num_object_clones': 0, 'num_object_copies': 22624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5658, 'num_write_kb': 45253, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3542}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:35.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:35.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5654", 'reported_seq': 5711, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:30.065219+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:47:30.065219+0000', 'last_peered': '2024-09-18T11:47:30.065219+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:47:30.065219+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5654, 'log_dups_size': 0, 'ondisk_log_size': 5654, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46290166, 'num_objects': 5651, 'num_object_clones': 0, 'num_object_copies': 22604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5652, 'num_write_kb': 45213, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3569}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:35.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:35.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5651", 'reported_seq': 5696, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:30.010974+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:47:30.010974+0000', 'last_peered': '2024-09-18T11:47:30.010974+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:47:30.010974+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5651, 'log_dups_size': 0, 'ondisk_log_size': 5651, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46292992, 'num_objects': 5651, 'num_object_clones': 0, 'num_object_copies': 22604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5651, 'num_write_kb': 45208, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:35.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:35.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5772", 'reported_seq': 5839, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:31.632967+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:31.632967+0000', 'last_peered': '2024-09-18T11:47:31.632967+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:31.632967+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5772, 'log_dups_size': 0, 'ondisk_log_size': 5772, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47232246, 'num_objects': 5766, 'num_object_clones': 0, 'num_object_copies': 23064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5770, 'num_write_kb': 46140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3632}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:35.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:35.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5675", 'reported_seq': 5738, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:31.555274+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:31.555274+0000', 'last_peered': '2024-09-18T11:47:31.555274+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:31.555274+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5675, 'log_dups_size': 0, 'ondisk_log_size': 5675, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46445814, 'num_objects': 5670, 'num_object_clones': 0, 'num_object_copies': 22680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5670, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5673, 'num_write_kb': 45372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3636}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:35.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:35.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5783", 'reported_seq': 5843, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:30.005882+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:47:30.005882+0000', 'last_peered': '2024-09-18T11:47:30.005882+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:47:30.005882+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5783, 'log_dups_size': 0, 'ondisk_log_size': 5783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47303148, 'num_objects': 5775, 'num_object_clones': 0, 'num_object_copies': 23100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5780, 'num_write_kb': 46217, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3640}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:35.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:35.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5556", 'reported_seq': 5642, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:31.406187+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:31.406187+0000', 'last_peered': '2024-09-18T11:47:31.406187+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:31.406187+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5556, 'log_dups_size': 0, 'ondisk_log_size': 5556, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45495542, 'num_objects': 5554, 'num_object_clones': 0, 'num_object_copies': 22216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5554, 'num_write_kb': 44430, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3579}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:35.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:35.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5672", 'reported_seq': 5730, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:30.100556+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:47:30.100556+0000', 'last_peered': '2024-09-18T11:47:30.100556+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:47:30.100556+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5672, 'log_dups_size': 0, 'ondisk_log_size': 5672, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46419825, 'num_objects': 5667, 'num_object_clones': 0, 'num_object_copies': 22668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5669, 'num_write_kb': 45340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3604}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:35.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:35.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5762", 'reported_seq': 5801, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:30.004757+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:47:30.004757+0000', 'last_peered': '2024-09-18T11:47:30.004757+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:47:30.004757+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5762, 'log_dups_size': 0, 'ondisk_log_size': 5762, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47192699, 'num_objects': 5761, 'num_object_clones': 0, 'num_object_copies': 23044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5761, 'num_write_kb': 46087, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3652}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:35.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:35.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5714", 'reported_seq': 5791, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:31.408474+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:31.408474+0000', 'last_peered': '2024-09-18T11:47:31.408474+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:31.408474+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5714, 'log_dups_size': 0, 'ondisk_log_size': 5714, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46799483, 'num_objects': 5713, 'num_object_clones': 0, 'num_object_copies': 22852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5713, 'num_write_kb': 45703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3659}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:35.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:35.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5688", 'reported_seq': 5739, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:30.099962+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:47:30.099962+0000', 'last_peered': '2024-09-18T11:47:30.099962+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:47:30.099962+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5688, 'log_dups_size': 0, 'ondisk_log_size': 5688, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46560502, 'num_objects': 5684, 'num_object_clones': 0, 'num_object_copies': 22736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5686, 'num_write_kb': 45484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3526}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:35.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:35.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5585", 'reported_seq': 5649, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:31.631709+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:31.631709+0000', 'last_peered': '2024-09-18T11:47:31.631709+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:31.631709+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5585, 'log_dups_size': 0, 'ondisk_log_size': 5585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45708534, 'num_objects': 5580, 'num_object_clones': 0, 'num_object_copies': 22320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5583, 'num_write_kb': 44652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3567}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:35.098 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:35.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 198 20 92223 92203 3.63717 2.01562 0.0735307 0.0429462 2024-09-18T11:47:35.768 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:35.769 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30991/363996 objects degraded (8.514%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":253,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":90999,"data_bytes":745424244,"bytes_used":2619998208,"bytes_avail":383927058432,"bytes_total":386547056640,"degraded_objects":30991,"degraded_total":363996,"degraded_ratio":0.085141045505994578,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":64817,"recovering_keys_per_sec":0,"num_objects_recovered":95,"num_bytes_recovered":778240,"num_keys_recovered":0,"write_bytes_sec":1972509,"read_op_per_sec":0,"write_op_per_sec":240},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:35.798 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:35.798 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:36.253 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 199 20 92483 92463 3.6291 2.03125 0.0419194 0.0430413 2024-09-18T11:47:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:36.326 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:36.358 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-18T11:47:35.901541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":751289716,"num_objects":91715,"num_object_clones":0,"num_object_copies":366860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91742,"num_write_kb":733818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":19939328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91769,"ondisk_log_size":91769,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2561912,"kb_used_data":1341976,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374925448,"statfs":{"total":386547056640,"available":383923658752,"internally_reserved":0,"allocated":1374183424,"data_stored":1373514684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,12,5,9,17,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29548544,"num_objects":3607,"num_object_clones":0,"num_object_copies":14428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":28856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006704"},"pg_stats":[{"pgid":"1.f","version":"33'5920","reported_seq":5962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.110888+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:35.110888+0000","last_peered":"2024-09-18T11:47:35.110888+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:35.110888+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":47360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5746","reported_seq":6498,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.601363+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:31.601363+0000","last_peered":"2024-09-18T11:47:31.601363+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:31.601363+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47035638,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":45948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":2809856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5640","reported_seq":5724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.247324+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:31.247324+0000","last_peered":"2024-09-18T11:47:31.247324+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:31.247324+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5640,"log_dups_size":0,"ondisk_log_size":5640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46176891,"num_objects":5637,"num_object_clones":0,"num_object_copies":22548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":45102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3583}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5660","reported_seq":5724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.578778+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:31.578778+0000","last_peered":"2024-09-18T11:47:31.578778+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:31.578778+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331126,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5729","reported_seq":5786,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.131949+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:35.131949+0000","last_peered":"2024-09-18T11:47:35.131949+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:35.131949+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46904566,"num_objects":5726,"num_object_clones":0,"num_object_copies":22904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5727,"num_write_kb":45813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5747","reported_seq":5792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.276590+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:35.276590+0000","last_peered":"2024-09-18T11:47:35.276590+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:35.276590+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5747,"log_dups_size":0,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47079424,"num_objects":5747,"num_object_clones":0,"num_object_copies":22988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5747,"num_write_kb":45976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5772","reported_seq":5839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.632967+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:31.632967+0000","last_peered":"2024-09-18T11:47:31.632967+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:31.632967+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232246,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3632}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5675","reported_seq":5738,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.555274+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:31.555274+0000","last_peered":"2024-09-18T11:47:31.555274+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:31.555274+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46445814,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5673,"num_write_kb":45372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5869","reported_seq":5929,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.178590+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:35.178590+0000","last_peered":"2024-09-18T11:47:35.178590+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:35.178590+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48007660,"num_objects":5861,"num_object_clones":0,"num_object_copies":23444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5866,"num_write_kb":46905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5556","reported_seq":5642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.406187+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:31.406187+0000","last_peered":"2024-09-18T11:47:31.406187+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:31.406187+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45495542,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":44430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3579}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5766","reported_seq":5825,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.014715+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:35.014715+0000","last_peered":"2024-09-18T11:47:35.014715+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:35.014715+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47189873,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5763,"num_write_kb":46092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5859","reported_seq":5897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.259010+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:35.259010+0000","last_peered":"2024-09-18T11:47:35.259010+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:35.259010+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5859,"log_dups_size":0,"ondisk_log_size":5859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47987323,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":46863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5714","reported_seq":5791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.408474+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:31.408474+0000","last_peered":"2024-09-18T11:47:31.408474+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:31.408474+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5714,"log_dups_size":0,"ondisk_log_size":5714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46799483,"num_objects":5713,"num_object_clones":0,"num_object_copies":22852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5713,"num_write_kb":45703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3659}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5785","reported_seq":5836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.239745+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:35.239745+0000","last_peered":"2024-09-18T11:47:35.239745+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:35.239745+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5785,"log_dups_size":0,"ondisk_log_size":5785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47355126,"num_objects":5781,"num_object_clones":0,"num_object_copies":23124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5783,"num_write_kb":46260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5746","reported_seq":10031,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:35.176600+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:35.176600+0000","last_peered":"2024-09-18T11:47:35.176600+0000","last_clean":"2024-09-18T11:47:35.176600+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:35.176600+0000","last_undegraded":"2024-09-18T11:47:35.176600+0000","last_fullsized":"2024-09-18T11:47:35.176600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47043830,"num_objects":5743,"num_object_clones":0,"num_object_copies":22972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":45949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5585","reported_seq":5649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:31.631709+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:31.631709+0000","last_peered":"2024-09-18T11:47:31.631709+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:31.631709+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5585,"log_dups_size":0,"ondisk_log_size":5585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708534,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5583,"num_write_kb":44652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":751289716,"num_objects":91715,"num_object_clones":0,"num_object_copies":366860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91742,"num_write_kb":733818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":19939328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1375825920,"data_stored":1375825920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91769,"ondisk_log_size":91769,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678760,"kb_used_data":368416,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93693080,"statfs":{"total":96636764160,"available":95941713920,"internally_reserved":0,"allocated":377257984,"data_stored":377090799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":12.220000000000001},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":12.247999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":10.978999999999999},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.2649999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.2060000000000004},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":7.3890000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661148,"kb_used_data":363796,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93710692,"statfs":{"total":96636764160,"available":95959748608,"internally_reserved":0,"allocated":372527104,"data_stored":372359919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":1.5449999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.43099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":2.5179999999999998},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":0.59599999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.45900000000000002},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.829}]}]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547308,"kb_used_data":245348,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93824532,"statfs":{"total":96636764160,"available":96076320768,"internally_reserved":0,"allocated":251236352,"data_stored":251069167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,1,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":9.4190000000000005},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":3.3149999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":1.647},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":9.4719999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":3.2810000000000001},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":1.7809999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674696,"kb_used_data":364416,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93697144,"statfs":{"total":96636764160,"available":95945875456,"internally_reserved":0,"allocated":373161984,"data_stored":372994799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,2,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.5350000000000001},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":12.351000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":4.9109999999999996},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.4990000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":2.5640000000000001},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":6.3390000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373526528,"data_stored":373526528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251207680,"data_stored":251207680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":373317632,"data_stored":373317632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":377774080,"data_stored":377774080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:37.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:47:37.253346+0000 min lat: 0.00272854 max lat: 2.42146 avg lat: 0.0431192 2024-09-18T11:47:37.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:47:37.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 200 20 92778 92758 3.62248 2.30469 0.0569173 0.0431192 2024-09-18T11:47:38.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 201 20 93079 93059 3.61615 2.35156 0.0854975 0.0431948 2024-09-18T11:47:39.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 202 20 93378 93358 3.60981 2.33594 0.0533133 0.0432725 2024-09-18T11:47:39.360 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:39.898 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:39.899 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:39.925 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-18T11:47:37.902400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":757179764,"num_objects":92434,"num_object_clones":0,"num_object_copies":369736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92461,"num_write_kb":739570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":20365312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92488,"ondisk_log_size":92488,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2564988,"kb_used_data":1353372,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374922372,"statfs":{"total":386547056640,"available":383920508928,"internally_reserved":0,"allocated":1385852928,"data_stored":1385184188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,12,6,10,19,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29564928,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3609,"num_write_kb":28872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006903"},"pg_stats":[{"pgid":"1.f","version":"33'5920","reported_seq":5962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.110888+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:35.110888+0000","last_peered":"2024-09-18T11:47:35.110888+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:35.110888+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":47360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5843","reported_seq":6697,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.689746+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:36.689746+0000","last_peered":"2024-09-18T11:47:36.689746+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:36.689746+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5843,"log_dups_size":0,"ondisk_log_size":5843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47830262,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5841,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":397,"num_bytes_recovered":3235840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5729","reported_seq":5813,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.477021+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:36.477021+0000","last_peered":"2024-09-18T11:47:36.477021+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:36.477021+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46905979,"num_objects":5726,"num_object_clones":0,"num_object_copies":22904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5728,"num_write_kb":45814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3672}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5751","reported_seq":5815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.516482+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:36.516482+0000","last_peered":"2024-09-18T11:47:36.516482+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:36.516482+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5751,"log_dups_size":0,"ondisk_log_size":5751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47076598,"num_objects":5747,"num_object_clones":0,"num_object_copies":22988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5749,"num_write_kb":45981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5729","reported_seq":5786,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.131949+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:35.131949+0000","last_peered":"2024-09-18T11:47:35.131949+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:35.131949+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46904566,"num_objects":5726,"num_object_clones":0,"num_object_copies":22904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5727,"num_write_kb":45813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5747","reported_seq":5792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.276590+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:35.276590+0000","last_peered":"2024-09-18T11:47:35.276590+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:35.276590+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5747,"log_dups_size":0,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47079424,"num_objects":5747,"num_object_clones":0,"num_object_copies":22988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5747,"num_write_kb":45976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5854","reported_seq":5921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.616250+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:36.616250+0000","last_peered":"2024-09-18T11:47:36.616250+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:36.616250+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47903990,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":46796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3714}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5768","reported_seq":5833,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.611799+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:36.611799+0000","last_peered":"2024-09-18T11:47:36.611799+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:36.611799+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47207670,"num_objects":5763,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5766,"num_write_kb":46116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5869","reported_seq":5929,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.178590+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:35.178590+0000","last_peered":"2024-09-18T11:47:35.178590+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:35.178590+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48007660,"num_objects":5861,"num_object_clones":0,"num_object_copies":23444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5866,"num_write_kb":46905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5635","reported_seq":5721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.283193+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:36.283193+0000","last_peered":"2024-09-18T11:47:36.283193+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:36.283193+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46142710,"num_objects":5633,"num_object_clones":0,"num_object_copies":22532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5633,"num_write_kb":45062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5766","reported_seq":5825,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.014715+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:35.014715+0000","last_peered":"2024-09-18T11:47:35.014715+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:35.014715+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47189873,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5763,"num_write_kb":46092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5859","reported_seq":5897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.259010+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:35.259010+0000","last_peered":"2024-09-18T11:47:35.259010+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:35.259010+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5859,"log_dups_size":0,"ondisk_log_size":5859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47987323,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":46863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5805","reported_seq":5883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.371566+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:36.371566+0000","last_peered":"2024-09-18T11:47:36.371566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:36.371566+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47544955,"num_objects":5804,"num_object_clones":0,"num_object_copies":23216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":46431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3750}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5785","reported_seq":5836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.239745+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:35.239745+0000","last_peered":"2024-09-18T11:47:35.239745+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:35.239745+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5785,"log_dups_size":0,"ondisk_log_size":5785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47355126,"num_objects":5781,"num_object_clones":0,"num_object_copies":23124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5783,"num_write_kb":46260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5746","reported_seq":10031,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:35.176600+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:35.176600+0000","last_peered":"2024-09-18T11:47:35.176600+0000","last_clean":"2024-09-18T11:47:35.176600+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:35.176600+0000","last_undegraded":"2024-09-18T11:47:35.176600+0000","last_fullsized":"2024-09-18T11:47:35.176600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47043830,"num_objects":5743,"num_object_clones":0,"num_object_copies":22972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":45949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5682","reported_seq":5744,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.652427+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:36.652427+0000","last_peered":"2024-09-18T11:47:36.652427+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:36.652427+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5682,"log_dups_size":0,"ondisk_log_size":5682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46503158,"num_objects":5677,"num_object_clones":0,"num_object_copies":22708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5680,"num_write_kb":45428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":757179764,"num_objects":92434,"num_object_clones":0,"num_object_copies":369736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92461,"num_write_kb":739570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":20365312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1387397120,"data_stored":1387397120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92488,"ondisk_log_size":92488,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678760,"kb_used_data":368416,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93693080,"statfs":{"total":96636764160,"available":95941713920,"internally_reserved":0,"allocated":377257984,"data_stored":377090799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":12.220000000000001},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":12.247999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":10.978999999999999},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.2649999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.2060000000000004},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":7.3890000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670948,"kb_used_data":369436,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93700892,"statfs":{"total":96636764160,"available":95949713408,"internally_reserved":0,"allocated":378302464,"data_stored":378135279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":1.083},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.29899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":20.443000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":0.32000000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":5.0940000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547308,"kb_used_data":245348,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93824532,"statfs":{"total":96636764160,"available":96076320768,"internally_reserved":0,"allocated":251236352,"data_stored":251069167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,1,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":9.4190000000000005},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":3.3149999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":1.647},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":9.4719999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":3.2810000000000001},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":1.7809999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667972,"kb_used_data":370172,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93703868,"statfs":{"total":96636764160,"available":95952760832,"internally_reserved":0,"allocated":379056128,"data_stored":378888943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,2,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.5350000000000001},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":12.351000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":4.9109999999999996},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.4990000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":2.5640000000000001},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":6.3390000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379318272,"data_stored":379318272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251207680,"data_stored":251207680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379097088,"data_stored":379097088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":377774080,"data_stored":377774080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:39.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:40.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 203 20 93628 93608 3.60165 1.95312 0.0647837 0.0433709 2024-09-18T11:47:40.570 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:40.571 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:40.600 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-18T11:47:39.903098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":757179764,"num_objects":92434,"num_object_clones":0,"num_object_copies":369736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92461,"num_write_kb":739570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":20365312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92488,"ondisk_log_size":92488,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2564988,"kb_used_data":1353372,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374922372,"statfs":{"total":386547056640,"available":383920508928,"internally_reserved":0,"allocated":1385852928,"data_stored":1385184188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,12,6,10,19,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23617536,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":23064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006852"},"pg_stats":[{"pgid":"1.f","version":"33'5920","reported_seq":5962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.110888+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:35.110888+0000","last_peered":"2024-09-18T11:47:35.110888+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:35.110888+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":47360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5843","reported_seq":6697,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.689746+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:36.689746+0000","last_peered":"2024-09-18T11:47:36.689746+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:36.689746+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5843,"log_dups_size":0,"ondisk_log_size":5843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47830262,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5841,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":397,"num_bytes_recovered":3235840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5729","reported_seq":5813,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.477021+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:36.477021+0000","last_peered":"2024-09-18T11:47:36.477021+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:36.477021+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46905979,"num_objects":5726,"num_object_clones":0,"num_object_copies":22904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5728,"num_write_kb":45814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3672}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5751","reported_seq":5815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.516482+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:36.516482+0000","last_peered":"2024-09-18T11:47:36.516482+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:36.516482+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5751,"log_dups_size":0,"ondisk_log_size":5751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47076598,"num_objects":5747,"num_object_clones":0,"num_object_copies":22988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5749,"num_write_kb":45981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5729","reported_seq":5786,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.131949+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:35.131949+0000","last_peered":"2024-09-18T11:47:35.131949+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:35.131949+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46904566,"num_objects":5726,"num_object_clones":0,"num_object_copies":22904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5727,"num_write_kb":45813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5747","reported_seq":5792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.276590+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:35.276590+0000","last_peered":"2024-09-18T11:47:35.276590+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:35.276590+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5747,"log_dups_size":0,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47079424,"num_objects":5747,"num_object_clones":0,"num_object_copies":22988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5747,"num_write_kb":45976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5854","reported_seq":5921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.616250+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:36.616250+0000","last_peered":"2024-09-18T11:47:36.616250+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:36.616250+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47903990,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":46796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3714}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5768","reported_seq":5833,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.611799+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:36.611799+0000","last_peered":"2024-09-18T11:47:36.611799+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:36.611799+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47207670,"num_objects":5763,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5766,"num_write_kb":46116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5869","reported_seq":5929,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.178590+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:35.178590+0000","last_peered":"2024-09-18T11:47:35.178590+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:35.178590+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48007660,"num_objects":5861,"num_object_clones":0,"num_object_copies":23444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5866,"num_write_kb":46905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5635","reported_seq":5721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.283193+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:36.283193+0000","last_peered":"2024-09-18T11:47:36.283193+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:36.283193+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46142710,"num_objects":5633,"num_object_clones":0,"num_object_copies":22532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5633,"num_write_kb":45062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5766","reported_seq":5825,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.014715+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:35.014715+0000","last_peered":"2024-09-18T11:47:35.014715+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:35.014715+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47189873,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5763,"num_write_kb":46092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5859","reported_seq":5897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.259010+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:35.259010+0000","last_peered":"2024-09-18T11:47:35.259010+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:35.259010+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5859,"log_dups_size":0,"ondisk_log_size":5859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47987323,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":46863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5805","reported_seq":5883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.371566+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:36.371566+0000","last_peered":"2024-09-18T11:47:36.371566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:36.371566+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47544955,"num_objects":5804,"num_object_clones":0,"num_object_copies":23216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":46431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3750}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5785","reported_seq":5836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.239745+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:35.239745+0000","last_peered":"2024-09-18T11:47:35.239745+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:35.239745+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5785,"log_dups_size":0,"ondisk_log_size":5785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47355126,"num_objects":5781,"num_object_clones":0,"num_object_copies":23124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5783,"num_write_kb":46260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5746","reported_seq":10031,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:35.176600+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:35.176600+0000","last_peered":"2024-09-18T11:47:35.176600+0000","last_clean":"2024-09-18T11:47:35.176600+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:35.176600+0000","last_undegraded":"2024-09-18T11:47:35.176600+0000","last_fullsized":"2024-09-18T11:47:35.176600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47043830,"num_objects":5743,"num_object_clones":0,"num_object_copies":22972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":45949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5682","reported_seq":5744,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.652427+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:36.652427+0000","last_peered":"2024-09-18T11:47:36.652427+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:36.652427+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5682,"log_dups_size":0,"ondisk_log_size":5682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46503158,"num_objects":5677,"num_object_clones":0,"num_object_copies":22708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5680,"num_write_kb":45428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":757179764,"num_objects":92434,"num_object_clones":0,"num_object_copies":369736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92461,"num_write_kb":739570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":20365312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1387397120,"data_stored":1387397120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92488,"ondisk_log_size":92488,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678760,"kb_used_data":368416,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93693080,"statfs":{"total":96636764160,"available":95941713920,"internally_reserved":0,"allocated":377257984,"data_stored":377090799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":12.220000000000001},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":12.247999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":10.978999999999999},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.2649999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.2060000000000004},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":7.3890000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670948,"kb_used_data":369436,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93700892,"statfs":{"total":96636764160,"available":95949713408,"internally_reserved":0,"allocated":378302464,"data_stored":378135279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":1.083},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.29899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":20.443000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":0.32000000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":5.0940000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547308,"kb_used_data":245348,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93824532,"statfs":{"total":96636764160,"available":96076320768,"internally_reserved":0,"allocated":251236352,"data_stored":251069167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,1,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":9.4190000000000005},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":3.3149999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":1.647},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":9.4719999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":3.2810000000000001},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":1.7809999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667972,"kb_used_data":370172,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93703868,"statfs":{"total":96636764160,"available":95952760832,"internally_reserved":0,"allocated":379056128,"data_stored":378888943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,2,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.5350000000000001},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":12.351000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":4.9109999999999996},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.4990000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":2.5640000000000001},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":6.3390000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379318272,"data_stored":379318272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251207680,"data_stored":251207680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379097088,"data_stored":379097088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":377774080,"data_stored":377774080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:40.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:40.601 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5920", 'reported_seq': 5962, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:35.110888+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:47:35.110888+0000', 'last_peered': '2024-09-18T11:47:35.110888+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:47:35.110888+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5920, 'log_dups_size': 0, 'ondisk_log_size': 5920, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48496640, 'num_objects': 5920, 'num_object_clones': 0, 'num_object_copies': 23680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5920, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5920, 'num_write_kb': 47360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3790}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:40.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5843", 'reported_seq': 6697, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:36.689746+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:36.689746+0000', 'last_peered': '2024-09-18T11:47:36.689746+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:36.689746+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5843, 'log_dups_size': 0, 'ondisk_log_size': 5843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47830262, 'num_objects': 5839, 'num_object_clones': 0, 'num_object_copies': 23356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5841, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 397, 'num_bytes_recovered': 3235840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4132}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5729", 'reported_seq': 5813, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:36.477021+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:36.477021+0000', 'last_peered': '2024-09-18T11:47:36.477021+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:36.477021+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5729, 'log_dups_size': 0, 'ondisk_log_size': 5729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46905979, 'num_objects': 5726, 'num_object_clones': 0, 'num_object_copies': 22904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5726, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5728, 'num_write_kb': 45814, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3672}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5751", 'reported_seq': 5815, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:36.516482+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:36.516482+0000', 'last_peered': '2024-09-18T11:47:36.516482+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:36.516482+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5751, 'log_dups_size': 0, 'ondisk_log_size': 5751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47076598, 'num_objects': 5747, 'num_object_clones': 0, 'num_object_copies': 22988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5749, 'num_write_kb': 45981, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3633}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5729", 'reported_seq': 5786, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:35.131949+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:47:35.131949+0000', 'last_peered': '2024-09-18T11:47:35.131949+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:47:35.131949+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5729, 'log_dups_size': 0, 'ondisk_log_size': 5729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46904566, 'num_objects': 5726, 'num_object_clones': 0, 'num_object_copies': 22904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5726, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5727, 'num_write_kb': 45813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3644}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5747", 'reported_seq': 5792, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:35.276590+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:47:35.276590+0000', 'last_peered': '2024-09-18T11:47:35.276590+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:47:35.276590+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5747, 'log_dups_size': 0, 'ondisk_log_size': 5747, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47079424, 'num_objects': 5747, 'num_object_clones': 0, 'num_object_copies': 22988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5747, 'num_write_kb': 45976, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3593}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:40.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5854", 'reported_seq': 5921, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:36.616250+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:36.616250+0000', 'last_peered': '2024-09-18T11:47:36.616250+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:36.616250+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5854, 'log_dups_size': 0, 'ondisk_log_size': 5854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47903990, 'num_objects': 5848, 'num_object_clones': 0, 'num_object_copies': 23392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5852, 'num_write_kb': 46796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3714}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5768", 'reported_seq': 5833, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:36.611799+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:36.611799+0000', 'last_peered': '2024-09-18T11:47:36.611799+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:36.611799+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5768, 'log_dups_size': 0, 'ondisk_log_size': 5768, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47207670, 'num_objects': 5763, 'num_object_clones': 0, 'num_object_copies': 23052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5766, 'num_write_kb': 46116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3729}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5869", 'reported_seq': 5929, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:35.178590+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:47:35.178590+0000', 'last_peered': '2024-09-18T11:47:35.178590+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:47:35.178590+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5869, 'log_dups_size': 0, 'ondisk_log_size': 5869, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48007660, 'num_objects': 5861, 'num_object_clones': 0, 'num_object_copies': 23444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5866, 'num_write_kb': 46905, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3726}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5635", 'reported_seq': 5721, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:36.283193+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:36.283193+0000', 'last_peered': '2024-09-18T11:47:36.283193+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:36.283193+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5635, 'log_dups_size': 0, 'ondisk_log_size': 5635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46142710, 'num_objects': 5633, 'num_object_clones': 0, 'num_object_copies': 22532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5633, 'num_write_kb': 45062, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3658}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5766", 'reported_seq': 5825, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:35.014715+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:47:35.014715+0000', 'last_peered': '2024-09-18T11:47:35.014715+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:47:35.014715+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5766, 'log_dups_size': 0, 'ondisk_log_size': 5766, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47189873, 'num_objects': 5761, 'num_object_clones': 0, 'num_object_copies': 23044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5763, 'num_write_kb': 46092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3698}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5859", 'reported_seq': 5897, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:35.259010+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:47:35.259010+0000', 'last_peered': '2024-09-18T11:47:35.259010+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:47:35.259010+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5859, 'log_dups_size': 0, 'ondisk_log_size': 5859, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47987323, 'num_objects': 5858, 'num_object_clones': 0, 'num_object_copies': 23432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5858, 'num_write_kb': 46863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:40.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5805", 'reported_seq': 5883, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:36.371566+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:36.371566+0000', 'last_peered': '2024-09-18T11:47:36.371566+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:36.371566+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5805, 'log_dups_size': 0, 'ondisk_log_size': 5805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47544955, 'num_objects': 5804, 'num_object_clones': 0, 'num_object_copies': 23216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5804, 'num_write_kb': 46431, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3750}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:40.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:40.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5785", 'reported_seq': 5836, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:35.239745+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:47:35.239745+0000', 'last_peered': '2024-09-18T11:47:35.239745+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:47:35.239745+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5785, 'log_dups_size': 0, 'ondisk_log_size': 5785, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47355126, 'num_objects': 5781, 'num_object_clones': 0, 'num_object_copies': 23124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5783, 'num_write_kb': 46260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3623}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:40.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:40.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5682", 'reported_seq': 5744, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:36.652427+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:36.652427+0000', 'last_peered': '2024-09-18T11:47:36.652427+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:36.652427+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5682, 'log_dups_size': 0, 'ondisk_log_size': 5682, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46503158, 'num_objects': 5677, 'num_object_clones': 0, 'num_object_copies': 22708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5680, 'num_write_kb': 45428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3664}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:40.604 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:41.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 204 20 93886 93866 3.59388 2.01562 0.0995056 0.0434637 2024-09-18T11:47:41.284 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:41.285 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30939/369736 objects degraded (8.368%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":259,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":92434,"data_bytes":757179764,"bytes_used":2626547712,"bytes_avail":383920508928,"bytes_total":386547056640,"degraded_objects":30939,"degraded_total":369736,"degraded_ratio":0.083678624748469183,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":68227,"recovering_keys_per_sec":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"write_bytes_sec":1967004,"read_op_per_sec":0,"write_op_per_sec":240},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:41.304 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:41.304 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:41.848 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:41.848 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:41.871 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-18T11:47:39.903098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":757179764,"num_objects":92434,"num_object_clones":0,"num_object_copies":369736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92461,"num_write_kb":739570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":20365312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92488,"ondisk_log_size":92488,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2564988,"kb_used_data":1353372,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374922372,"statfs":{"total":386547056640,"available":383920508928,"internally_reserved":0,"allocated":1385852928,"data_stored":1385184188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,12,6,10,19,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23617536,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":23064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006852"},"pg_stats":[{"pgid":"1.f","version":"33'5920","reported_seq":5962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.110888+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:35.110888+0000","last_peered":"2024-09-18T11:47:35.110888+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:35.110888+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":47360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5843","reported_seq":6697,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.689746+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:36.689746+0000","last_peered":"2024-09-18T11:47:36.689746+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:36.689746+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5843,"log_dups_size":0,"ondisk_log_size":5843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47830262,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5841,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":397,"num_bytes_recovered":3235840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5729","reported_seq":5813,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.477021+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:36.477021+0000","last_peered":"2024-09-18T11:47:36.477021+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:36.477021+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46905979,"num_objects":5726,"num_object_clones":0,"num_object_copies":22904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5728,"num_write_kb":45814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3672}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5751","reported_seq":5815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.516482+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:36.516482+0000","last_peered":"2024-09-18T11:47:36.516482+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:36.516482+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5751,"log_dups_size":0,"ondisk_log_size":5751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47076598,"num_objects":5747,"num_object_clones":0,"num_object_copies":22988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5749,"num_write_kb":45981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5729","reported_seq":5786,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.131949+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:35.131949+0000","last_peered":"2024-09-18T11:47:35.131949+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:35.131949+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46904566,"num_objects":5726,"num_object_clones":0,"num_object_copies":22904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5727,"num_write_kb":45813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5747","reported_seq":5792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.276590+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:35.276590+0000","last_peered":"2024-09-18T11:47:35.276590+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:35.276590+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5747,"log_dups_size":0,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47079424,"num_objects":5747,"num_object_clones":0,"num_object_copies":22988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5747,"num_write_kb":45976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3593}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5854","reported_seq":5921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.616250+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:36.616250+0000","last_peered":"2024-09-18T11:47:36.616250+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:36.616250+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47903990,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":46796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3714}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5768","reported_seq":5833,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.611799+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:36.611799+0000","last_peered":"2024-09-18T11:47:36.611799+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:36.611799+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47207670,"num_objects":5763,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5766,"num_write_kb":46116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5869","reported_seq":5929,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.178590+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:35.178590+0000","last_peered":"2024-09-18T11:47:35.178590+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:35.178590+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48007660,"num_objects":5861,"num_object_clones":0,"num_object_copies":23444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5866,"num_write_kb":46905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5635","reported_seq":5721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.283193+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:36.283193+0000","last_peered":"2024-09-18T11:47:36.283193+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:36.283193+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46142710,"num_objects":5633,"num_object_clones":0,"num_object_copies":22532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5633,"num_write_kb":45062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5766","reported_seq":5825,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.014715+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:35.014715+0000","last_peered":"2024-09-18T11:47:35.014715+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:35.014715+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47189873,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5763,"num_write_kb":46092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5859","reported_seq":5897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.259010+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:35.259010+0000","last_peered":"2024-09-18T11:47:35.259010+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:35.259010+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5859,"log_dups_size":0,"ondisk_log_size":5859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47987323,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":46863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5805","reported_seq":5883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.371566+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:36.371566+0000","last_peered":"2024-09-18T11:47:36.371566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:36.371566+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47544955,"num_objects":5804,"num_object_clones":0,"num_object_copies":23216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":46431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3750}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5785","reported_seq":5836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:35.239745+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:35.239745+0000","last_peered":"2024-09-18T11:47:35.239745+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:35.239745+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5785,"log_dups_size":0,"ondisk_log_size":5785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47355126,"num_objects":5781,"num_object_clones":0,"num_object_copies":23124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5783,"num_write_kb":46260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5746","reported_seq":10031,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:35.176600+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:35.176600+0000","last_peered":"2024-09-18T11:47:35.176600+0000","last_clean":"2024-09-18T11:47:35.176600+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:35.176600+0000","last_undegraded":"2024-09-18T11:47:35.176600+0000","last_fullsized":"2024-09-18T11:47:35.176600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47043830,"num_objects":5743,"num_object_clones":0,"num_object_copies":22972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":45949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5682","reported_seq":5744,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:36.652427+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:36.652427+0000","last_peered":"2024-09-18T11:47:36.652427+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:36.652427+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5682,"log_dups_size":0,"ondisk_log_size":5682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46503158,"num_objects":5677,"num_object_clones":0,"num_object_copies":22708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5680,"num_write_kb":45428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":757179764,"num_objects":92434,"num_object_clones":0,"num_object_copies":369736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92461,"num_write_kb":739570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":20365312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1387397120,"data_stored":1387397120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92488,"ondisk_log_size":92488,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678760,"kb_used_data":368416,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93693080,"statfs":{"total":96636764160,"available":95941713920,"internally_reserved":0,"allocated":377257984,"data_stored":377090799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":12.220000000000001},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":12.247999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":10.978999999999999},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.2649999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.2060000000000004},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":7.3890000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670948,"kb_used_data":369436,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93700892,"statfs":{"total":96636764160,"available":95949713408,"internally_reserved":0,"allocated":378302464,"data_stored":378135279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":1.083},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.29899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":20.443000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":0.32000000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":5.0940000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547308,"kb_used_data":245348,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93824532,"statfs":{"total":96636764160,"available":96076320768,"internally_reserved":0,"allocated":251236352,"data_stored":251069167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,1,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":9.4190000000000005},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":3.3149999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":1.647},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":9.4719999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":3.2810000000000001},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":1.7809999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667972,"kb_used_data":370172,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93703868,"statfs":{"total":96636764160,"available":95952760832,"internally_reserved":0,"allocated":379056128,"data_stored":378888943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,2,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.5350000000000001},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":12.351000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":4.9109999999999996},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.4990000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":2.5640000000000001},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":6.3390000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379318272,"data_stored":379318272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251207680,"data_stored":251207680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379097088,"data_stored":379097088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":377774080,"data_stored":377774080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:42.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 205 20 94161 94141 3.58683 2.14844 0.0599062 0.0435526 2024-09-18T11:47:43.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 206 20 94455 94435 3.58057 2.29688 0.0516022 0.0436255 2024-09-18T11:47:44.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 207 20 94754 94734 3.57455 2.33594 0.0699409 0.0436983 2024-09-18T11:47:44.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:45.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 208 20 95040 95020 3.56811 2.23438 0.0905085 0.0437768 2024-09-18T11:47:45.426 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:45.426 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:45.445 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-09-18T11:47:43.904061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":768632180,"num_objects":93832,"num_object_clones":0,"num_object_copies":375328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93859,"num_write_kb":750754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2566,"num_bytes_recovered":20783104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93886,"ondisk_log_size":93886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2587752,"kb_used_data":1376136,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374899608,"statfs":{"total":386547056640,"available":383897198592,"internally_reserved":0,"allocated":1409163264,"data_stored":1408494524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,7,5,8,6,6,7,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23207936,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2833,"num_write_kb":22664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006572"},"pg_stats":[{"pgid":"1.f","version":"33'6006","reported_seq":6049,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:40.206260+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:40.206260+0000","last_peered":"2024-09-18T11:47:40.206260+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:40.206260+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49201152,"num_objects":6006,"num_object_clones":0,"num_object_copies":24024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":48048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3876}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5929","reported_seq":6885,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.705066+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:41.705066+0000","last_peered":"2024-09-18T11:47:41.705066+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:41.705066+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5929,"log_dups_size":0,"ondisk_log_size":5929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48534774,"num_objects":5925,"num_object_clones":0,"num_object_copies":23700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5927,"num_write_kb":47412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":448,"num_bytes_recovered":3653632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5819","reported_seq":5903,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.271636+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:41.271636+0000","last_peered":"2024-09-18T11:47:41.271636+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:41.271636+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47643259,"num_objects":5816,"num_object_clones":0,"num_object_copies":23264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5818,"num_write_kb":46534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3762}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5833","reported_seq":5897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.540696+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:41.540696+0000","last_peered":"2024-09-18T11:47:41.540696+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:41.540696+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5833,"log_dups_size":0,"ondisk_log_size":5833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47748342,"num_objects":5829,"num_object_clones":0,"num_object_copies":23316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5831,"num_write_kb":46637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5820","reported_seq":5877,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:40.251149+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:40.251149+0000","last_peered":"2024-09-18T11:47:40.251149+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:40.251149+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5820,"log_dups_size":0,"ondisk_log_size":5820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47650038,"num_objects":5817,"num_object_clones":0,"num_object_copies":23268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5818,"num_write_kb":46541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5824","reported_seq":5869,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:40.191662+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:40.191662+0000","last_peered":"2024-09-18T11:47:40.191662+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:40.191662+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":5824,"num_object_clones":0,"num_object_copies":23296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5824,"num_write_kb":46592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3670}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5934","reported_seq":6001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.725419+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:41.725419+0000","last_peered":"2024-09-18T11:47:41.725419+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:41.725419+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48559350,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5932,"num_write_kb":47436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5839","reported_seq":5904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.546636+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:41.546636+0000","last_peered":"2024-09-18T11:47:41.546636+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:41.546636+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47789302,"num_objects":5834,"num_object_clones":0,"num_object_copies":23336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":46684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3800}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5963","reported_seq":6023,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:40.196510+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:40.196510+0000","last_peered":"2024-09-18T11:47:40.196510+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:40.196510+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5963,"log_dups_size":0,"ondisk_log_size":5963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48777708,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":47657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3820}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5735","reported_seq":5819,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.431987+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:41.431987+0000","last_peered":"2024-09-18T11:47:41.431987+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:41.431987+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5735,"log_dups_size":0,"ondisk_log_size":5735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46961910,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5733,"num_write_kb":45862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5857","reported_seq":5913,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:40.156599+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:40.156599+0000","last_peered":"2024-09-18T11:47:40.156599+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:40.156599+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47935345,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5854,"num_write_kb":46820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5945","reported_seq":5984,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:40.070324+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:40.070324+0000","last_peered":"2024-09-18T11:47:40.070324+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:40.070324+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5945,"log_dups_size":0,"ondisk_log_size":5945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48691835,"num_objects":5944,"num_object_clones":0,"num_object_copies":23776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5944,"num_write_kb":47551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3835}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5883","reported_seq":5961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.376221+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:41.376221+0000","last_peered":"2024-09-18T11:47:41.376221+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:41.376221+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48183931,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":47055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5882","reported_seq":5933,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:40.157281+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:40.157281+0000","last_peered":"2024-09-18T11:47:40.157281+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:40.157281+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5882,"log_dups_size":0,"ondisk_log_size":5882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48149750,"num_objects":5878,"num_object_clones":0,"num_object_copies":23512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5880,"num_write_kb":47036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3720}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5836","reported_seq":10120,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:40.270231+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:40.270231+0000","last_peered":"2024-09-18T11:47:40.270231+0000","last_clean":"2024-09-18T11:47:40.270231+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:40.270231+0000","last_undegraded":"2024-09-18T11:47:40.270231+0000","last_fullsized":"2024-09-18T11:47:40.270231+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47781110,"num_objects":5833,"num_object_clones":0,"num_object_copies":23332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5834,"num_write_kb":46669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5781","reported_seq":5845,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.709147+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:41.709147+0000","last_peered":"2024-09-18T11:47:41.709147+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:41.709147+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314166,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5779,"num_write_kb":46220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":768632180,"num_objects":93832,"num_object_clones":0,"num_object_copies":375328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93859,"num_write_kb":750754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2566,"num_bytes_recovered":20783104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1410613248,"data_stored":1410613248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93886,"ondisk_log_size":93886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672076,"kb_used_data":374212,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93699764,"statfs":{"total":96636764160,"available":95948558336,"internally_reserved":0,"allocated":383193088,"data_stored":383025903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":1.3899999999999999},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.421}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":3.504},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":0.55100000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":4.1790000000000003},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.57699999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680696,"kb_used_data":375024,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93691144,"statfs":{"total":96636764160,"available":95939731456,"internally_reserved":0,"allocated":384024576,"data_stored":383857391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":6.8010000000000002},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.435}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":5.0490000000000004},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":5.7149999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.7759999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557344,"kb_used_data":251224,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93814496,"statfs":{"total":96636764160,"available":96066043904,"internally_reserved":0,"allocated":257253376,"data_stored":257086191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,7,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":45.462000000000003},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.22700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":15.359999999999999},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":15.255000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":1.4850000000000001},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":15.429}]}]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677636,"kb_used_data":375676,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93694204,"statfs":{"total":96636764160,"available":95942864896,"internally_reserved":0,"allocated":384692224,"data_stored":384525039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":8.5109999999999992},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":7.1799999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":3.7469999999999999},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":2.3010000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":10.619999999999999},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":7.2149999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385060864,"data_stored":385060864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257212416,"data_stored":257212416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":384806912,"data_stored":384806912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383533056,"data_stored":383533056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:45.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:45.961 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:45.961 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:45.978 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-18T11:47:45.904647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":774391156,"num_objects":94535,"num_object_clones":0,"num_object_copies":378140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94562,"num_write_kb":756378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2566,"num_bytes_recovered":20783104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94589,"ondisk_log_size":94589,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2607404,"kb_used_data":1387468,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374879956,"statfs":{"total":386547056640,"available":383877074944,"internally_reserved":0,"allocated":1420767232,"data_stored":1420098492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,6,15,17,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28966912,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3536,"num_write_kb":28288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003881"},"pg_stats":[{"pgid":"1.f","version":"33'6091","reported_seq":6134,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.151938+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:45.151938+0000","last_peered":"2024-09-18T11:47:45.151938+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:45.151938+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6091,"log_dups_size":0,"ondisk_log_size":6091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49897472,"num_objects":6091,"num_object_clones":0,"num_object_copies":24364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":48728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5929","reported_seq":6885,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.705066+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:41.705066+0000","last_peered":"2024-09-18T11:47:41.705066+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:41.705066+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5929,"log_dups_size":0,"ondisk_log_size":5929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48534774,"num_objects":5925,"num_object_clones":0,"num_object_copies":23700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5927,"num_write_kb":47412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":448,"num_bytes_recovered":3653632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5819","reported_seq":5903,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.271636+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:41.271636+0000","last_peered":"2024-09-18T11:47:41.271636+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:41.271636+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47643259,"num_objects":5816,"num_object_clones":0,"num_object_copies":23264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5818,"num_write_kb":46534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3762}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5833","reported_seq":5897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.540696+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:41.540696+0000","last_peered":"2024-09-18T11:47:41.540696+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:41.540696+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5833,"log_dups_size":0,"ondisk_log_size":5833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47748342,"num_objects":5829,"num_object_clones":0,"num_object_copies":23316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5831,"num_write_kb":46637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5916","reported_seq":5973,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.143034+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:45.143034+0000","last_peered":"2024-09-18T11:47:45.143034+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:45.143034+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48436470,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":47309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5903","reported_seq":5947,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.302505+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:45.302505+0000","last_peered":"2024-09-18T11:47:45.302505+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:45.302505+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5903,"log_dups_size":0,"ondisk_log_size":5903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48357376,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5903,"num_write_kb":47224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5934","reported_seq":6001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.725419+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:41.725419+0000","last_peered":"2024-09-18T11:47:41.725419+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:41.725419+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48559350,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5932,"num_write_kb":47436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5839","reported_seq":5904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.546636+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:41.546636+0000","last_peered":"2024-09-18T11:47:41.546636+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:41.546636+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47789302,"num_objects":5834,"num_object_clones":0,"num_object_copies":23336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":46684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3800}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6052","reported_seq":6112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.168009+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:45.168009+0000","last_peered":"2024-09-18T11:47:45.168009+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:45.168009+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49506796,"num_objects":6044,"num_object_clones":0,"num_object_copies":24176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":48369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5735","reported_seq":5819,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.431987+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:41.431987+0000","last_peered":"2024-09-18T11:47:41.431987+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:41.431987+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5735,"log_dups_size":0,"ondisk_log_size":5735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46961910,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5733,"num_write_kb":45862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5948","reported_seq":6007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.297702+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:45.297702+0000","last_peered":"2024-09-18T11:47:45.297702+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:45.297702+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48680817,"num_objects":5943,"num_object_clones":0,"num_object_copies":23772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":47548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6025","reported_seq":6064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.228622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:45.228622+0000","last_peered":"2024-09-18T11:47:45.228622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:45.228622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6025,"log_dups_size":0,"ondisk_log_size":6025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49347195,"num_objects":6024,"num_object_clones":0,"num_object_copies":24096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6024,"num_write_kb":48191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3915}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5883","reported_seq":5961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.376221+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:41.376221+0000","last_peered":"2024-09-18T11:47:41.376221+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:41.376221+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48183931,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":47055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5983","reported_seq":6034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.223494+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:45.223494+0000","last_peered":"2024-09-18T11:47:45.223494+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:45.223494+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5983,"log_dups_size":0,"ondisk_log_size":5983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48977142,"num_objects":5979,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5981,"num_write_kb":47844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3821}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5918","reported_seq":10203,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:45.252964+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:45.252964+0000","last_peered":"2024-09-18T11:47:45.252964+0000","last_clean":"2024-09-18T11:47:45.252964+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:45.252964+0000","last_undegraded":"2024-09-18T11:47:45.252964+0000","last_fullsized":"2024-09-18T11:47:45.252964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48452854,"num_objects":5915,"num_object_clones":0,"num_object_copies":23660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5916,"num_write_kb":47325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5781","reported_seq":5845,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.709147+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:41.709147+0000","last_peered":"2024-09-18T11:47:41.709147+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:41.709147+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314166,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5779,"num_write_kb":46220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":774391156,"num_objects":94535,"num_object_clones":0,"num_object_copies":378140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94562,"num_write_kb":756378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2566,"num_bytes_recovered":20783104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1422508032,"data_stored":1422508032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94589,"ondisk_log_size":94589,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681680,"kb_used_data":379656,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93690160,"statfs":{"total":96636764160,"available":95938723840,"internally_reserved":0,"allocated":388767744,"data_stored":388600559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":6.883},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.9089999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":6.9390000000000001},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.9119999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.8779999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680696,"kb_used_data":375024,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93691144,"statfs":{"total":96636764160,"available":95939731456,"internally_reserved":0,"allocated":384024576,"data_stored":383857391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":6.8010000000000002},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.435}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":5.0490000000000004},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":5.7149999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.7759999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567392,"kb_used_data":257112,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93804448,"statfs":{"total":96636764160,"available":96055754752,"internally_reserved":0,"allocated":263282688,"data_stored":263115503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":4.2000000000000002},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":4.1479999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":1.0029999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":6.1660000000000004},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.33000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677636,"kb_used_data":375676,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93694204,"statfs":{"total":96636764160,"available":95942864896,"internally_reserved":0,"allocated":384692224,"data_stored":384525039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":8.5109999999999992},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":7.1799999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":3.7469999999999999},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":2.3010000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":10.619999999999999},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":7.2149999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385060864,"data_stored":385060864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263270400,"data_stored":263270400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":384806912,"data_stored":384806912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389369856,"data_stored":389369856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:45.980 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:45.980 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6091", 'reported_seq': 6134, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.151938+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:47:45.151938+0000', 'last_peered': '2024-09-18T11:47:45.151938+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:47:45.151938+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6091, 'log_dups_size': 0, 'ondisk_log_size': 6091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49897472, 'num_objects': 6091, 'num_object_clones': 0, 'num_object_copies': 24364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6091, 'num_write_kb': 48728, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:45.980 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:45.980 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5929", 'reported_seq': 6885, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:41.705066+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:41.705066+0000', 'last_peered': '2024-09-18T11:47:41.705066+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:41.705066+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5929, 'log_dups_size': 0, 'ondisk_log_size': 5929, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48534774, 'num_objects': 5925, 'num_object_clones': 0, 'num_object_copies': 23700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1656, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5927, 'num_write_kb': 47412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 448, 'num_bytes_recovered': 3653632, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4269}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:45.980 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:45.980 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5819", 'reported_seq': 5903, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:41.271636+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:41.271636+0000', 'last_peered': '2024-09-18T11:47:41.271636+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:41.271636+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5819, 'log_dups_size': 0, 'ondisk_log_size': 5819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47643259, 'num_objects': 5816, 'num_object_clones': 0, 'num_object_copies': 23264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5816, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5818, 'num_write_kb': 46534, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3762}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5833", 'reported_seq': 5897, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:41.540696+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:41.540696+0000', 'last_peered': '2024-09-18T11:47:41.540696+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:41.540696+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5833, 'log_dups_size': 0, 'ondisk_log_size': 5833, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47748342, 'num_objects': 5829, 'num_object_clones': 0, 'num_object_copies': 23316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5831, 'num_write_kb': 46637, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3715}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5916", 'reported_seq': 5973, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.143034+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:47:45.143034+0000', 'last_peered': '2024-09-18T11:47:45.143034+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:47:45.143034+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5916, 'log_dups_size': 0, 'ondisk_log_size': 5916, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48436470, 'num_objects': 5913, 'num_object_clones': 0, 'num_object_copies': 23652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5914, 'num_write_kb': 47309, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3831}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5903", 'reported_seq': 5947, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.302505+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:47:45.302505+0000', 'last_peered': '2024-09-18T11:47:45.302505+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:47:45.302505+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5903, 'log_dups_size': 0, 'ondisk_log_size': 5903, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48357376, 'num_objects': 5903, 'num_object_clones': 0, 'num_object_copies': 23612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5903, 'num_write_kb': 47224, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3749}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5934", 'reported_seq': 6001, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:41.725419+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:41.725419+0000', 'last_peered': '2024-09-18T11:47:41.725419+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:41.725419+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5934, 'log_dups_size': 0, 'ondisk_log_size': 5934, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48559350, 'num_objects': 5928, 'num_object_clones': 0, 'num_object_copies': 23712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5932, 'num_write_kb': 47436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5839", 'reported_seq': 5904, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:41.546636+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:41.546636+0000', 'last_peered': '2024-09-18T11:47:41.546636+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:41.546636+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5839, 'log_dups_size': 0, 'ondisk_log_size': 5839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47789302, 'num_objects': 5834, 'num_object_clones': 0, 'num_object_copies': 23336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5837, 'num_write_kb': 46684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3800}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:45.981 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6052", 'reported_seq': 6112, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.168009+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:47:45.168009+0000', 'last_peered': '2024-09-18T11:47:45.168009+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:47:45.168009+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6052, 'log_dups_size': 0, 'ondisk_log_size': 6052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49506796, 'num_objects': 6044, 'num_object_clones': 0, 'num_object_copies': 24176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6049, 'num_write_kb': 48369, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3909}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5735", 'reported_seq': 5819, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:41.431987+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:41.431987+0000', 'last_peered': '2024-09-18T11:47:41.431987+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:41.431987+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5735, 'log_dups_size': 0, 'ondisk_log_size': 5735, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46961910, 'num_objects': 5733, 'num_object_clones': 0, 'num_object_copies': 22932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5733, 'num_write_kb': 45862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3758}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5948", 'reported_seq': 6007, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.297702+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:47:45.297702+0000', 'last_peered': '2024-09-18T11:47:45.297702+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:47:45.297702+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5948, 'log_dups_size': 0, 'ondisk_log_size': 5948, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48680817, 'num_objects': 5943, 'num_object_clones': 0, 'num_object_copies': 23772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5945, 'num_write_kb': 47548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3880}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6025", 'reported_seq': 6064, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.228622+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:47:45.228622+0000', 'last_peered': '2024-09-18T11:47:45.228622+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:47:45.228622+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6025, 'log_dups_size': 0, 'ondisk_log_size': 6025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49347195, 'num_objects': 6024, 'num_object_clones': 0, 'num_object_copies': 24096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6024, 'num_write_kb': 48191, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3915}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5883", 'reported_seq': 5961, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:41.376221+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:41.376221+0000', 'last_peered': '2024-09-18T11:47:41.376221+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:41.376221+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5883, 'log_dups_size': 0, 'ondisk_log_size': 5883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48183931, 'num_objects': 5882, 'num_object_clones': 0, 'num_object_copies': 23528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5882, 'num_write_kb': 47055, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:45.982 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5983", 'reported_seq': 6034, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.223494+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:47:45.223494+0000', 'last_peered': '2024-09-18T11:47:45.223494+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:47:45.223494+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5983, 'log_dups_size': 0, 'ondisk_log_size': 5983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48977142, 'num_objects': 5979, 'num_object_clones': 0, 'num_object_copies': 23916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5981, 'num_write_kb': 47844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3821}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:45.983 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:45.983 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5781", 'reported_seq': 5845, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:41.709147+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:41.709147+0000', 'last_peered': '2024-09-18T11:47:41.709147+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:41.709147+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5781, 'log_dups_size': 0, 'ondisk_log_size': 5781, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47314166, 'num_objects': 5776, 'num_object_clones': 0, 'num_object_copies': 23104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5779, 'num_write_kb': 46220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:45.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:46.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 209 20 95299 95279 3.56068 2.02344 0.127145 0.0438689 2024-09-18T11:47:46.699 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:46.700 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30888/378140 objects degraded (8.168%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":264,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94535,"data_bytes":774391156,"bytes_used":2669981696,"bytes_avail":383877074944,"bytes_total":386547056640,"degraded_objects":30888,"degraded_total":378140,"degraded_ratio":0.081684032368963877,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":70291,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"write_bytes_sec":2413128,"read_op_per_sec":0,"write_op_per_sec":294},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:46.727 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:46.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:47.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 210 20 95538 95518 3.55262 1.86719 0.0858251 0.0439643 2024-09-18T11:47:47.384 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:47.384 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:47.417 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-18T11:47:45.904647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":774391156,"num_objects":94535,"num_object_clones":0,"num_object_copies":378140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94562,"num_write_kb":756378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2566,"num_bytes_recovered":20783104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94589,"ondisk_log_size":94589,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2607404,"kb_used_data":1387468,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374879956,"statfs":{"total":386547056640,"available":383877074944,"internally_reserved":0,"allocated":1420767232,"data_stored":1420098492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,6,15,17,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28966912,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3536,"num_write_kb":28288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003881"},"pg_stats":[{"pgid":"1.f","version":"33'6091","reported_seq":6134,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.151938+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:45.151938+0000","last_peered":"2024-09-18T11:47:45.151938+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:45.151938+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6091,"log_dups_size":0,"ondisk_log_size":6091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49897472,"num_objects":6091,"num_object_clones":0,"num_object_copies":24364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":48728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5929","reported_seq":6885,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.705066+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:41.705066+0000","last_peered":"2024-09-18T11:47:41.705066+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:41.705066+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5929,"log_dups_size":0,"ondisk_log_size":5929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48534774,"num_objects":5925,"num_object_clones":0,"num_object_copies":23700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5927,"num_write_kb":47412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":448,"num_bytes_recovered":3653632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5819","reported_seq":5903,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.271636+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:41.271636+0000","last_peered":"2024-09-18T11:47:41.271636+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:41.271636+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47643259,"num_objects":5816,"num_object_clones":0,"num_object_copies":23264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5818,"num_write_kb":46534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3762}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5833","reported_seq":5897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.540696+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:41.540696+0000","last_peered":"2024-09-18T11:47:41.540696+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:41.540696+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5833,"log_dups_size":0,"ondisk_log_size":5833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47748342,"num_objects":5829,"num_object_clones":0,"num_object_copies":23316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5831,"num_write_kb":46637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5916","reported_seq":5973,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.143034+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:45.143034+0000","last_peered":"2024-09-18T11:47:45.143034+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:45.143034+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48436470,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":47309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5903","reported_seq":5947,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.302505+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:45.302505+0000","last_peered":"2024-09-18T11:47:45.302505+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:45.302505+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5903,"log_dups_size":0,"ondisk_log_size":5903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48357376,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5903,"num_write_kb":47224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5934","reported_seq":6001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.725419+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:41.725419+0000","last_peered":"2024-09-18T11:47:41.725419+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:41.725419+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48559350,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5932,"num_write_kb":47436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5839","reported_seq":5904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.546636+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:41.546636+0000","last_peered":"2024-09-18T11:47:41.546636+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:41.546636+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47789302,"num_objects":5834,"num_object_clones":0,"num_object_copies":23336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":46684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3800}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6052","reported_seq":6112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.168009+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:45.168009+0000","last_peered":"2024-09-18T11:47:45.168009+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:45.168009+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49506796,"num_objects":6044,"num_object_clones":0,"num_object_copies":24176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":48369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5735","reported_seq":5819,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.431987+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:41.431987+0000","last_peered":"2024-09-18T11:47:41.431987+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:41.431987+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5735,"log_dups_size":0,"ondisk_log_size":5735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46961910,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5733,"num_write_kb":45862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5948","reported_seq":6007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.297702+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:45.297702+0000","last_peered":"2024-09-18T11:47:45.297702+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:45.297702+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48680817,"num_objects":5943,"num_object_clones":0,"num_object_copies":23772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":47548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6025","reported_seq":6064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.228622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:45.228622+0000","last_peered":"2024-09-18T11:47:45.228622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:45.228622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6025,"log_dups_size":0,"ondisk_log_size":6025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49347195,"num_objects":6024,"num_object_clones":0,"num_object_copies":24096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6024,"num_write_kb":48191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3915}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5883","reported_seq":5961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.376221+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:41.376221+0000","last_peered":"2024-09-18T11:47:41.376221+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:41.376221+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48183931,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":47055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5983","reported_seq":6034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.223494+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:45.223494+0000","last_peered":"2024-09-18T11:47:45.223494+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:45.223494+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5983,"log_dups_size":0,"ondisk_log_size":5983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48977142,"num_objects":5979,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5981,"num_write_kb":47844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3821}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5918","reported_seq":10203,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:45.252964+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:45.252964+0000","last_peered":"2024-09-18T11:47:45.252964+0000","last_clean":"2024-09-18T11:47:45.252964+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:45.252964+0000","last_undegraded":"2024-09-18T11:47:45.252964+0000","last_fullsized":"2024-09-18T11:47:45.252964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48452854,"num_objects":5915,"num_object_clones":0,"num_object_copies":23660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5916,"num_write_kb":47325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5781","reported_seq":5845,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:41.709147+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:41.709147+0000","last_peered":"2024-09-18T11:47:41.709147+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:41.709147+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314166,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5779,"num_write_kb":46220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":774391156,"num_objects":94535,"num_object_clones":0,"num_object_copies":378140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94562,"num_write_kb":756378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2566,"num_bytes_recovered":20783104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1422508032,"data_stored":1422508032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94589,"ondisk_log_size":94589,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681680,"kb_used_data":379656,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93690160,"statfs":{"total":96636764160,"available":95938723840,"internally_reserved":0,"allocated":388767744,"data_stored":388600559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":6.883},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.9089999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":6.9390000000000001},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.9119999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.8779999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680696,"kb_used_data":375024,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93691144,"statfs":{"total":96636764160,"available":95939731456,"internally_reserved":0,"allocated":384024576,"data_stored":383857391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":6.8010000000000002},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.435}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":5.0490000000000004},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":5.7149999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.7759999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567392,"kb_used_data":257112,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93804448,"statfs":{"total":96636764160,"available":96055754752,"internally_reserved":0,"allocated":263282688,"data_stored":263115503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":4.2000000000000002},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":4.1479999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":1.0029999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":6.1660000000000004},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.33000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677636,"kb_used_data":375676,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93694204,"statfs":{"total":96636764160,"available":95942864896,"internally_reserved":0,"allocated":384692224,"data_stored":384525039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":8.5109999999999992},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":7.1799999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":3.7469999999999999},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":2.3010000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":10.619999999999999},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":7.2149999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385060864,"data_stored":385060864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263270400,"data_stored":263270400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":384806912,"data_stored":384806912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389369856,"data_stored":389369856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:48.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 211 20 95775 95755 3.54455 1.85156 0.0615012 0.0440676 2024-09-18T11:47:49.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 212 20 96061 96041 3.53837 2.23438 0.0801172 0.0441462 2024-09-18T11:47:50.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 213 20 96355 96335 3.53255 2.29688 0.0163659 0.0442159 2024-09-18T11:47:50.420 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:50.982 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:50.983 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:51.014 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-18T11:47:49.905788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":780232052,"num_objects":95248,"num_object_clones":0,"num_object_copies":380992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95275,"num_write_kb":762082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2617,"num_bytes_recovered":21200896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95302,"ondisk_log_size":95302,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2628116,"kb_used_data":1399860,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":374859244,"statfs":{"total":386547056640,"available":383855865856,"internally_reserved":0,"allocated":1433456640,"data_stored":1432787900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,6,19,19,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23052288,"num_objects":2814,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":22512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003387"},"pg_stats":[{"pgid":"1.f","version":"33'6091","reported_seq":6134,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.151938+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:45.151938+0000","last_peered":"2024-09-18T11:47:45.151938+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:45.151938+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6091,"log_dups_size":0,"ondisk_log_size":6091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49897472,"num_objects":6091,"num_object_clones":0,"num_object_copies":24364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":48728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6038","reported_seq":7097,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.765627+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:46.765627+0000","last_peered":"2024-09-18T11:47:46.765627+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:46.765627+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49427702,"num_objects":6034,"num_object_clones":0,"num_object_copies":24136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6036,"num_write_kb":48284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":4071424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5901","reported_seq":5985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.496756+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:46.496756+0000","last_peered":"2024-09-18T11:47:46.496756+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:46.496756+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5901,"log_dups_size":0,"ondisk_log_size":5901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48315003,"num_objects":5898,"num_object_clones":0,"num_object_copies":23592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5900,"num_write_kb":47190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3844}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5921","reported_seq":5985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.692976+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:46.692976+0000","last_peered":"2024-09-18T11:47:46.692976+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:46.692976+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48469238,"num_objects":5917,"num_object_clones":0,"num_object_copies":23668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5919,"num_write_kb":47341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5916","reported_seq":5973,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.143034+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:45.143034+0000","last_peered":"2024-09-18T11:47:45.143034+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:45.143034+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48436470,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":47309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5903","reported_seq":5947,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.302505+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:45.302505+0000","last_peered":"2024-09-18T11:47:45.302505+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:45.302505+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5903,"log_dups_size":0,"ondisk_log_size":5903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48357376,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5903,"num_write_kb":47224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6023","reported_seq":6089,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.720670+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:46.720670+0000","last_peered":"2024-09-18T11:47:46.720670+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:46.720670+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6023,"log_dups_size":0,"ondisk_log_size":6023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49288438,"num_objects":6017,"num_object_clones":0,"num_object_copies":24068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6021,"num_write_kb":48148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3883}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5922","reported_seq":5985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.676349+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:46.676349+0000","last_peered":"2024-09-18T11:47:46.676349+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:46.676349+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5922,"log_dups_size":0,"ondisk_log_size":5922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48469238,"num_objects":5917,"num_object_clones":0,"num_object_copies":23668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":47348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6052","reported_seq":6112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.168009+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:45.168009+0000","last_peered":"2024-09-18T11:47:45.168009+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:45.168009+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49506796,"num_objects":6044,"num_object_clones":0,"num_object_copies":24176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":48369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5812","reported_seq":5898,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.444298+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:46.444298+0000","last_peered":"2024-09-18T11:47:46.444298+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:46.444298+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5812,"log_dups_size":0,"ondisk_log_size":5812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47592694,"num_objects":5810,"num_object_clones":0,"num_object_copies":23240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5810,"num_write_kb":46478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5948","reported_seq":6007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.297702+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:45.297702+0000","last_peered":"2024-09-18T11:47:45.297702+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:45.297702+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48680817,"num_objects":5943,"num_object_clones":0,"num_object_copies":23772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":47548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6025","reported_seq":6064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.228622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:45.228622+0000","last_peered":"2024-09-18T11:47:45.228622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:45.228622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6025,"log_dups_size":0,"ondisk_log_size":6025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49347195,"num_objects":6024,"num_object_clones":0,"num_object_copies":24096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6024,"num_write_kb":48191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3915}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5975","reported_seq":6053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.264424+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:46.264424+0000","last_peered":"2024-09-18T11:47:46.264424+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:46.264424+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5975,"log_dups_size":0,"ondisk_log_size":5975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48937595,"num_objects":5974,"num_object_clones":0,"num_object_copies":23896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5974,"num_write_kb":47791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3920}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5983","reported_seq":6034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.223494+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:45.223494+0000","last_peered":"2024-09-18T11:47:45.223494+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:45.223494+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5983,"log_dups_size":0,"ondisk_log_size":5983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48977142,"num_objects":5979,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5981,"num_write_kb":47844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3821}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5918","reported_seq":10203,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:45.252964+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:45.252964+0000","last_peered":"2024-09-18T11:47:45.252964+0000","last_clean":"2024-09-18T11:47:45.252964+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:45.252964+0000","last_undegraded":"2024-09-18T11:47:45.252964+0000","last_fullsized":"2024-09-18T11:47:45.252964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48452854,"num_objects":5915,"num_object_clones":0,"num_object_copies":23660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5916,"num_write_kb":47325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5874","reported_seq":5938,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.579971+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:46.579971+0000","last_peered":"2024-09-18T11:47:46.579971+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:46.579971+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5874,"log_dups_size":0,"ondisk_log_size":5874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48076022,"num_objects":5869,"num_object_clones":0,"num_object_copies":23476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":46964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3856}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":780232052,"num_objects":95248,"num_object_clones":0,"num_object_copies":380992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95275,"num_write_kb":762082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2617,"num_bytes_recovered":21200896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1434169344,"data_stored":1434169344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95302,"ondisk_log_size":95302,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681680,"kb_used_data":379656,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93690160,"statfs":{"total":96636764160,"available":95938723840,"internally_reserved":0,"allocated":388767744,"data_stored":388600559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":6.883},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.9089999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":6.9390000000000001},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.9119999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.8779999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691564,"kb_used_data":381732,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93680276,"statfs":{"total":96636764160,"available":95928602624,"internally_reserved":0,"allocated":390893568,"data_stored":390726383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,1,3,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":5.093},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":5.1230000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":11.964},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":5.173}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":5.1509999999999998},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.67900000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567392,"kb_used_data":257112,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93804448,"statfs":{"total":96636764160,"available":96055754752,"internally_reserved":0,"allocated":263282688,"data_stored":263115503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":4.2000000000000002},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":4.1479999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":1.0029999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":6.1660000000000004},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.33000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687480,"kb_used_data":381360,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93684360,"statfs":{"total":96636764160,"available":95932784640,"internally_reserved":0,"allocated":390512640,"data_stored":390345455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":5.7670000000000003},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":19.077999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":19.837},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":9.9770000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":3.968},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":19.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":390877184,"data_stored":390877184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263270400,"data_stored":263270400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":390651904,"data_stored":390651904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389369856,"data_stored":389369856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:51.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:51.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 214 20 96619 96599 3.52568 2.0625 0.0480379 0.0443043 2024-09-18T11:47:51.552 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:51.553 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:51.572 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-18T11:47:49.905788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":780232052,"num_objects":95248,"num_object_clones":0,"num_object_copies":380992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95275,"num_write_kb":762082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2617,"num_bytes_recovered":21200896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95302,"ondisk_log_size":95302,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2628116,"kb_used_data":1399860,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":374859244,"statfs":{"total":386547056640,"available":383855865856,"internally_reserved":0,"allocated":1433456640,"data_stored":1432787900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,6,19,19,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23052288,"num_objects":2814,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":22512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003387"},"pg_stats":[{"pgid":"1.f","version":"33'6091","reported_seq":6134,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.151938+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:45.151938+0000","last_peered":"2024-09-18T11:47:45.151938+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:45.151938+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6091,"log_dups_size":0,"ondisk_log_size":6091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49897472,"num_objects":6091,"num_object_clones":0,"num_object_copies":24364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":48728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6038","reported_seq":7097,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.765627+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:46.765627+0000","last_peered":"2024-09-18T11:47:46.765627+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:46.765627+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49427702,"num_objects":6034,"num_object_clones":0,"num_object_copies":24136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6036,"num_write_kb":48284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":4071424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5901","reported_seq":5985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.496756+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:46.496756+0000","last_peered":"2024-09-18T11:47:46.496756+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:46.496756+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5901,"log_dups_size":0,"ondisk_log_size":5901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48315003,"num_objects":5898,"num_object_clones":0,"num_object_copies":23592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5900,"num_write_kb":47190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3844}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5921","reported_seq":5985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.692976+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:46.692976+0000","last_peered":"2024-09-18T11:47:46.692976+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:46.692976+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48469238,"num_objects":5917,"num_object_clones":0,"num_object_copies":23668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5919,"num_write_kb":47341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5916","reported_seq":5973,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.143034+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:45.143034+0000","last_peered":"2024-09-18T11:47:45.143034+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:45.143034+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48436470,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":47309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5903","reported_seq":5947,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.302505+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:45.302505+0000","last_peered":"2024-09-18T11:47:45.302505+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:45.302505+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5903,"log_dups_size":0,"ondisk_log_size":5903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48357376,"num_objects":5903,"num_object_clones":0,"num_object_copies":23612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5903,"num_write_kb":47224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6023","reported_seq":6089,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.720670+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:46.720670+0000","last_peered":"2024-09-18T11:47:46.720670+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:46.720670+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6023,"log_dups_size":0,"ondisk_log_size":6023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49288438,"num_objects":6017,"num_object_clones":0,"num_object_copies":24068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6021,"num_write_kb":48148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3883}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5922","reported_seq":5985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.676349+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:46.676349+0000","last_peered":"2024-09-18T11:47:46.676349+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:46.676349+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5922,"log_dups_size":0,"ondisk_log_size":5922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48469238,"num_objects":5917,"num_object_clones":0,"num_object_copies":23668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":47348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6052","reported_seq":6112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.168009+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:45.168009+0000","last_peered":"2024-09-18T11:47:45.168009+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:45.168009+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49506796,"num_objects":6044,"num_object_clones":0,"num_object_copies":24176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":48369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5812","reported_seq":5898,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.444298+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:46.444298+0000","last_peered":"2024-09-18T11:47:46.444298+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:46.444298+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5812,"log_dups_size":0,"ondisk_log_size":5812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47592694,"num_objects":5810,"num_object_clones":0,"num_object_copies":23240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5810,"num_write_kb":46478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5948","reported_seq":6007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.297702+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:45.297702+0000","last_peered":"2024-09-18T11:47:45.297702+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:45.297702+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48680817,"num_objects":5943,"num_object_clones":0,"num_object_copies":23772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":47548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6025","reported_seq":6064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.228622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:45.228622+0000","last_peered":"2024-09-18T11:47:45.228622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:45.228622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6025,"log_dups_size":0,"ondisk_log_size":6025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49347195,"num_objects":6024,"num_object_clones":0,"num_object_copies":24096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6024,"num_write_kb":48191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3915}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5975","reported_seq":6053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.264424+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:46.264424+0000","last_peered":"2024-09-18T11:47:46.264424+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:46.264424+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5975,"log_dups_size":0,"ondisk_log_size":5975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48937595,"num_objects":5974,"num_object_clones":0,"num_object_copies":23896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5974,"num_write_kb":47791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3920}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5983","reported_seq":6034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:45.223494+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:45.223494+0000","last_peered":"2024-09-18T11:47:45.223494+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:45.223494+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5983,"log_dups_size":0,"ondisk_log_size":5983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48977142,"num_objects":5979,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5981,"num_write_kb":47844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3821}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5918","reported_seq":10203,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:45.252964+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:45.252964+0000","last_peered":"2024-09-18T11:47:45.252964+0000","last_clean":"2024-09-18T11:47:45.252964+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:45.252964+0000","last_undegraded":"2024-09-18T11:47:45.252964+0000","last_fullsized":"2024-09-18T11:47:45.252964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48452854,"num_objects":5915,"num_object_clones":0,"num_object_copies":23660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5916,"num_write_kb":47325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5874","reported_seq":5938,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:46.579971+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:46.579971+0000","last_peered":"2024-09-18T11:47:46.579971+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:46.579971+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5874,"log_dups_size":0,"ondisk_log_size":5874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48076022,"num_objects":5869,"num_object_clones":0,"num_object_copies":23476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":46964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3856}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":780232052,"num_objects":95248,"num_object_clones":0,"num_object_copies":380992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95275,"num_write_kb":762082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2617,"num_bytes_recovered":21200896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1434169344,"data_stored":1434169344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95302,"ondisk_log_size":95302,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681680,"kb_used_data":379656,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93690160,"statfs":{"total":96636764160,"available":95938723840,"internally_reserved":0,"allocated":388767744,"data_stored":388600559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":6.883},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.9089999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:46:47 2024","interfaces":[{"interface":"back","average":{"1min":4.452,"5min":5.2759999999999998,"15min":5.4139999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.307,"5min":13.68,"15min":13.68},"last":6.9390000000000001},{"interface":"front","average":{"1min":4.1059999999999999,"5min":4.181,"15min":4.1929999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":11.904,"5min":14.941000000000001,"15min":14.941000000000001},"last":1.9119999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.8779999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691564,"kb_used_data":381732,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93680276,"statfs":{"total":96636764160,"available":95928602624,"internally_reserved":0,"allocated":390893568,"data_stored":390726383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,1,3,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":5.093},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":5.1230000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:46:46 2024","interfaces":[{"interface":"back","average":{"1min":6.9800000000000004,"5min":6.1399999999999997,"15min":6},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.626999999999999,"5min":21.626999999999999,"15min":21.626999999999999},"last":11.964},{"interface":"front","average":{"1min":3.5019999999999998,"5min":3.625,"15min":3.645},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":14.31,"5min":14.31,"15min":14.31},"last":5.173}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":5.1509999999999998},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.67900000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567392,"kb_used_data":257112,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93804448,"statfs":{"total":96636764160,"available":96055754752,"internally_reserved":0,"allocated":263282688,"data_stored":263115503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":4.2000000000000002},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":4.1479999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":1.0029999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":6.1660000000000004},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":0.33000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687480,"kb_used_data":381360,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93684360,"statfs":{"total":96636764160,"available":95932784640,"internally_reserved":0,"allocated":390512640,"data_stored":390345455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":5.7670000000000003},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":19.077999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":19.837},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":9.9770000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":3.968},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":19.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":390877184,"data_stored":390877184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263270400,"data_stored":263270400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":390651904,"data_stored":390651904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389369856,"data_stored":389369856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:51.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:51.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6091", 'reported_seq': 6134, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.151938+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:47:45.151938+0000', 'last_peered': '2024-09-18T11:47:45.151938+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:47:45.151938+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6091, 'log_dups_size': 0, 'ondisk_log_size': 6091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49897472, 'num_objects': 6091, 'num_object_clones': 0, 'num_object_copies': 24364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6091, 'num_write_kb': 48728, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:51.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:51.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6038", 'reported_seq': 7097, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:46.765627+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:46.765627+0000', 'last_peered': '2024-09-18T11:47:46.765627+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:46.765627+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6038, 'log_dups_size': 0, 'ondisk_log_size': 6038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49427702, 'num_objects': 6034, 'num_object_clones': 0, 'num_object_copies': 24136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1605, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6034, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6036, 'num_write_kb': 48284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 499, 'num_bytes_recovered': 4071424, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4429}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:51.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:51.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5901", 'reported_seq': 5985, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:46.496756+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:46.496756+0000', 'last_peered': '2024-09-18T11:47:46.496756+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:46.496756+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5901, 'log_dups_size': 0, 'ondisk_log_size': 5901, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48315003, 'num_objects': 5898, 'num_object_clones': 0, 'num_object_copies': 23592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5900, 'num_write_kb': 47190, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3844}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:51.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:51.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5921", 'reported_seq': 5985, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:46.692976+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:46.692976+0000', 'last_peered': '2024-09-18T11:47:46.692976+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:46.692976+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5921, 'log_dups_size': 0, 'ondisk_log_size': 5921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48469238, 'num_objects': 5917, 'num_object_clones': 0, 'num_object_copies': 23668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5917, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5919, 'num_write_kb': 47341, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3803}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:51.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:51.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5916", 'reported_seq': 5973, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.143034+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:47:45.143034+0000', 'last_peered': '2024-09-18T11:47:45.143034+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:47:45.143034+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5916, 'log_dups_size': 0, 'ondisk_log_size': 5916, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48436470, 'num_objects': 5913, 'num_object_clones': 0, 'num_object_copies': 23652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5914, 'num_write_kb': 47309, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3831}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:51.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:51.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5903", 'reported_seq': 5947, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.302505+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:47:45.302505+0000', 'last_peered': '2024-09-18T11:47:45.302505+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:47:45.302505+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5903, 'log_dups_size': 0, 'ondisk_log_size': 5903, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48357376, 'num_objects': 5903, 'num_object_clones': 0, 'num_object_copies': 23612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5903, 'num_write_kb': 47224, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3749}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:51.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:51.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6023", 'reported_seq': 6089, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:46.720670+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:46.720670+0000', 'last_peered': '2024-09-18T11:47:46.720670+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:46.720670+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6023, 'log_dups_size': 0, 'ondisk_log_size': 6023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49288438, 'num_objects': 6017, 'num_object_clones': 0, 'num_object_copies': 24068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6021, 'num_write_kb': 48148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3883}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5922", 'reported_seq': 5985, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:46.676349+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:46.676349+0000', 'last_peered': '2024-09-18T11:47:46.676349+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:46.676349+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5922, 'log_dups_size': 0, 'ondisk_log_size': 5922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48469238, 'num_objects': 5917, 'num_object_clones': 0, 'num_object_copies': 23668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5917, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5920, 'num_write_kb': 47348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3883}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6052", 'reported_seq': 6112, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.168009+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:47:45.168009+0000', 'last_peered': '2024-09-18T11:47:45.168009+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:47:45.168009+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6052, 'log_dups_size': 0, 'ondisk_log_size': 6052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49506796, 'num_objects': 6044, 'num_object_clones': 0, 'num_object_copies': 24176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6049, 'num_write_kb': 48369, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3909}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5812", 'reported_seq': 5898, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:46.444298+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:46.444298+0000', 'last_peered': '2024-09-18T11:47:46.444298+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:46.444298+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5812, 'log_dups_size': 0, 'ondisk_log_size': 5812, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47592694, 'num_objects': 5810, 'num_object_clones': 0, 'num_object_copies': 23240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5810, 'num_write_kb': 46478, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3835}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5948", 'reported_seq': 6007, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.297702+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:47:45.297702+0000', 'last_peered': '2024-09-18T11:47:45.297702+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:47:45.297702+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5948, 'log_dups_size': 0, 'ondisk_log_size': 5948, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48680817, 'num_objects': 5943, 'num_object_clones': 0, 'num_object_copies': 23772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5945, 'num_write_kb': 47548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3880}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6025", 'reported_seq': 6064, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.228622+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:47:45.228622+0000', 'last_peered': '2024-09-18T11:47:45.228622+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:47:45.228622+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6025, 'log_dups_size': 0, 'ondisk_log_size': 6025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49347195, 'num_objects': 6024, 'num_object_clones': 0, 'num_object_copies': 24096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6024, 'num_write_kb': 48191, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3915}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:51.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5975", 'reported_seq': 6053, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:46.264424+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:46.264424+0000', 'last_peered': '2024-09-18T11:47:46.264424+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:46.264424+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5975, 'log_dups_size': 0, 'ondisk_log_size': 5975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48937595, 'num_objects': 5974, 'num_object_clones': 0, 'num_object_copies': 23896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5974, 'num_write_kb': 47791, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3920}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:51.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:51.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5983", 'reported_seq': 6034, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:45.223494+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:47:45.223494+0000', 'last_peered': '2024-09-18T11:47:45.223494+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:47:45.223494+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5983, 'log_dups_size': 0, 'ondisk_log_size': 5983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48977142, 'num_objects': 5979, 'num_object_clones': 0, 'num_object_copies': 23916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5981, 'num_write_kb': 47844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3821}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:51.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:51.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5874", 'reported_seq': 5938, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:46.579971+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:46.579971+0000', 'last_peered': '2024-09-18T11:47:46.579971+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:46.579971+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5874, 'log_dups_size': 0, 'ondisk_log_size': 5874, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48076022, 'num_objects': 5869, 'num_object_clones': 0, 'num_object_copies': 23476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5872, 'num_write_kb': 46964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3856}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:51.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:52.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 215 20 96874 96854 3.51855 1.99219 0.0838634 0.0443944 2024-09-18T11:47:52.275 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:52.275 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30763/386336 objects degraded (7.963%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":270,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":96584,"data_bytes":791176564,"bytes_used":2712887296,"bytes_avail":383834169344,"bytes_total":386547056640,"degraded_objects":30763,"degraded_total":386336,"degraded_ratio":0.079627578066760546,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":120114,"recovering_keys_per_sec":0,"num_objects_recovered":176,"num_bytes_recovered":1441792,"num_keys_recovered":0,"write_bytes_sec":2832240,"read_op_per_sec":0,"write_op_per_sec":345},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:52.296 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:52.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:52.846 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:52.847 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:52.866 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-18T11:47:51.906600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":791176564,"num_objects":96584,"num_object_clones":0,"num_object_copies":386336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96611,"num_write_kb":772770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2691,"num_bytes_recovered":21807104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96638,"ondisk_log_size":96638,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2649304,"kb_used_data":1421048,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":374838056,"statfs":{"total":386547056640,"available":383834169344,"internally_reserved":0,"allocated":1455153152,"data_stored":1454484412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,11,7,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33996800,"num_objects":4150,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4150,"num_write_kb":33200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003502"},"pg_stats":[{"pgid":"1.f","version":"33'6193","reported_seq":6236,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:50.291226+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:50.291226+0000","last_peered":"2024-09-18T11:47:50.291226+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:50.291226+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6193,"log_dups_size":0,"ondisk_log_size":6193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50733056,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6193,"num_write_kb":49544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4063}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6126","reported_seq":7334,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.760064+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:51.760064+0000","last_peered":"2024-09-18T11:47:51.760064+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:51.760064+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6126,"log_dups_size":0,"ondisk_log_size":6126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50148598,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6124,"num_write_kb":48988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5980","reported_seq":6063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.501292+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:51.501292+0000","last_peered":"2024-09-18T11:47:51.501292+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:51.501292+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48962171,"num_objects":5977,"num_object_clones":0,"num_object_copies":23908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":47822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3923}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6008","reported_seq":6072,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.670216+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:51.670216+0000","last_peered":"2024-09-18T11:47:51.670216+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:51.670216+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6008,"log_dups_size":0,"ondisk_log_size":6008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49181942,"num_objects":6004,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":48037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5998","reported_seq":6055,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:50.079913+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:50.079913+0000","last_peered":"2024-09-18T11:47:50.079913+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:50.079913+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5998,"log_dups_size":0,"ondisk_log_size":5998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49108214,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5996,"num_write_kb":47965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3913}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5975","reported_seq":6020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:50.051108+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:50.051108+0000","last_peered":"2024-09-18T11:47:50.051108+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:50.051108+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5975,"log_dups_size":0,"ondisk_log_size":5975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48947200,"num_objects":5975,"num_object_clones":0,"num_object_copies":23900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5975,"num_write_kb":47800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6108","reported_seq":6174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.787932+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:51.787932+0000","last_peered":"2024-09-18T11:47:51.787932+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:51.787932+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6108,"log_dups_size":0,"ondisk_log_size":6108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49984758,"num_objects":6102,"num_object_clones":0,"num_object_copies":24408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6106,"num_write_kb":48828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3968}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6019","reported_seq":6084,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.560385+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:51.560385+0000","last_peered":"2024-09-18T11:47:51.560385+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:51.560385+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263862,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":48124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6137","reported_seq":6197,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:50.286753+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:50.286753+0000","last_peered":"2024-09-18T11:47:50.286753+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:50.286753+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6137,"log_dups_size":0,"ondisk_log_size":6137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50203116,"num_objects":6129,"num_object_clones":0,"num_object_copies":24516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6134,"num_write_kb":49049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3994}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5884","reported_seq":5970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.437247+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:51.437247+0000","last_peered":"2024-09-18T11:47:51.437247+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:51.437247+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5884,"log_dups_size":0,"ondisk_log_size":5884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48182518,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":47054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6038","reported_seq":6094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:50.304200+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:50.304200+0000","last_peered":"2024-09-18T11:47:50.304200+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:50.304200+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49418097,"num_objects":6033,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6035,"num_write_kb":48268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6092","reported_seq":6131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:50.287206+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:50.287206+0000","last_peered":"2024-09-18T11:47:50.287206+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:50.287206+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6092,"log_dups_size":0,"ondisk_log_size":6092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49896059,"num_objects":6091,"num_object_clones":0,"num_object_copies":24364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":48727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6048","reported_seq":6124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.492804+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:51.492804+0000","last_peered":"2024-09-18T11:47:51.492804+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:51.492804+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6048,"log_dups_size":0,"ondisk_log_size":6048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49535611,"num_objects":6047,"num_object_clones":0,"num_object_copies":24188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6047,"num_write_kb":48375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6076","reported_seq":6127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:50.364386+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:50.364386+0000","last_peered":"2024-09-18T11:47:50.364386+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:50.364386+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49738998,"num_objects":6072,"num_object_clones":0,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6074,"num_write_kb":48588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3914}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6005","reported_seq":10290,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:50.169688+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:50.169688+0000","last_peered":"2024-09-18T11:47:50.169688+0000","last_clean":"2024-09-18T11:47:50.169688+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:50.169688+0000","last_undegraded":"2024-09-18T11:47:50.169688+0000","last_fullsized":"2024-09-18T11:47:50.169688+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6005,"log_dups_size":0,"ondisk_log_size":6005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49165558,"num_objects":6002,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6003,"num_write_kb":48021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5951","reported_seq":6015,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.735482+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:51.735482+0000","last_peered":"2024-09-18T11:47:51.735482+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:51.735482+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5951,"log_dups_size":0,"ondisk_log_size":5951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48706806,"num_objects":5946,"num_object_clones":0,"num_object_copies":23784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5949,"num_write_kb":47580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":791176564,"num_objects":96584,"num_object_clones":0,"num_object_copies":386336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96611,"num_write_kb":772770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2691,"num_bytes_recovered":21807104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1456209920,"data_stored":1456209920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96638,"ondisk_log_size":96638,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691012,"kb_used_data":384828,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93680828,"statfs":{"total":96636764160,"available":95929167872,"internally_reserved":0,"allocated":394063872,"data_stored":393896687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,6,0,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":0.499},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.51900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":3.2749999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.20499999999999999},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.46300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684340,"kb_used_data":386988,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93687500,"statfs":{"total":96636764160,"available":95936000000,"internally_reserved":0,"allocated":396275712,"data_stored":396108527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":11.986000000000001},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.2799999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.786},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":3.3439999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":5.742},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.085}]}]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577160,"kb_used_data":262720,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93794680,"statfs":{"total":96636764160,"available":96045752320,"internally_reserved":0,"allocated":269025280,"data_stored":268858095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.3239999999999998,"15min":3.3010000000000002},"min":{"1min":0.22600000000000001,"5min":0.159,"15min":0.159},"max":{"1min":10.48,"5min":13.808999999999999,"15min":13.808999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":3.2320000000000002,"5min":4.7279999999999998,"15min":4.9779999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":17.605,"5min":18.221,"15min":18.221},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":4.7830000000000004,"5min":4.1020000000000003,"15min":3.988},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":2.476},{"interface":"front","average":{"1min":5.585,"5min":4.1710000000000003,"15min":3.9350000000000001},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":26.719000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.30299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.9780000000000002,"15min":4.1139999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":14.185,"5min":14.185,"15min":14.185},"last":3.0720000000000001},{"interface":"front","average":{"1min":4.4989999999999997,"5min":4.5910000000000002,"15min":4.6059999999999999},"min":{"1min":0.31900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":3.169}]}]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696792,"kb_used_data":386512,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93675048,"statfs":{"total":96636764160,"available":95923249152,"internally_reserved":0,"allocated":395788288,"data_stored":395621103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.23400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.29999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.249},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":396308480,"data_stored":396308480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269033472,"data_stored":269033472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396062720,"data_stored":396062720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394805248,"data_stored":394805248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:53.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 216 20 97146 97126 3.51209 2.125 0.0696218 0.0444767 2024-09-18T11:47:54.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 217 20 97433 97413 3.50624 2.24219 0.0838475 0.0445493 2024-09-18T11:47:55.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 218 20 97724 97704 3.50059 2.27344 0.0659424 0.0446224 2024-09-18T11:47:55.868 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:56.261 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 219 20 98001 97981 3.49443 2.16406 0.0573397 0.0447024 2024-09-18T11:47:56.418 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:56.418 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:56.445 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-18T11:47:55.908314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":796755316,"num_objects":97265,"num_object_clones":0,"num_object_copies":389060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97292,"num_write_kb":778218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2691,"num_bytes_recovered":21807104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97319,"ondisk_log_size":97319,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2652096,"kb_used_data":1432160,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374835264,"statfs":{"total":386547056640,"available":383831310336,"internally_reserved":0,"allocated":1466531840,"data_stored":1465863100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,8,12,9,4,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28123136,"num_objects":3433,"num_object_clones":0,"num_object_copies":13732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3433,"num_write_kb":27464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004252"},"pg_stats":[{"pgid":"1.f","version":"33'6293","reported_seq":6336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.325538+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:55.325538+0000","last_peered":"2024-09-18T11:47:55.325538+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:55.325538+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51552256,"num_objects":6293,"num_object_clones":0,"num_object_copies":25172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6293,"num_write_kb":50344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6126","reported_seq":7334,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.760064+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:51.760064+0000","last_peered":"2024-09-18T11:47:51.760064+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:51.760064+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6126,"log_dups_size":0,"ondisk_log_size":6126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50148598,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6124,"num_write_kb":48988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5980","reported_seq":6063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.501292+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:51.501292+0000","last_peered":"2024-09-18T11:47:51.501292+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:51.501292+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48962171,"num_objects":5977,"num_object_clones":0,"num_object_copies":23908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":47822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3923}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6008","reported_seq":6072,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.670216+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:51.670216+0000","last_peered":"2024-09-18T11:47:51.670216+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:51.670216+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6008,"log_dups_size":0,"ondisk_log_size":6008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49181942,"num_objects":6004,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":48037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6087","reported_seq":6143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.300276+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:55.300276+0000","last_peered":"2024-09-18T11:47:55.300276+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:55.300276+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49837302,"num_objects":6084,"num_object_clones":0,"num_object_copies":24336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":48677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6065","reported_seq":6110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.370355+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:55.370355+0000","last_peered":"2024-09-18T11:47:55.370355+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:55.370355+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49684480,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6065,"num_write_kb":48520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6108","reported_seq":6174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.787932+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:51.787932+0000","last_peered":"2024-09-18T11:47:51.787932+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:51.787932+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6108,"log_dups_size":0,"ondisk_log_size":6108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49984758,"num_objects":6102,"num_object_clones":0,"num_object_copies":24408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6106,"num_write_kb":48828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3968}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6019","reported_seq":6084,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.560385+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:51.560385+0000","last_peered":"2024-09-18T11:47:51.560385+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:51.560385+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263862,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":48124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6209","reported_seq":6269,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.360617+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:55.360617+0000","last_peered":"2024-09-18T11:47:55.360617+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:55.360617+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50792940,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5884","reported_seq":5970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.437247+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:51.437247+0000","last_peered":"2024-09-18T11:47:51.437247+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:51.437247+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5884,"log_dups_size":0,"ondisk_log_size":5884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48182518,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":47054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6135","reported_seq":6194,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.233546+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:55.233546+0000","last_peered":"2024-09-18T11:47:55.233546+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:55.233546+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50212721,"num_objects":6130,"num_object_clones":0,"num_object_copies":24520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6132,"num_write_kb":49044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6164","reported_seq":6203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.187290+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:55.187290+0000","last_peered":"2024-09-18T11:47:55.187290+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:55.187290+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6164,"log_dups_size":0,"ondisk_log_size":6164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50485883,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":49303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4054}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6048","reported_seq":6124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.492804+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:51.492804+0000","last_peered":"2024-09-18T11:47:51.492804+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:51.492804+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6048,"log_dups_size":0,"ondisk_log_size":6048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49535611,"num_objects":6047,"num_object_clones":0,"num_object_copies":24188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6047,"num_write_kb":48375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6155","reported_seq":6206,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.273357+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:55.273357+0000","last_peered":"2024-09-18T11:47:55.273357+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:55.273357+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50386166,"num_objects":6151,"num_object_clones":0,"num_object_copies":24604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3993}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6087","reported_seq":10372,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:55.410139+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:55.410139+0000","last_peered":"2024-09-18T11:47:55.410139+0000","last_clean":"2024-09-18T11:47:55.410139+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:55.410139+0000","last_undegraded":"2024-09-18T11:47:55.410139+0000","last_fullsized":"2024-09-18T11:47:55.410139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49837302,"num_objects":6084,"num_object_clones":0,"num_object_copies":24336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":48677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5951","reported_seq":6015,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.735482+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:51.735482+0000","last_peered":"2024-09-18T11:47:51.735482+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:51.735482+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5951,"log_dups_size":0,"ondisk_log_size":5951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48706806,"num_objects":5946,"num_object_clones":0,"num_object_copies":23784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5949,"num_write_kb":47580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":796755316,"num_objects":97265,"num_object_clones":0,"num_object_copies":389060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97292,"num_write_kb":778218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2691,"num_bytes_recovered":21807104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1467793408,"data_stored":1467793408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97319,"ondisk_log_size":97319,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700656,"kb_used_data":390312,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93671184,"statfs":{"total":96636764160,"available":95919292416,"internally_reserved":0,"allocated":399679488,"data_stored":399512303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,3,1,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":2.9710000000000001},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.9980000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.0259999999999998},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":4.9329999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":29.087},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684340,"kb_used_data":386988,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93687500,"statfs":{"total":96636764160,"available":95936000000,"internally_reserved":0,"allocated":396275712,"data_stored":396108527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":11.986000000000001},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.2799999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.786},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":3.3439999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":5.742},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.085}]}]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570308,"kb_used_data":268348,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93801532,"statfs":{"total":96636764160,"available":96052768768,"internally_reserved":0,"allocated":274788352,"data_stored":274621167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,8,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.9079999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":2.8420000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":3.504},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.8820000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":5.8849999999999998},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.29299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696792,"kb_used_data":386512,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93675048,"statfs":{"total":96636764160,"available":95923249152,"internally_reserved":0,"allocated":395788288,"data_stored":395621103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.23400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.29999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.249},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":396308480,"data_stored":396308480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274984960,"data_stored":274984960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396062720,"data_stored":396062720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":400437248,"data_stored":400437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:56.447 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:56.981 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:56.982 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:57.003 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-18T11:47:55.908314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":796755316,"num_objects":97265,"num_object_clones":0,"num_object_copies":389060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97292,"num_write_kb":778218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2691,"num_bytes_recovered":21807104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97319,"ondisk_log_size":97319,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2652096,"kb_used_data":1432160,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374835264,"statfs":{"total":386547056640,"available":383831310336,"internally_reserved":0,"allocated":1466531840,"data_stored":1465863100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,8,12,9,4,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28123136,"num_objects":3433,"num_object_clones":0,"num_object_copies":13732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3433,"num_write_kb":27464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004252"},"pg_stats":[{"pgid":"1.f","version":"33'6293","reported_seq":6336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.325538+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:55.325538+0000","last_peered":"2024-09-18T11:47:55.325538+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:55.325538+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51552256,"num_objects":6293,"num_object_clones":0,"num_object_copies":25172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6293,"num_write_kb":50344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6126","reported_seq":7334,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.760064+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:51.760064+0000","last_peered":"2024-09-18T11:47:51.760064+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:51.760064+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6126,"log_dups_size":0,"ondisk_log_size":6126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50148598,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6124,"num_write_kb":48988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5980","reported_seq":6063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.501292+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:51.501292+0000","last_peered":"2024-09-18T11:47:51.501292+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:51.501292+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48962171,"num_objects":5977,"num_object_clones":0,"num_object_copies":23908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":47822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3923}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6008","reported_seq":6072,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.670216+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:51.670216+0000","last_peered":"2024-09-18T11:47:51.670216+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:51.670216+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6008,"log_dups_size":0,"ondisk_log_size":6008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49181942,"num_objects":6004,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":48037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6087","reported_seq":6143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.300276+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:55.300276+0000","last_peered":"2024-09-18T11:47:55.300276+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:55.300276+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49837302,"num_objects":6084,"num_object_clones":0,"num_object_copies":24336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":48677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6065","reported_seq":6110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.370355+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:55.370355+0000","last_peered":"2024-09-18T11:47:55.370355+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:55.370355+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49684480,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6065,"num_write_kb":48520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6108","reported_seq":6174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.787932+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:51.787932+0000","last_peered":"2024-09-18T11:47:51.787932+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:51.787932+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6108,"log_dups_size":0,"ondisk_log_size":6108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49984758,"num_objects":6102,"num_object_clones":0,"num_object_copies":24408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6106,"num_write_kb":48828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3968}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6019","reported_seq":6084,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.560385+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:51.560385+0000","last_peered":"2024-09-18T11:47:51.560385+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:51.560385+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263862,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":48124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6209","reported_seq":6269,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.360617+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:55.360617+0000","last_peered":"2024-09-18T11:47:55.360617+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:55.360617+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50792940,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5884","reported_seq":5970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.437247+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:51.437247+0000","last_peered":"2024-09-18T11:47:51.437247+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:51.437247+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5884,"log_dups_size":0,"ondisk_log_size":5884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48182518,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":47054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6135","reported_seq":6194,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.233546+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:55.233546+0000","last_peered":"2024-09-18T11:47:55.233546+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:55.233546+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50212721,"num_objects":6130,"num_object_clones":0,"num_object_copies":24520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6132,"num_write_kb":49044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6164","reported_seq":6203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.187290+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:55.187290+0000","last_peered":"2024-09-18T11:47:55.187290+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:55.187290+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6164,"log_dups_size":0,"ondisk_log_size":6164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50485883,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":49303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4054}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6048","reported_seq":6124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.492804+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:51.492804+0000","last_peered":"2024-09-18T11:47:51.492804+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:51.492804+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6048,"log_dups_size":0,"ondisk_log_size":6048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49535611,"num_objects":6047,"num_object_clones":0,"num_object_copies":24188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6047,"num_write_kb":48375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6155","reported_seq":6206,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.273357+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:55.273357+0000","last_peered":"2024-09-18T11:47:55.273357+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:55.273357+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50386166,"num_objects":6151,"num_object_clones":0,"num_object_copies":24604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3993}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6087","reported_seq":10372,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:55.410139+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:55.410139+0000","last_peered":"2024-09-18T11:47:55.410139+0000","last_clean":"2024-09-18T11:47:55.410139+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:55.410139+0000","last_undegraded":"2024-09-18T11:47:55.410139+0000","last_fullsized":"2024-09-18T11:47:55.410139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49837302,"num_objects":6084,"num_object_clones":0,"num_object_copies":24336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":48677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'5951","reported_seq":6015,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:51.735482+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:51.735482+0000","last_peered":"2024-09-18T11:47:51.735482+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:51.735482+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5951,"log_dups_size":0,"ondisk_log_size":5951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48706806,"num_objects":5946,"num_object_clones":0,"num_object_copies":23784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5949,"num_write_kb":47580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":796755316,"num_objects":97265,"num_object_clones":0,"num_object_copies":389060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97292,"num_write_kb":778218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2691,"num_bytes_recovered":21807104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1467793408,"data_stored":1467793408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97319,"ondisk_log_size":97319,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700656,"kb_used_data":390312,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93671184,"statfs":{"total":96636764160,"available":95919292416,"internally_reserved":0,"allocated":399679488,"data_stored":399512303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,3,1,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":2.9710000000000001},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.9980000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.0259999999999998},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":4.9329999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":29.087},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684340,"kb_used_data":386988,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93687500,"statfs":{"total":96636764160,"available":95936000000,"internally_reserved":0,"allocated":396275712,"data_stored":396108527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":11.986000000000001},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.2799999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.786},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":3.3439999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":5.742},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.085}]}]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570308,"kb_used_data":268348,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93801532,"statfs":{"total":96636764160,"available":96052768768,"internally_reserved":0,"allocated":274788352,"data_stored":274621167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,8,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.9079999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":2.8420000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":3.504},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.8820000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":5.8849999999999998},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.29299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696792,"kb_used_data":386512,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93675048,"statfs":{"total":96636764160,"available":95923249152,"internally_reserved":0,"allocated":395788288,"data_stored":395621103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:46:50 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":3.7170000000000001,"15min":3.6930000000000001},"min":{"1min":0.23599999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":9.5679999999999996,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":3.069,"5min":3.6230000000000002,"15min":3.7149999999999999},"min":{"1min":0.35099999999999998,"5min":0.185,"15min":0.185},"max":{"1min":7.75,"5min":8.6400000000000006,"15min":8.6400000000000006},"last":0.23400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.29999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.249},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":396308480,"data_stored":396308480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274984960,"data_stored":274984960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396062720,"data_stored":396062720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":400437248,"data_stored":400437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:57.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:47:57.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6293", 'reported_seq': 6336, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:55.325538+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:47:55.325538+0000', 'last_peered': '2024-09-18T11:47:55.325538+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:47:55.325538+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6293, 'log_dups_size': 0, 'ondisk_log_size': 6293, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51552256, 'num_objects': 6293, 'num_object_clones': 0, 'num_object_copies': 25172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6293, 'num_write_kb': 50344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4163}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:57.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:47:57.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6126", 'reported_seq': 7334, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:51.760064+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:47:51.760064+0000', 'last_peered': '2024-09-18T11:47:51.760064+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:47:51.760064+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6126, 'log_dups_size': 0, 'ondisk_log_size': 6126, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50148598, 'num_objects': 6122, 'num_object_clones': 0, 'num_object_copies': 24488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1531, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6124, 'num_write_kb': 48988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 573, 'num_bytes_recovered': 4677632, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:57.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:47:57.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5980", 'reported_seq': 6063, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:51.501292+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:47:51.501292+0000', 'last_peered': '2024-09-18T11:47:51.501292+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:47:51.501292+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5980, 'log_dups_size': 0, 'ondisk_log_size': 5980, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48962171, 'num_objects': 5977, 'num_object_clones': 0, 'num_object_copies': 23908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5979, 'num_write_kb': 47822, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3923}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:57.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:47:57.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6008", 'reported_seq': 6072, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:51.670216+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:47:51.670216+0000', 'last_peered': '2024-09-18T11:47:51.670216+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:47:51.670216+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6008, 'log_dups_size': 0, 'ondisk_log_size': 6008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49181942, 'num_objects': 6004, 'num_object_clones': 0, 'num_object_copies': 24016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6006, 'num_write_kb': 48037, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:57.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:47:57.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6087", 'reported_seq': 6143, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:55.300276+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:47:55.300276+0000', 'last_peered': '2024-09-18T11:47:55.300276+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:47:55.300276+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6087, 'log_dups_size': 0, 'ondisk_log_size': 6087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49837302, 'num_objects': 6084, 'num_object_clones': 0, 'num_object_copies': 24336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6085, 'num_write_kb': 48677, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4002}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:57.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:47:57.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6065", 'reported_seq': 6110, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:55.370355+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:47:55.370355+0000', 'last_peered': '2024-09-18T11:47:55.370355+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:47:55.370355+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6065, 'log_dups_size': 0, 'ondisk_log_size': 6065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49684480, 'num_objects': 6065, 'num_object_clones': 0, 'num_object_copies': 24260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6065, 'num_write_kb': 48520, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3911}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:57.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:47:57.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6108", 'reported_seq': 6174, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:51.787932+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:47:51.787932+0000', 'last_peered': '2024-09-18T11:47:51.787932+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:47:51.787932+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6108, 'log_dups_size': 0, 'ondisk_log_size': 6108, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49984758, 'num_objects': 6102, 'num_object_clones': 0, 'num_object_copies': 24408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6106, 'num_write_kb': 48828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3968}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:57.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:47:57.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6019", 'reported_seq': 6084, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:51.560385+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:47:51.560385+0000', 'last_peered': '2024-09-18T11:47:51.560385+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:47:51.560385+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6019, 'log_dups_size': 0, 'ondisk_log_size': 6019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49263862, 'num_objects': 6014, 'num_object_clones': 0, 'num_object_copies': 24056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6017, 'num_write_kb': 48124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:57.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:47:57.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6209", 'reported_seq': 6269, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:55.360617+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:47:55.360617+0000', 'last_peered': '2024-09-18T11:47:55.360617+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:47:55.360617+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6209, 'log_dups_size': 0, 'ondisk_log_size': 6209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50792940, 'num_objects': 6201, 'num_object_clones': 0, 'num_object_copies': 24804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6206, 'num_write_kb': 49625, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4066}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:57.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5884", 'reported_seq': 5970, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:51.437247+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:47:51.437247+0000', 'last_peered': '2024-09-18T11:47:51.437247+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:47:51.437247+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5884, 'log_dups_size': 0, 'ondisk_log_size': 5884, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48182518, 'num_objects': 5882, 'num_object_clones': 0, 'num_object_copies': 23528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5882, 'num_write_kb': 47054, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6135", 'reported_seq': 6194, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:55.233546+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:47:55.233546+0000', 'last_peered': '2024-09-18T11:47:55.233546+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:47:55.233546+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6135, 'log_dups_size': 0, 'ondisk_log_size': 6135, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50212721, 'num_objects': 6130, 'num_object_clones': 0, 'num_object_copies': 24520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6132, 'num_write_kb': 49044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4067}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6164", 'reported_seq': 6203, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:55.187290+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:47:55.187290+0000', 'last_peered': '2024-09-18T11:47:55.187290+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:47:55.187290+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6164, 'log_dups_size': 0, 'ondisk_log_size': 6164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50485883, 'num_objects': 6163, 'num_object_clones': 0, 'num_object_copies': 24652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6163, 'num_write_kb': 49303, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6048", 'reported_seq': 6124, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:51.492804+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:47:51.492804+0000', 'last_peered': '2024-09-18T11:47:51.492804+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:47:51.492804+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6048, 'log_dups_size': 0, 'ondisk_log_size': 6048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49535611, 'num_objects': 6047, 'num_object_clones': 0, 'num_object_copies': 24188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6047, 'num_write_kb': 48375, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6155", 'reported_seq': 6206, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:55.273357+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:47:55.273357+0000', 'last_peered': '2024-09-18T11:47:55.273357+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:47:55.273357+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6155, 'log_dups_size': 0, 'ondisk_log_size': 6155, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50386166, 'num_objects': 6151, 'num_object_clones': 0, 'num_object_copies': 24604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6153, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:47:57.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5951", 'reported_seq': 6015, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:47:51.735482+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:47:51.735482+0000', 'last_peered': '2024-09-18T11:47:51.735482+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:47:51.735482+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 5951, 'log_dups_size': 0, 'ondisk_log_size': 5951, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48706806, 'num_objects': 5946, 'num_object_clones': 0, 'num_object_copies': 23784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5949, 'num_write_kb': 47580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3933}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:47:57.008 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:47:57.261 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:2024-09-18T11:47:57.260844+0000 min lat: 0.00272854 max lat: 2.42146 avg lat: 0.0447947 2024-09-18T11:47:57.261 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:47:57.261 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 220 20 98246 98226 3.48725 1.91406 0.109479 0.0447947 2024-09-18T11:47:57.656 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:57.656 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30763/389060 objects degraded (7.907%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":275,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":97265,"data_bytes":796755316,"bytes_used":2715746304,"bytes_avail":383831310336,"bytes_total":386547056640,"degraded_objects":30763,"degraded_total":389060,"degraded_ratio":0.079070066313679119,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":85303,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"write_bytes_sec":2342764,"read_op_per_sec":0,"write_op_per_sec":285},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:47:57.677 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:47:57.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:47:58.261 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 221 20 98489 98469 3.48006 1.89844 0.123394 0.0448807 2024-09-18T11:47:58.289 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:47:58.289 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:47:58.320 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-18T11:47:57.908964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":802276724,"num_objects":97939,"num_object_clones":0,"num_object_copies":391756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97966,"num_write_kb":783610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2739,"num_bytes_recovered":22200320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97993,"ondisk_log_size":97993,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2654892,"kb_used_data":1443276,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374832468,"statfs":{"total":386547056640,"available":383828447232,"internally_reserved":0,"allocated":1477914624,"data_stored":1477245884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,10,17,8,5,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27885568,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004317"},"pg_stats":[{"pgid":"1.f","version":"33'6293","reported_seq":6336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.325538+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:55.325538+0000","last_peered":"2024-09-18T11:47:55.325538+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:55.325538+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51552256,"num_objects":6293,"num_object_clones":0,"num_object_copies":25172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6293,"num_write_kb":50344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6208","reported_seq":7512,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.810974+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:56.810974+0000","last_peered":"2024-09-18T11:47:56.810974+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:56.810974+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50820342,"num_objects":6204,"num_object_clones":0,"num_object_copies":24816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":5070848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6057","reported_seq":6141,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.438512+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:56.438512+0000","last_peered":"2024-09-18T11:47:56.438512+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:56.438512+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6057,"log_dups_size":0,"ondisk_log_size":6057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49592955,"num_objects":6054,"num_object_clones":0,"num_object_copies":24216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6056,"num_write_kb":48438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4000}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6104","reported_seq":6168,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.766019+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:56.766019+0000","last_peered":"2024-09-18T11:47:56.766019+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:56.766019+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6104,"log_dups_size":0,"ondisk_log_size":6104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49968374,"num_objects":6100,"num_object_clones":0,"num_object_copies":24400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6102,"num_write_kb":48805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3986}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6087","reported_seq":6143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.300276+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:55.300276+0000","last_peered":"2024-09-18T11:47:55.300276+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:55.300276+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49837302,"num_objects":6084,"num_object_clones":0,"num_object_copies":24336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":48677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6065","reported_seq":6110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.370355+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:55.370355+0000","last_peered":"2024-09-18T11:47:55.370355+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:55.370355+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49684480,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6065,"num_write_kb":48520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6198","reported_seq":6264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.700271+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:56.700271+0000","last_peered":"2024-09-18T11:47:56.700271+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:56.700271+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50722038,"num_objects":6192,"num_object_clones":0,"num_object_copies":24768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4058}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6098","reported_seq":6163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.776283+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:56.776283+0000","last_peered":"2024-09-18T11:47:56.776283+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:56.776283+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6098,"log_dups_size":0,"ondisk_log_size":6098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49911030,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6096,"num_write_kb":48756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6209","reported_seq":6269,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.360617+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:55.360617+0000","last_peered":"2024-09-18T11:47:55.360617+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:55.360617+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50792940,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5970","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.311559+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:56.311559+0000","last_peered":"2024-09-18T11:47:56.311559+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:56.311559+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5970,"log_dups_size":0,"ondisk_log_size":5970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48887030,"num_objects":5968,"num_object_clones":0,"num_object_copies":23872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5968,"num_write_kb":47742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6135","reported_seq":6194,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.233546+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:55.233546+0000","last_peered":"2024-09-18T11:47:55.233546+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:55.233546+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50212721,"num_objects":6130,"num_object_clones":0,"num_object_copies":24520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6132,"num_write_kb":49044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6164","reported_seq":6203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.187290+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:55.187290+0000","last_peered":"2024-09-18T11:47:55.187290+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:55.187290+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6164,"log_dups_size":0,"ondisk_log_size":6164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50485883,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":49303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4054}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6123","reported_seq":6201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.467835+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:56.467835+0000","last_peered":"2024-09-18T11:47:56.467835+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:56.467835+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50150011,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6122,"num_write_kb":48975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4068}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6155","reported_seq":6206,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.273357+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:55.273357+0000","last_peered":"2024-09-18T11:47:55.273357+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:55.273357+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50386166,"num_objects":6151,"num_object_clones":0,"num_object_copies":24604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3993}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6087","reported_seq":10372,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:55.410139+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:55.410139+0000","last_peered":"2024-09-18T11:47:55.410139+0000","last_clean":"2024-09-18T11:47:55.410139+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:55.410139+0000","last_undegraded":"2024-09-18T11:47:55.410139+0000","last_fullsized":"2024-09-18T11:47:55.410139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49837302,"num_objects":6084,"num_object_clones":0,"num_object_copies":24336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":48677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6040","reported_seq":6104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.777142+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:56.777142+0000","last_peered":"2024-09-18T11:47:56.777142+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:56.777142+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49435894,"num_objects":6035,"num_object_clones":0,"num_object_copies":24140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":48292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":802276724,"num_objects":97939,"num_object_clones":0,"num_object_copies":391756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97966,"num_write_kb":783610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2739,"num_bytes_recovered":22200320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1479073792,"data_stored":1479073792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97993,"ondisk_log_size":97993,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700656,"kb_used_data":390312,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93671184,"statfs":{"total":96636764160,"available":95919292416,"internally_reserved":0,"allocated":399679488,"data_stored":399512303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,3,1,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":2.9710000000000001},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.9980000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.0259999999999998},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":4.9329999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":29.087},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694032,"kb_used_data":392520,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93677808,"statfs":{"total":96636764160,"available":95926075392,"internally_reserved":0,"allocated":401940480,"data_stored":401773295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":19.923999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.0519999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":5.008},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":5.3600000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":3.988},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.28999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570308,"kb_used_data":268348,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93801532,"statfs":{"total":96636764160,"available":96052768768,"internally_reserved":0,"allocated":274788352,"data_stored":274621167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,8,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.9079999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":2.8420000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":3.504},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.8820000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":5.8849999999999998},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.29299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689896,"kb_used_data":392096,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93681944,"statfs":{"total":96636764160,"available":95930310656,"internally_reserved":0,"allocated":401506304,"data_stored":401339119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.0819999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":6.2489999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401956864,"data_stored":401956864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274984960,"data_stored":274984960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401694720,"data_stored":401694720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":400437248,"data_stored":400437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:47:59.261 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 222 20 98742 98722 3.47329 1.97656 0.137306 0.0449711 2024-09-18T11:48:00.266 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 223 20 98999 98979 3.46669 2.00781 0.0340999 0.0450584 2024-09-18T11:48:01.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 224 20 99273 99253 3.46077 2.14062 0.163132 0.0451367 2024-09-18T11:48:01.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:01.900 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:01.901 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:01.926 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-18T11:47:59.909719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":802276724,"num_objects":97939,"num_object_clones":0,"num_object_copies":391756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97966,"num_write_kb":783610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2739,"num_bytes_recovered":22200320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97993,"ondisk_log_size":97993,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2654892,"kb_used_data":1443276,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374832468,"statfs":{"total":386547056640,"available":383828447232,"internally_reserved":0,"allocated":1477914624,"data_stored":1477245884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,10,17,8,5,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22044672,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2691,"num_write_kb":21528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004429"},"pg_stats":[{"pgid":"1.f","version":"33'6293","reported_seq":6336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.325538+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:47:55.325538+0000","last_peered":"2024-09-18T11:47:55.325538+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:47:55.325538+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51552256,"num_objects":6293,"num_object_clones":0,"num_object_copies":25172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6293,"num_write_kb":50344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6208","reported_seq":7512,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.810974+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:47:56.810974+0000","last_peered":"2024-09-18T11:47:56.810974+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:47:56.810974+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50820342,"num_objects":6204,"num_object_clones":0,"num_object_copies":24816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":5070848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6057","reported_seq":6141,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.438512+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:47:56.438512+0000","last_peered":"2024-09-18T11:47:56.438512+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:47:56.438512+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6057,"log_dups_size":0,"ondisk_log_size":6057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49592955,"num_objects":6054,"num_object_clones":0,"num_object_copies":24216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6056,"num_write_kb":48438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4000}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6104","reported_seq":6168,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.766019+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:47:56.766019+0000","last_peered":"2024-09-18T11:47:56.766019+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:47:56.766019+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6104,"log_dups_size":0,"ondisk_log_size":6104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49968374,"num_objects":6100,"num_object_clones":0,"num_object_copies":24400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6102,"num_write_kb":48805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3986}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6087","reported_seq":6143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.300276+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:47:55.300276+0000","last_peered":"2024-09-18T11:47:55.300276+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:47:55.300276+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49837302,"num_objects":6084,"num_object_clones":0,"num_object_copies":24336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":48677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6065","reported_seq":6110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.370355+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:47:55.370355+0000","last_peered":"2024-09-18T11:47:55.370355+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:47:55.370355+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49684480,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6065,"num_write_kb":48520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6198","reported_seq":6264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.700271+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:47:56.700271+0000","last_peered":"2024-09-18T11:47:56.700271+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:47:56.700271+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50722038,"num_objects":6192,"num_object_clones":0,"num_object_copies":24768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4058}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6098","reported_seq":6163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.776283+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:47:56.776283+0000","last_peered":"2024-09-18T11:47:56.776283+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:47:56.776283+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6098,"log_dups_size":0,"ondisk_log_size":6098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49911030,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6096,"num_write_kb":48756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6209","reported_seq":6269,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.360617+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:47:55.360617+0000","last_peered":"2024-09-18T11:47:55.360617+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:47:55.360617+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50792940,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5970","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.311559+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:47:56.311559+0000","last_peered":"2024-09-18T11:47:56.311559+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:47:56.311559+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":5970,"log_dups_size":0,"ondisk_log_size":5970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48887030,"num_objects":5968,"num_object_clones":0,"num_object_copies":23872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5968,"num_write_kb":47742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6135","reported_seq":6194,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.233546+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:47:55.233546+0000","last_peered":"2024-09-18T11:47:55.233546+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:47:55.233546+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50212721,"num_objects":6130,"num_object_clones":0,"num_object_copies":24520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6132,"num_write_kb":49044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6164","reported_seq":6203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.187290+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:47:55.187290+0000","last_peered":"2024-09-18T11:47:55.187290+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:47:55.187290+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6164,"log_dups_size":0,"ondisk_log_size":6164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50485883,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":49303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4054}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6123","reported_seq":6201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.467835+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:47:56.467835+0000","last_peered":"2024-09-18T11:47:56.467835+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:47:56.467835+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50150011,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6122,"num_write_kb":48975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4068}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6155","reported_seq":6206,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:55.273357+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:47:55.273357+0000","last_peered":"2024-09-18T11:47:55.273357+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:47:55.273357+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50386166,"num_objects":6151,"num_object_clones":0,"num_object_copies":24604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3993}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6087","reported_seq":10372,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:47:55.410139+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:47:55.410139+0000","last_peered":"2024-09-18T11:47:55.410139+0000","last_clean":"2024-09-18T11:47:55.410139+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:47:55.410139+0000","last_undegraded":"2024-09-18T11:47:55.410139+0000","last_fullsized":"2024-09-18T11:47:55.410139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49837302,"num_objects":6084,"num_object_clones":0,"num_object_copies":24336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":48677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6040","reported_seq":6104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:47:56.777142+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:47:56.777142+0000","last_peered":"2024-09-18T11:47:56.777142+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:47:56.777142+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49435894,"num_objects":6035,"num_object_clones":0,"num_object_copies":24140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":48292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":802276724,"num_objects":97939,"num_object_clones":0,"num_object_copies":391756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97966,"num_write_kb":783610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2739,"num_bytes_recovered":22200320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1479073792,"data_stored":1479073792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97993,"ondisk_log_size":97993,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700656,"kb_used_data":390312,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93671184,"statfs":{"total":96636764160,"available":95919292416,"internally_reserved":0,"allocated":399679488,"data_stored":399512303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,3,1,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":2.9710000000000001},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.9980000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.0259999999999998},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":4.9329999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":29.087},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694032,"kb_used_data":392520,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93677808,"statfs":{"total":96636764160,"available":95926075392,"internally_reserved":0,"allocated":401940480,"data_stored":401773295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:46:58 2024","interfaces":[{"interface":"back","average":{"1min":5.0800000000000001,"5min":2.7490000000000001,"15min":1.9339999999999999},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":15.637,"5min":15.637,"15min":15.637},"last":19.923999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":2.2949999999999999,"15min":1.7070000000000001},"min":{"1min":0.313,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":14.537000000000001,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.0519999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":5.008},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":5.3600000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":3.988},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.28999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570308,"kb_used_data":268348,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93801532,"statfs":{"total":96636764160,"available":96052768768,"internally_reserved":0,"allocated":274788352,"data_stored":274621167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,8,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.9079999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":2.8420000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":3.504},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.8820000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":5.8849999999999998},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.29299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689896,"kb_used_data":392096,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93681944,"statfs":{"total":96636764160,"available":95930310656,"internally_reserved":0,"allocated":401506304,"data_stored":401339119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.0819999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":6.2489999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401956864,"data_stored":401956864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274984960,"data_stored":274984960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401694720,"data_stored":401694720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":400437248,"data_stored":400437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:01.928 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:02.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 225 20 99521 99501 3.454 1.9375 0.0455948 0.045223 2024-09-18T11:48:02.484 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:02.485 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:02.508 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-09-18T11:48:01.910455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":812877172,"num_objects":99233,"num_object_clones":0,"num_object_copies":396932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99260,"num_write_kb":793962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2789,"num_bytes_recovered":22609920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99287,"ondisk_log_size":99287,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670248,"kb_used_data":1462792,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":374817112,"statfs":{"total":386547056640,"available":383812722688,"internally_reserved":0,"allocated":1497899008,"data_stored":1497230268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,8,10,25,18,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32645120,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":31880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004667"},"pg_stats":[{"pgid":"1.f","version":"33'6357","reported_seq":6400,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.292982+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:00.292982+0000","last_peered":"2024-09-18T11:48:00.292982+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:00.292982+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6357,"log_dups_size":0,"ondisk_log_size":6357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52076544,"num_objects":6357,"num_object_clones":0,"num_object_copies":25428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6357,"num_write_kb":50856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6279","reported_seq":7683,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.704434+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:01.704434+0000","last_peered":"2024-09-18T11:48:01.704434+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:01.704434+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51401974,"num_objects":6275,"num_object_clones":0,"num_object_copies":25100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6277,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":671,"num_bytes_recovered":5480448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6125","reported_seq":6209,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.560212+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:01.560212+0000","last_peered":"2024-09-18T11:48:01.560212+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:01.560212+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50150011,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6124,"num_write_kb":48982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4068}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6201","reported_seq":6265,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.701597+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:01.701597+0000","last_peered":"2024-09-18T11:48:01.701597+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:01.701597+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6201,"log_dups_size":0,"ondisk_log_size":6201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50762998,"num_objects":6197,"num_object_clones":0,"num_object_copies":24788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":49581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6166","reported_seq":6223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.406266+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:00.406266+0000","last_peered":"2024-09-18T11:48:00.406266+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:00.406266+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50484470,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6164,"num_write_kb":49309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6148","reported_seq":6193,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.410205+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:00.410205+0000","last_peered":"2024-09-18T11:48:00.410205+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:00.410205+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50364416,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":49184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3994}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6282","reported_seq":6349,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.818104+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:01.818104+0000","last_peered":"2024-09-18T11:48:01.818104+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:01.818104+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51410166,"num_objects":6276,"num_object_clones":0,"num_object_copies":25104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":50220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4142}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6179","reported_seq":6244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.735289+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:01.735289+0000","last_peered":"2024-09-18T11:48:01.735289+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:01.735289+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50574582,"num_objects":6174,"num_object_clones":0,"num_object_copies":24696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":49404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6294","reported_seq":6354,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.417166+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:00.417166+0000","last_peered":"2024-09-18T11:48:00.417166+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:00.417166+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6294,"log_dups_size":0,"ondisk_log_size":6294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51489260,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":50305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4151}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6045","reported_seq":6131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.520097+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:01.520097+0000","last_peered":"2024-09-18T11:48:01.520097+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:01.520097+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6045,"log_dups_size":0,"ondisk_log_size":6045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49501430,"num_objects":6043,"num_object_clones":0,"num_object_copies":24172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6043,"num_write_kb":48342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4068}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6221","reported_seq":6278,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.184237+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:00.184237+0000","last_peered":"2024-09-18T11:48:00.184237+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:00.184237+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50917233,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6218,"num_write_kb":49732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6245","reported_seq":6284,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.162324+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:00.162324+0000","last_peered":"2024-09-18T11:48:00.162324+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:00.162324+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51149435,"num_objects":6244,"num_object_clones":0,"num_object_copies":24976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6244,"num_write_kb":49951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4135}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6207","reported_seq":6284,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.476467+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:01.476467+0000","last_peered":"2024-09-18T11:48:01.476467+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:01.476467+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50838139,"num_objects":6206,"num_object_clones":0,"num_object_copies":24824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4152}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6233","reported_seq":6284,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.437222+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:00.437222+0000","last_peered":"2024-09-18T11:48:00.437222+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:00.437222+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6233,"log_dups_size":0,"ondisk_log_size":6233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51025142,"num_objects":6229,"num_object_clones":0,"num_object_copies":24916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6231,"num_write_kb":49844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4071}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6182","reported_seq":10467,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:00.532104+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:00.532104+0000","last_peered":"2024-09-18T11:48:00.532104+0000","last_clean":"2024-09-18T11:48:00.532104+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:00.532104+0000","last_undegraded":"2024-09-18T11:48:00.532104+0000","last_fullsized":"2024-09-18T11:48:00.532104+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50615542,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6180,"num_write_kb":49437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6123","reported_seq":6187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.763760+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:01.763760+0000","last_peered":"2024-09-18T11:48:01.763760+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:01.763760+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115830,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":48956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":812877172,"num_objects":99233,"num_object_clones":0,"num_object_copies":396932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99260,"num_write_kb":793962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2789,"num_bytes_recovered":22609920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1500352512,"data_stored":1500352512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99287,"ondisk_log_size":99287,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693308,"kb_used_data":395444,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93678532,"statfs":{"total":96636764160,"available":95926816768,"internally_reserved":0,"allocated":404934656,"data_stored":404767471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,6,4,4,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":0.42399999999999999},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.45400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":4.4740000000000002},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":4.4299999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":3.5329999999999999},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.437}]}]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702232,"kb_used_data":396560,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93669608,"statfs":{"total":96636764160,"available":95917678592,"internally_reserved":0,"allocated":406077440,"data_stored":405910255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,1,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":2.911},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.8879999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":3.9039999999999999},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":3.9239999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":3.363},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":2.8479999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575552,"kb_used_data":273592,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93796288,"statfs":{"total":96636764160,"available":96047398912,"internally_reserved":0,"allocated":280158208,"data_stored":279991023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.907},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.753}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":8.4550000000000001},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.2750000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":4.3099999999999996},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.9269999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699156,"kb_used_data":397196,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93672684,"statfs":{"total":96636764160,"available":95920828416,"internally_reserved":0,"allocated":406728704,"data_stored":406561519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,11,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.248},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":14.557}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":15.103999999999999},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":14.586}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":4.0979999999999999},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":14.516999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":407232512,"data_stored":407232512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280465408,"data_stored":280465408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":406966272,"data_stored":406966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405688320,"data_stored":405688320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:02.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:02.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6357", 'reported_seq': 6400, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:00.292982+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:00.292982+0000', 'last_peered': '2024-09-18T11:48:00.292982+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:00.292982+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6357, 'log_dups_size': 0, 'ondisk_log_size': 6357, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52076544, 'num_objects': 6357, 'num_object_clones': 0, 'num_object_copies': 25428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6357, 'num_write_kb': 50856, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4227}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:02.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:02.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6279", 'reported_seq': 7683, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:01.704434+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:01.704434+0000', 'last_peered': '2024-09-18T11:48:01.704434+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:01.704434+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6279, 'log_dups_size': 0, 'ondisk_log_size': 6279, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51401974, 'num_objects': 6275, 'num_object_clones': 0, 'num_object_copies': 25100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1433, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6277, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 671, 'num_bytes_recovered': 5480448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4842}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:02.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6125", 'reported_seq': 6209, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:01.560212+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:01.560212+0000', 'last_peered': '2024-09-18T11:48:01.560212+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:01.560212+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6125, 'log_dups_size': 0, 'ondisk_log_size': 6125, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50150011, 'num_objects': 6122, 'num_object_clones': 0, 'num_object_copies': 24488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6124, 'num_write_kb': 48982, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4068}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6201", 'reported_seq': 6265, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:01.701597+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:01.701597+0000', 'last_peered': '2024-09-18T11:48:01.701597+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:01.701597+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6201, 'log_dups_size': 0, 'ondisk_log_size': 6201, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50762998, 'num_objects': 6197, 'num_object_clones': 0, 'num_object_copies': 24788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6199, 'num_write_kb': 49581, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6166", 'reported_seq': 6223, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:00.406266+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:00.406266+0000', 'last_peered': '2024-09-18T11:48:00.406266+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:00.406266+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6166, 'log_dups_size': 0, 'ondisk_log_size': 6166, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50484470, 'num_objects': 6163, 'num_object_clones': 0, 'num_object_copies': 24652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6164, 'num_write_kb': 49309, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4081}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6148", 'reported_seq': 6193, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:00.410205+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:00.410205+0000', 'last_peered': '2024-09-18T11:48:00.410205+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:00.410205+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6148, 'log_dups_size': 0, 'ondisk_log_size': 6148, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50364416, 'num_objects': 6148, 'num_object_clones': 0, 'num_object_copies': 24592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6148, 'num_write_kb': 49184, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3994}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6282", 'reported_seq': 6349, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:01.818104+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:01.818104+0000', 'last_peered': '2024-09-18T11:48:01.818104+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:01.818104+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6282, 'log_dups_size': 0, 'ondisk_log_size': 6282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51410166, 'num_objects': 6276, 'num_object_clones': 0, 'num_object_copies': 25104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6280, 'num_write_kb': 50220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4142}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:02.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6179", 'reported_seq': 6244, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:01.735289+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:01.735289+0000', 'last_peered': '2024-09-18T11:48:01.735289+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:01.735289+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6179, 'log_dups_size': 0, 'ondisk_log_size': 6179, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50574582, 'num_objects': 6174, 'num_object_clones': 0, 'num_object_copies': 24696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6177, 'num_write_kb': 49404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:02.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:02.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6294", 'reported_seq': 6354, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:00.417166+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:00.417166+0000', 'last_peered': '2024-09-18T11:48:00.417166+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:00.417166+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6294, 'log_dups_size': 0, 'ondisk_log_size': 6294, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51489260, 'num_objects': 6286, 'num_object_clones': 0, 'num_object_copies': 25144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6291, 'num_write_kb': 50305, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4151}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:02.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:02.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6045", 'reported_seq': 6131, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:01.520097+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:01.520097+0000', 'last_peered': '2024-09-18T11:48:01.520097+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:01.520097+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6045, 'log_dups_size': 0, 'ondisk_log_size': 6045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49501430, 'num_objects': 6043, 'num_object_clones': 0, 'num_object_copies': 24172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6043, 'num_write_kb': 48342, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4068}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:02.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:02.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6221", 'reported_seq': 6278, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:00.184237+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:00.184237+0000', 'last_peered': '2024-09-18T11:48:00.184237+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:00.184237+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6221, 'log_dups_size': 0, 'ondisk_log_size': 6221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50917233, 'num_objects': 6216, 'num_object_clones': 0, 'num_object_copies': 24864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6218, 'num_write_kb': 49732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4153}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:02.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:02.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6245", 'reported_seq': 6284, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:00.162324+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:00.162324+0000', 'last_peered': '2024-09-18T11:48:00.162324+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:00.162324+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6245, 'log_dups_size': 0, 'ondisk_log_size': 6245, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51149435, 'num_objects': 6244, 'num_object_clones': 0, 'num_object_copies': 24976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6244, 'num_write_kb': 49951, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4135}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:02.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:02.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6207", 'reported_seq': 6284, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:01.476467+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:01.476467+0000', 'last_peered': '2024-09-18T11:48:01.476467+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:01.476467+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6207, 'log_dups_size': 0, 'ondisk_log_size': 6207, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50838139, 'num_objects': 6206, 'num_object_clones': 0, 'num_object_copies': 24824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6206, 'num_write_kb': 49647, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4152}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:02.512 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:02.512 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6233", 'reported_seq': 6284, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:00.437222+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:00.437222+0000', 'last_peered': '2024-09-18T11:48:00.437222+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:00.437222+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6233, 'log_dups_size': 0, 'ondisk_log_size': 6233, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51025142, 'num_objects': 6229, 'num_object_clones': 0, 'num_object_copies': 24916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6231, 'num_write_kb': 49844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4071}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:02.512 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:02.512 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6123", 'reported_seq': 6187, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:01.763760+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:01.763760+0000', 'last_peered': '2024-09-18T11:48:01.763760+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:01.763760+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6123, 'log_dups_size': 0, 'ondisk_log_size': 6123, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50115830, 'num_objects': 6118, 'num_object_clones': 0, 'num_object_copies': 24472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6121, 'num_write_kb': 48956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:02.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:03.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 226 20 99777 99757 3.44757 2 0.0546781 0.0453107 2024-09-18T11:48:03.277 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:03.277 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30665/396932 objects degraded (7.726%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":281,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":99233,"data_bytes":812877172,"bytes_used":2734333952,"bytes_avail":383812722688,"bytes_total":386547056640,"degraded_objects":30665,"degraded_total":396932,"degraded_ratio":0.077255046204387653,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":117373,"recovering_keys_per_sec":0,"num_objects_recovered":172,"num_bytes_recovered":1409024,"num_keys_recovered":0,"write_bytes_sec":2719368,"read_op_per_sec":0,"write_op_per_sec":331},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:03.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:03.900 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:03.900 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:03.920 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-09-18T11:48:01.910455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":812877172,"num_objects":99233,"num_object_clones":0,"num_object_copies":396932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99260,"num_write_kb":793962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2789,"num_bytes_recovered":22609920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99287,"ondisk_log_size":99287,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670248,"kb_used_data":1462792,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":374817112,"statfs":{"total":386547056640,"available":383812722688,"internally_reserved":0,"allocated":1497899008,"data_stored":1497230268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,8,10,25,18,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32645120,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":31880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004667"},"pg_stats":[{"pgid":"1.f","version":"33'6357","reported_seq":6400,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.292982+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:00.292982+0000","last_peered":"2024-09-18T11:48:00.292982+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:00.292982+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6357,"log_dups_size":0,"ondisk_log_size":6357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52076544,"num_objects":6357,"num_object_clones":0,"num_object_copies":25428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6357,"num_write_kb":50856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6279","reported_seq":7683,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.704434+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:01.704434+0000","last_peered":"2024-09-18T11:48:01.704434+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:01.704434+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51401974,"num_objects":6275,"num_object_clones":0,"num_object_copies":25100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6277,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":671,"num_bytes_recovered":5480448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6125","reported_seq":6209,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.560212+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:01.560212+0000","last_peered":"2024-09-18T11:48:01.560212+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:01.560212+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50150011,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6124,"num_write_kb":48982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4068}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6201","reported_seq":6265,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.701597+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:01.701597+0000","last_peered":"2024-09-18T11:48:01.701597+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:01.701597+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6201,"log_dups_size":0,"ondisk_log_size":6201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50762998,"num_objects":6197,"num_object_clones":0,"num_object_copies":24788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":49581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6166","reported_seq":6223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.406266+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:00.406266+0000","last_peered":"2024-09-18T11:48:00.406266+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:00.406266+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50484470,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6164,"num_write_kb":49309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6148","reported_seq":6193,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.410205+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:00.410205+0000","last_peered":"2024-09-18T11:48:00.410205+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:00.410205+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50364416,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":49184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3994}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6282","reported_seq":6349,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.818104+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:01.818104+0000","last_peered":"2024-09-18T11:48:01.818104+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:01.818104+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51410166,"num_objects":6276,"num_object_clones":0,"num_object_copies":25104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":50220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4142}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6179","reported_seq":6244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.735289+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:01.735289+0000","last_peered":"2024-09-18T11:48:01.735289+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:01.735289+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50574582,"num_objects":6174,"num_object_clones":0,"num_object_copies":24696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":49404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6294","reported_seq":6354,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.417166+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:00.417166+0000","last_peered":"2024-09-18T11:48:00.417166+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:00.417166+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6294,"log_dups_size":0,"ondisk_log_size":6294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51489260,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":50305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4151}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6045","reported_seq":6131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.520097+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:01.520097+0000","last_peered":"2024-09-18T11:48:01.520097+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:01.520097+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6045,"log_dups_size":0,"ondisk_log_size":6045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49501430,"num_objects":6043,"num_object_clones":0,"num_object_copies":24172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6043,"num_write_kb":48342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4068}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6221","reported_seq":6278,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.184237+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:00.184237+0000","last_peered":"2024-09-18T11:48:00.184237+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:00.184237+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50917233,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6218,"num_write_kb":49732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6245","reported_seq":6284,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.162324+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:00.162324+0000","last_peered":"2024-09-18T11:48:00.162324+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:00.162324+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51149435,"num_objects":6244,"num_object_clones":0,"num_object_copies":24976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6244,"num_write_kb":49951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4135}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6207","reported_seq":6284,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.476467+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:01.476467+0000","last_peered":"2024-09-18T11:48:01.476467+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:01.476467+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50838139,"num_objects":6206,"num_object_clones":0,"num_object_copies":24824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4152}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6233","reported_seq":6284,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:00.437222+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:00.437222+0000","last_peered":"2024-09-18T11:48:00.437222+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:00.437222+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6233,"log_dups_size":0,"ondisk_log_size":6233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51025142,"num_objects":6229,"num_object_clones":0,"num_object_copies":24916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6231,"num_write_kb":49844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4071}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6182","reported_seq":10467,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:00.532104+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:00.532104+0000","last_peered":"2024-09-18T11:48:00.532104+0000","last_clean":"2024-09-18T11:48:00.532104+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:00.532104+0000","last_undegraded":"2024-09-18T11:48:00.532104+0000","last_fullsized":"2024-09-18T11:48:00.532104+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50615542,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6180,"num_write_kb":49437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6123","reported_seq":6187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.763760+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:01.763760+0000","last_peered":"2024-09-18T11:48:01.763760+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:01.763760+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115830,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":48956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":812877172,"num_objects":99233,"num_object_clones":0,"num_object_copies":396932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99260,"num_write_kb":793962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2789,"num_bytes_recovered":22609920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1500352512,"data_stored":1500352512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99287,"ondisk_log_size":99287,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693308,"kb_used_data":395444,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93678532,"statfs":{"total":96636764160,"available":95926816768,"internally_reserved":0,"allocated":404934656,"data_stored":404767471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,6,4,4,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":0.42399999999999999},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.45400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":4.4740000000000002},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":4.4299999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":3.5329999999999999},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.437}]}]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702232,"kb_used_data":396560,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93669608,"statfs":{"total":96636764160,"available":95917678592,"internally_reserved":0,"allocated":406077440,"data_stored":405910255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,1,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":2.911},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.8879999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":3.9039999999999999},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":3.9239999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":3.363},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":2.8479999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575552,"kb_used_data":273592,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93796288,"statfs":{"total":96636764160,"available":96047398912,"internally_reserved":0,"allocated":280158208,"data_stored":279991023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.907},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.753}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":8.4550000000000001},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.2750000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":4.3099999999999996},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.9269999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699156,"kb_used_data":397196,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93672684,"statfs":{"total":96636764160,"available":95920828416,"internally_reserved":0,"allocated":406728704,"data_stored":406561519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,11,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.248},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":14.557}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":15.103999999999999},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":14.586}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":4.0979999999999999},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":14.516999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":407232512,"data_stored":407232512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280465408,"data_stored":280465408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":406966272,"data_stored":406966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405688320,"data_stored":405688320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:04.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 227 20 100039 100019 3.4414 2.04688 0.035038 0.0453902 2024-09-18T11:48:05.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 228 20 100329 100309 3.43624 2.26562 0.0572425 0.0454605 2024-09-18T11:48:06.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 229 20 100612 100592 3.43089 2.21094 0.0471999 0.0455308 2024-09-18T11:48:06.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:07.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 230 20 100882 100862 3.42514 2.10938 0.154704 0.0456015 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:07.520 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-18T11:48:05.912133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":818447732,"num_objects":99913,"num_object_clones":0,"num_object_copies":399652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99940,"num_write_kb":799402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2789,"num_bytes_recovered":22609920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99967,"ondisk_log_size":99967,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690508,"kb_used_data":1474732,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374796852,"statfs":{"total":386547056640,"available":383791976448,"internally_reserved":0,"allocated":1510125568,"data_stored":1509456828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,7,5,33,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27271168,"num_objects":3329,"num_object_clones":0,"num_object_copies":13316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3329,"num_write_kb":26632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004661"},"pg_stats":[{"pgid":"1.f","version":"33'6447","reported_seq":6490,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.488456+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:05.488456+0000","last_peered":"2024-09-18T11:48:05.488456+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:05.488456+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52813824,"num_objects":6447,"num_object_clones":0,"num_object_copies":25788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":51576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4317}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6279","reported_seq":7683,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.704434+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:01.704434+0000","last_peered":"2024-09-18T11:48:01.704434+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:01.704434+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51401974,"num_objects":6275,"num_object_clones":0,"num_object_copies":25100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6277,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":671,"num_bytes_recovered":5480448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6125","reported_seq":6209,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.560212+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:01.560212+0000","last_peered":"2024-09-18T11:48:01.560212+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:01.560212+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50150011,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6124,"num_write_kb":48982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4068}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6201","reported_seq":6265,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.701597+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:01.701597+0000","last_peered":"2024-09-18T11:48:01.701597+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:01.701597+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6201,"log_dups_size":0,"ondisk_log_size":6201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50762998,"num_objects":6197,"num_object_clones":0,"num_object_copies":24788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":49581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6260","reported_seq":6317,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.343885+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:05.343885+0000","last_peered":"2024-09-18T11:48:05.343885+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:05.343885+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51254518,"num_objects":6257,"num_object_clones":0,"num_object_copies":25028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6258,"num_write_kb":50061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6210","reported_seq":6255,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.474807+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:05.474807+0000","last_peered":"2024-09-18T11:48:05.474807+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:05.474807+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6210,"log_dups_size":0,"ondisk_log_size":6210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50872320,"num_objects":6210,"num_object_clones":0,"num_object_copies":24840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6210,"num_write_kb":49680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6282","reported_seq":6349,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.818104+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:01.818104+0000","last_peered":"2024-09-18T11:48:01.818104+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:01.818104+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51410166,"num_objects":6276,"num_object_clones":0,"num_object_copies":25104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":50220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4142}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6179","reported_seq":6244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.735289+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:01.735289+0000","last_peered":"2024-09-18T11:48:01.735289+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:01.735289+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50574582,"num_objects":6174,"num_object_clones":0,"num_object_copies":24696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":49404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6382","reported_seq":6441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.480380+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:05.480380+0000","last_peered":"2024-09-18T11:48:05.480380+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:05.480380+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6382,"log_dups_size":0,"ondisk_log_size":6382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52210156,"num_objects":6374,"num_object_clones":0,"num_object_copies":25496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6379,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6045","reported_seq":6131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.520097+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:01.520097+0000","last_peered":"2024-09-18T11:48:01.520097+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:01.520097+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6045,"log_dups_size":0,"ondisk_log_size":6045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49501430,"num_objects":6043,"num_object_clones":0,"num_object_copies":24172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6043,"num_write_kb":48342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4068}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6325","reported_seq":6384,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.383826+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:05.383826+0000","last_peered":"2024-09-18T11:48:05.383826+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:05.383826+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51769201,"num_objects":6320,"num_object_clones":0,"num_object_copies":25280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":50564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6325","reported_seq":6364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.411259+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:05.411259+0000","last_peered":"2024-09-18T11:48:05.411259+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:05.411259+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51804795,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6324,"num_write_kb":50591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4215}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6207","reported_seq":6284,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.476467+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:01.476467+0000","last_peered":"2024-09-18T11:48:01.476467+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:01.476467+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50838139,"num_objects":6206,"num_object_clones":0,"num_object_copies":24824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4152}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6321","reported_seq":6372,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.333391+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:05.333391+0000","last_peered":"2024-09-18T11:48:05.333391+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:05.333391+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51746038,"num_objects":6317,"num_object_clones":0,"num_object_copies":25268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6319,"num_write_kb":50548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6256","reported_seq":10541,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:05.710756+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:05.710756+0000","last_peered":"2024-09-18T11:48:05.710756+0000","last_clean":"2024-09-18T11:48:05.710756+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:05.710756+0000","last_undegraded":"2024-09-18T11:48:05.710756+0000","last_fullsized":"2024-09-18T11:48:05.710756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51221750,"num_objects":6253,"num_object_clones":0,"num_object_copies":25012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6254,"num_write_kb":50029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6123","reported_seq":6187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:01.763760+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:01.763760+0000","last_peered":"2024-09-18T11:48:01.763760+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:01.763760+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115830,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":48956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":818447732,"num_objects":99913,"num_object_clones":0,"num_object_copies":399652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99940,"num_write_kb":799402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2789,"num_bytes_recovered":22609920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1511620608,"data_stored":1511620608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99967,"ondisk_log_size":99967,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703812,"kb_used_data":401788,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93668028,"statfs":{"total":96636764160,"available":95916060672,"internally_reserved":0,"allocated":411430912,"data_stored":411263727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":4.9409999999999998},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.9619999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":4.907},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":1.8839999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.915}]}]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702232,"kb_used_data":396560,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93669608,"statfs":{"total":96636764160,"available":95917678592,"internally_reserved":0,"allocated":406077440,"data_stored":405910255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,1,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":2.911},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.8879999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":3.9039999999999999},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":3.9239999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":3.363},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":2.8479999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585308,"kb_used_data":279188,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93786532,"statfs":{"total":96636764160,"available":96037408768,"internally_reserved":0,"allocated":285888512,"data_stored":285721327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":1.845}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.495},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.97999999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.33700000000000002},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.373}]}]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699156,"kb_used_data":397196,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93672684,"statfs":{"total":96636764160,"available":95920828416,"internally_reserved":0,"allocated":406728704,"data_stored":406561519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,11,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.248},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":14.557}]},{"osd":2,"last update":"Wed Sep 18 11:47:03 2024","interfaces":[{"interface":"back","average":{"1min":4.9020000000000001,"5min":2.3820000000000001,"15min":1.355},"min":{"1min":0.23999999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":12.034000000000001,"5min":12.034000000000001,"15min":12.034000000000001},"last":15.103999999999999},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.4940000000000002,"15min":1.6220000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":22.739999999999998,"5min":22.739999999999998,"15min":22.739999999999998},"last":14.586}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":4.0979999999999999},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":14.516999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":407232512,"data_stored":407232512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":286212096,"data_stored":286212096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":406966272,"data_stored":406966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":411209728,"data_stored":411209728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:07.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:08.109 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:08.110 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:08.127 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-18T11:48:07.912957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":824116596,"num_objects":100605,"num_object_clones":0,"num_object_copies":402420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100632,"num_write_kb":804938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2840,"num_bytes_recovered":23027712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100659,"ondisk_log_size":100659,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2710648,"kb_used_data":1486552,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374776712,"statfs":{"total":386547056640,"available":383771353088,"internally_reserved":0,"allocated":1522229248,"data_stored":1521560508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,5,7,20,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27361280,"num_objects":3340,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3340,"num_write_kb":26720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004642"},"pg_stats":[{"pgid":"1.f","version":"33'6447","reported_seq":6490,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.488456+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:05.488456+0000","last_peered":"2024-09-18T11:48:05.488456+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:05.488456+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52813824,"num_objects":6447,"num_object_clones":0,"num_object_copies":25788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":51576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4317}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6366","reported_seq":7872,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.821470+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:06.821470+0000","last_peered":"2024-09-18T11:48:06.821470+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:06.821470+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52114678,"num_objects":6362,"num_object_clones":0,"num_object_copies":25448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6364,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":722,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6221","reported_seq":6304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.588427+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:06.588427+0000","last_peered":"2024-09-18T11:48:06.588427+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:06.588427+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50936443,"num_objects":6218,"num_object_clones":0,"num_object_copies":24872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6220,"num_write_kb":49750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6296","reported_seq":6360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.860754+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:06.860754+0000","last_peered":"2024-09-18T11:48:06.860754+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:06.860754+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51541238,"num_objects":6292,"num_object_clones":0,"num_object_copies":25168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":50341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6260","reported_seq":6317,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.343885+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:05.343885+0000","last_peered":"2024-09-18T11:48:05.343885+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:05.343885+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51254518,"num_objects":6257,"num_object_clones":0,"num_object_copies":25028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6258,"num_write_kb":50061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6210","reported_seq":6255,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.474807+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:05.474807+0000","last_peered":"2024-09-18T11:48:05.474807+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:05.474807+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6210,"log_dups_size":0,"ondisk_log_size":6210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50872320,"num_objects":6210,"num_object_clones":0,"num_object_copies":24840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6210,"num_write_kb":49680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6364","reported_seq":6431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.723341+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:06.723341+0000","last_peered":"2024-09-18T11:48:06.723341+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:06.723341+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52081910,"num_objects":6358,"num_object_clones":0,"num_object_copies":25432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6362,"num_write_kb":50876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4224}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6258","reported_seq":6322,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.697904+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:06.697904+0000","last_peered":"2024-09-18T11:48:06.697904+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:06.697904+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51221750,"num_objects":6253,"num_object_clones":0,"num_object_copies":25012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6256,"num_write_kb":50036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6382","reported_seq":6441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.480380+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:05.480380+0000","last_peered":"2024-09-18T11:48:05.480380+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:05.480380+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6382,"log_dups_size":0,"ondisk_log_size":6382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52210156,"num_objects":6374,"num_object_clones":0,"num_object_copies":25496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6379,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6135","reported_seq":6221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.526338+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:06.526338+0000","last_peered":"2024-09-18T11:48:06.526338+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:06.526338+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50238710,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4158}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6325","reported_seq":6384,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.383826+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:05.383826+0000","last_peered":"2024-09-18T11:48:05.383826+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:05.383826+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51769201,"num_objects":6320,"num_object_clones":0,"num_object_copies":25280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":50564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6325","reported_seq":6364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.411259+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:05.411259+0000","last_peered":"2024-09-18T11:48:05.411259+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:05.411259+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51804795,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6324,"num_write_kb":50591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4215}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6295","reported_seq":6372,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.525231+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:06.525231+0000","last_peered":"2024-09-18T11:48:06.525231+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:06.525231+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51559035,"num_objects":6294,"num_object_clones":0,"num_object_copies":25176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":50351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4240}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6321","reported_seq":6372,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.333391+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:05.333391+0000","last_peered":"2024-09-18T11:48:05.333391+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:05.333391+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51746038,"num_objects":6317,"num_object_clones":0,"num_object_copies":25268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6319,"num_write_kb":50548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6256","reported_seq":10541,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:05.710756+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:05.710756+0000","last_peered":"2024-09-18T11:48:05.710756+0000","last_clean":"2024-09-18T11:48:05.710756+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:05.710756+0000","last_undegraded":"2024-09-18T11:48:05.710756+0000","last_fullsized":"2024-09-18T11:48:05.710756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51221750,"num_objects":6253,"num_object_clones":0,"num_object_copies":25012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6254,"num_write_kb":50029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6198","reported_seq":6262,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.863368+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:06.863368+0000","last_peered":"2024-09-18T11:48:06.863368+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:06.863368+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50730230,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":824116596,"num_objects":100605,"num_object_clones":0,"num_object_copies":402420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100632,"num_write_kb":804938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2840,"num_bytes_recovered":23027712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1522757632,"data_stored":1522757632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100659,"ondisk_log_size":100659,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703812,"kb_used_data":401788,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93668028,"statfs":{"total":96636764160,"available":95916060672,"internally_reserved":0,"allocated":411430912,"data_stored":411263727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":4.9409999999999998},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.9619999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":4.907},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":1.8839999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.915}]}]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712912,"kb_used_data":403080,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93658928,"statfs":{"total":96636764160,"available":95906742272,"internally_reserved":0,"allocated":412753920,"data_stored":412586735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":2.5030000000000001},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.089}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":7.1740000000000004},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":0.39200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":3.7890000000000001},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":3.363}]}]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585308,"kb_used_data":279188,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93786532,"statfs":{"total":96636764160,"available":96037408768,"internally_reserved":0,"allocated":285888512,"data_stored":285721327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":1.845}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.495},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.97999999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.33700000000000002},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.373}]}]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708616,"kb_used_data":402496,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93663224,"statfs":{"total":96636764160,"available":95911141376,"internally_reserved":0,"allocated":412155904,"data_stored":411988719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.66500000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.371},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":0.999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412835840,"data_stored":412835840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":286212096,"data_stored":286212096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412499968,"data_stored":412499968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":411209728,"data_stored":411209728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:08.129 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:08.129 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6447", 'reported_seq': 6490, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:05.488456+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:05.488456+0000', 'last_peered': '2024-09-18T11:48:05.488456+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:05.488456+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6447, 'log_dups_size': 0, 'ondisk_log_size': 6447, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52813824, 'num_objects': 6447, 'num_object_clones': 0, 'num_object_copies': 25788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6447, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6447, 'num_write_kb': 51576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4317}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:08.129 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:08.129 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6366", 'reported_seq': 7872, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.821470+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:06.821470+0000', 'last_peered': '2024-09-18T11:48:06.821470+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:06.821470+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6366, 'log_dups_size': 0, 'ondisk_log_size': 6366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52114678, 'num_objects': 6362, 'num_object_clones': 0, 'num_object_copies': 25448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6364, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 722, 'num_bytes_recovered': 5898240, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:08.129 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:08.129 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6221", 'reported_seq': 6304, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.588427+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:06.588427+0000', 'last_peered': '2024-09-18T11:48:06.588427+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:06.588427+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6221, 'log_dups_size': 0, 'ondisk_log_size': 6221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50936443, 'num_objects': 6218, 'num_object_clones': 0, 'num_object_copies': 24872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6220, 'num_write_kb': 49750, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4164}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:08.129 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6296", 'reported_seq': 6360, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.860754+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:06.860754+0000', 'last_peered': '2024-09-18T11:48:06.860754+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:06.860754+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6296, 'log_dups_size': 0, 'ondisk_log_size': 6296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51541238, 'num_objects': 6292, 'num_object_clones': 0, 'num_object_copies': 25168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6294, 'num_write_kb': 50341, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6260", 'reported_seq': 6317, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:05.343885+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:05.343885+0000', 'last_peered': '2024-09-18T11:48:05.343885+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:05.343885+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6260, 'log_dups_size': 0, 'ondisk_log_size': 6260, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51254518, 'num_objects': 6257, 'num_object_clones': 0, 'num_object_copies': 25028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6258, 'num_write_kb': 50061, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4175}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6210", 'reported_seq': 6255, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:05.474807+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:05.474807+0000', 'last_peered': '2024-09-18T11:48:05.474807+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:05.474807+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6210, 'log_dups_size': 0, 'ondisk_log_size': 6210, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50872320, 'num_objects': 6210, 'num_object_clones': 0, 'num_object_copies': 24840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6210, 'num_write_kb': 49680, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4056}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6364", 'reported_seq': 6431, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.723341+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:06.723341+0000', 'last_peered': '2024-09-18T11:48:06.723341+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:06.723341+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6364, 'log_dups_size': 0, 'ondisk_log_size': 6364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52081910, 'num_objects': 6358, 'num_object_clones': 0, 'num_object_copies': 25432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6362, 'num_write_kb': 50876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4224}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6258", 'reported_seq': 6322, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.697904+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:06.697904+0000', 'last_peered': '2024-09-18T11:48:06.697904+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:06.697904+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6258, 'log_dups_size': 0, 'ondisk_log_size': 6258, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51221750, 'num_objects': 6253, 'num_object_clones': 0, 'num_object_copies': 25012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6256, 'num_write_kb': 50036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4219}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:08.130 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6382", 'reported_seq': 6441, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:05.480380+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:05.480380+0000', 'last_peered': '2024-09-18T11:48:05.480380+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:05.480380+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6382, 'log_dups_size': 0, 'ondisk_log_size': 6382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52210156, 'num_objects': 6374, 'num_object_clones': 0, 'num_object_copies': 25496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6379, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4239}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:08.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:08.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6135", 'reported_seq': 6221, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.526338+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:06.526338+0000', 'last_peered': '2024-09-18T11:48:06.526338+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:06.526338+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6135, 'log_dups_size': 0, 'ondisk_log_size': 6135, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50238710, 'num_objects': 6133, 'num_object_clones': 0, 'num_object_copies': 24532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6133, 'num_write_kb': 49062, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4158}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:08.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:08.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6325", 'reported_seq': 6384, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:05.383826+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:05.383826+0000', 'last_peered': '2024-09-18T11:48:05.383826+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:05.383826+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6325, 'log_dups_size': 0, 'ondisk_log_size': 6325, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51769201, 'num_objects': 6320, 'num_object_clones': 0, 'num_object_copies': 25280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6322, 'num_write_kb': 50564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4257}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:08.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:08.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6325", 'reported_seq': 6364, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:05.411259+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:05.411259+0000', 'last_peered': '2024-09-18T11:48:05.411259+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:05.411259+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6325, 'log_dups_size': 0, 'ondisk_log_size': 6325, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51804795, 'num_objects': 6324, 'num_object_clones': 0, 'num_object_copies': 25296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6324, 'num_write_kb': 50591, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4215}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:08.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:08.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6295", 'reported_seq': 6372, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.525231+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:06.525231+0000', 'last_peered': '2024-09-18T11:48:06.525231+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:06.525231+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6295, 'log_dups_size': 0, 'ondisk_log_size': 6295, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51559035, 'num_objects': 6294, 'num_object_clones': 0, 'num_object_copies': 25176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6294, 'num_write_kb': 50351, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4240}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:08.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:08.131 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6321", 'reported_seq': 6372, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:05.333391+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:05.333391+0000', 'last_peered': '2024-09-18T11:48:05.333391+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:05.333391+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6321, 'log_dups_size': 0, 'ondisk_log_size': 6321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51746038, 'num_objects': 6317, 'num_object_clones': 0, 'num_object_copies': 25268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6319, 'num_write_kb': 50548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:08.132 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:08.132 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6198", 'reported_seq': 6262, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.863368+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:06.863368+0000', 'last_peered': '2024-09-18T11:48:06.863368+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:06.863368+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6198, 'log_dups_size': 0, 'ondisk_log_size': 6198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50730230, 'num_objects': 6193, 'num_object_clones': 0, 'num_object_copies': 24772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6196, 'num_write_kb': 49556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:08.132 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:08.264 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 231 20 101130 101110 3.4187 1.9375 0.0552304 0.0456903 2024-09-18T11:48:08.752 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:08.753 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30614/402420 objects degraded (7.607%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":286,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":100605,"data_bytes":824116596,"bytes_used":2775703552,"bytes_avail":383771353088,"bytes_total":386547056640,"degraded_objects":30614,"degraded_total":402420,"degraded_ratio":0.076074747775955465,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":101677,"recovering_keys_per_sec":0,"num_objects_recovered":149,"num_bytes_recovered":1220608,"num_keys_recovered":0,"write_bytes_sec":2279224,"read_op_per_sec":0,"write_op_per_sec":278},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:08.772 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:08.772 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:09.267 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 232 20 101378 101358 3.41232 1.9375 0.0432732 0.0457783 2024-09-18T11:48:09.354 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:09.354 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:09.383 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-18T11:48:07.912957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":824116596,"num_objects":100605,"num_object_clones":0,"num_object_copies":402420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100632,"num_write_kb":804938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2840,"num_bytes_recovered":23027712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100659,"ondisk_log_size":100659,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2710648,"kb_used_data":1486552,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374776712,"statfs":{"total":386547056640,"available":383771353088,"internally_reserved":0,"allocated":1522229248,"data_stored":1521560508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,5,7,20,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27361280,"num_objects":3340,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3340,"num_write_kb":26720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004642"},"pg_stats":[{"pgid":"1.f","version":"33'6447","reported_seq":6490,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.488456+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:05.488456+0000","last_peered":"2024-09-18T11:48:05.488456+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:05.488456+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52813824,"num_objects":6447,"num_object_clones":0,"num_object_copies":25788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":51576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4317}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6366","reported_seq":7872,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.821470+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:06.821470+0000","last_peered":"2024-09-18T11:48:06.821470+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:06.821470+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52114678,"num_objects":6362,"num_object_clones":0,"num_object_copies":25448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6364,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":722,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6221","reported_seq":6304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.588427+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:06.588427+0000","last_peered":"2024-09-18T11:48:06.588427+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:06.588427+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50936443,"num_objects":6218,"num_object_clones":0,"num_object_copies":24872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6220,"num_write_kb":49750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6296","reported_seq":6360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.860754+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:06.860754+0000","last_peered":"2024-09-18T11:48:06.860754+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:06.860754+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51541238,"num_objects":6292,"num_object_clones":0,"num_object_copies":25168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":50341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6260","reported_seq":6317,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.343885+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:05.343885+0000","last_peered":"2024-09-18T11:48:05.343885+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:05.343885+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51254518,"num_objects":6257,"num_object_clones":0,"num_object_copies":25028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6258,"num_write_kb":50061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6210","reported_seq":6255,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.474807+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:05.474807+0000","last_peered":"2024-09-18T11:48:05.474807+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:05.474807+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6210,"log_dups_size":0,"ondisk_log_size":6210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50872320,"num_objects":6210,"num_object_clones":0,"num_object_copies":24840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6210,"num_write_kb":49680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6364","reported_seq":6431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.723341+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:06.723341+0000","last_peered":"2024-09-18T11:48:06.723341+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:06.723341+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52081910,"num_objects":6358,"num_object_clones":0,"num_object_copies":25432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6362,"num_write_kb":50876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4224}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6258","reported_seq":6322,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.697904+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:06.697904+0000","last_peered":"2024-09-18T11:48:06.697904+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:06.697904+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51221750,"num_objects":6253,"num_object_clones":0,"num_object_copies":25012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6256,"num_write_kb":50036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6382","reported_seq":6441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.480380+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:05.480380+0000","last_peered":"2024-09-18T11:48:05.480380+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:05.480380+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6382,"log_dups_size":0,"ondisk_log_size":6382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52210156,"num_objects":6374,"num_object_clones":0,"num_object_copies":25496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6379,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6135","reported_seq":6221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.526338+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:06.526338+0000","last_peered":"2024-09-18T11:48:06.526338+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:06.526338+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50238710,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4158}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6325","reported_seq":6384,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.383826+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:05.383826+0000","last_peered":"2024-09-18T11:48:05.383826+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:05.383826+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51769201,"num_objects":6320,"num_object_clones":0,"num_object_copies":25280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":50564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6325","reported_seq":6364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.411259+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:05.411259+0000","last_peered":"2024-09-18T11:48:05.411259+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:05.411259+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51804795,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6324,"num_write_kb":50591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4215}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6295","reported_seq":6372,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.525231+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:06.525231+0000","last_peered":"2024-09-18T11:48:06.525231+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:06.525231+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51559035,"num_objects":6294,"num_object_clones":0,"num_object_copies":25176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":50351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4240}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6321","reported_seq":6372,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:05.333391+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:05.333391+0000","last_peered":"2024-09-18T11:48:05.333391+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:05.333391+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51746038,"num_objects":6317,"num_object_clones":0,"num_object_copies":25268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6319,"num_write_kb":50548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6256","reported_seq":10541,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:05.710756+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:05.710756+0000","last_peered":"2024-09-18T11:48:05.710756+0000","last_clean":"2024-09-18T11:48:05.710756+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:05.710756+0000","last_undegraded":"2024-09-18T11:48:05.710756+0000","last_fullsized":"2024-09-18T11:48:05.710756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51221750,"num_objects":6253,"num_object_clones":0,"num_object_copies":25012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6254,"num_write_kb":50029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6198","reported_seq":6262,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.863368+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:06.863368+0000","last_peered":"2024-09-18T11:48:06.863368+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:06.863368+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50730230,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":824116596,"num_objects":100605,"num_object_clones":0,"num_object_copies":402420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100632,"num_write_kb":804938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2840,"num_bytes_recovered":23027712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1522757632,"data_stored":1522757632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100659,"ondisk_log_size":100659,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703812,"kb_used_data":401788,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93668028,"statfs":{"total":96636764160,"available":95916060672,"internally_reserved":0,"allocated":411430912,"data_stored":411263727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":4.9409999999999998},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.9619999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":4.907},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":1.8839999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":1.915}]}]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712912,"kb_used_data":403080,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93658928,"statfs":{"total":96636764160,"available":95906742272,"internally_reserved":0,"allocated":412753920,"data_stored":412586735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":2.5030000000000001},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.089}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":7.1740000000000004},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":0.39200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":3.7890000000000001},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":3.363}]}]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585308,"kb_used_data":279188,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93786532,"statfs":{"total":96636764160,"available":96037408768,"internally_reserved":0,"allocated":285888512,"data_stored":285721327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":1.845}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.495},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.97999999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.33700000000000002},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.373}]}]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708616,"kb_used_data":402496,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93663224,"statfs":{"total":96636764160,"available":95911141376,"internally_reserved":0,"allocated":412155904,"data_stored":411988719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.66500000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.371},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":0.999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412835840,"data_stored":412835840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":286212096,"data_stored":286212096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412499968,"data_stored":412499968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":411209728,"data_stored":411209728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:10.264 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 233 20 101650 101630 3.4068 2.125 0.095108 0.045849 2024-09-18T11:48:11.264 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 234 20 101927 101907 3.40149 2.16406 0.0928596 0.0459216 2024-09-18T11:48:12.264 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 235 20 102210 102190 3.39642 2.21094 0.0953218 0.0459915 2024-09-18T11:48:12.384 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:12.958 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:12.958 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:12.999 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-18T11:48:11.914348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":831800692,"num_objects":101543,"num_object_clones":0,"num_object_copies":406172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101570,"num_write_kb":812442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2840,"num_bytes_recovered":23027712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101597,"ondisk_log_size":101597,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722584,"kb_used_data":1502648,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374764776,"statfs":{"total":386547056640,"available":383759130624,"internally_reserved":0,"allocated":1538711552,"data_stored":1538042812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,10,2,12,18,23,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29523968,"num_objects":3604,"num_object_clones":0,"num_object_copies":14416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3604,"num_write_kb":28832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":827392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004629"},"pg_stats":[{"pgid":"1.f","version":"33'6516","reported_seq":6559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.470328+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:10.470328+0000","last_peered":"2024-09-18T11:48:10.470328+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:10.470328+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6516,"log_dups_size":0,"ondisk_log_size":6516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53379072,"num_objects":6516,"num_object_clones":0,"num_object_copies":26064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6516,"num_write_kb":52128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6366","reported_seq":7872,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.821470+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:06.821470+0000","last_peered":"2024-09-18T11:48:06.821470+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:06.821470+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52114678,"num_objects":6362,"num_object_clones":0,"num_object_copies":25448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6364,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":722,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6313","reported_seq":6397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.575598+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:11.575598+0000","last_peered":"2024-09-18T11:48:11.575598+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:11.575598+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51690107,"num_objects":6310,"num_object_clones":0,"num_object_copies":25240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":50486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6296","reported_seq":6360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.860754+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:06.860754+0000","last_peered":"2024-09-18T11:48:06.860754+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:06.860754+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51541238,"num_objects":6292,"num_object_clones":0,"num_object_copies":25168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":50341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6342","reported_seq":6399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.452776+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:10.452776+0000","last_peered":"2024-09-18T11:48:10.452776+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:10.452776+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6342,"log_dups_size":0,"ondisk_log_size":6342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51926262,"num_objects":6339,"num_object_clones":0,"num_object_copies":25356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6340,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6292","reported_seq":6336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.516834+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:10.516834+0000","last_peered":"2024-09-18T11:48:10.516834+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:10.516834+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51544064,"num_objects":6292,"num_object_clones":0,"num_object_copies":25168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6292,"num_write_kb":50336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6364","reported_seq":6431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.723341+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:06.723341+0000","last_peered":"2024-09-18T11:48:06.723341+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:06.723341+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52081910,"num_objects":6358,"num_object_clones":0,"num_object_copies":25432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6362,"num_write_kb":50876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4224}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6258","reported_seq":6322,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.697904+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:06.697904+0000","last_peered":"2024-09-18T11:48:06.697904+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:06.697904+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51221750,"num_objects":6253,"num_object_clones":0,"num_object_copies":25012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6256,"num_write_kb":50036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6455","reported_seq":6513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.404188+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:10.404188+0000","last_peered":"2024-09-18T11:48:10.404188+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:10.404188+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52808172,"num_objects":6447,"num_object_clones":0,"num_object_copies":25788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6452,"num_write_kb":51593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4312}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6223","reported_seq":6309,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.493353+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:11.493353+0000","last_peered":"2024-09-18T11:48:11.493353+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:11.493353+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6223,"log_dups_size":0,"ondisk_log_size":6223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50959606,"num_objects":6221,"num_object_clones":0,"num_object_copies":24884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6221,"num_write_kb":49766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4246}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6410","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.518230+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:10.518230+0000","last_peered":"2024-09-18T11:48:10.518230+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:10.518230+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52465521,"num_objects":6405,"num_object_clones":0,"num_object_copies":25620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":51244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6403","reported_seq":6442,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.483749+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:10.483749+0000","last_peered":"2024-09-18T11:48:10.483749+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:10.483749+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52443771,"num_objects":6402,"num_object_clones":0,"num_object_copies":25608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6402,"num_write_kb":51215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4293}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6391","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.527235+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:11.527235+0000","last_peered":"2024-09-18T11:48:11.527235+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:11.527235+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52345467,"num_objects":6390,"num_object_clones":0,"num_object_copies":25560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6390,"num_write_kb":51119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4336}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6425","reported_seq":6476,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.570968+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:10.570968+0000","last_peered":"2024-09-18T11:48:10.570968+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:10.570968+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6425,"log_dups_size":0,"ondisk_log_size":6425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52598006,"num_objects":6421,"num_object_clones":0,"num_object_copies":25684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6423,"num_write_kb":51380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6345","reported_seq":10630,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:10.762388+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:10.762388+0000","last_peered":"2024-09-18T11:48:10.762388+0000","last_clean":"2024-09-18T11:48:10.762388+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:10.762388+0000","last_undegraded":"2024-09-18T11:48:10.762388+0000","last_fullsized":"2024-09-18T11:48:10.762388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6345,"log_dups_size":0,"ondisk_log_size":6345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51950838,"num_objects":6342,"num_object_clones":0,"num_object_copies":25368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6343,"num_write_kb":50741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6198","reported_seq":6262,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.863368+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:06.863368+0000","last_peered":"2024-09-18T11:48:06.863368+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:06.863368+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50730230,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":831800692,"num_objects":101543,"num_object_clones":0,"num_object_copies":406172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101570,"num_write_kb":812442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2840,"num_bytes_recovered":23027712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1539375104,"data_stored":1539375104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101597,"ondisk_log_size":101597,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713304,"kb_used_data":407120,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93658536,"statfs":{"total":96636764160,"available":95906340864,"internally_reserved":0,"allocated":416890880,"data_stored":416723695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,7,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":1.494},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.53}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.556},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.92700000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.44900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705640,"kb_used_data":408288,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93666200,"statfs":{"total":96636764160,"available":95914188800,"internally_reserved":0,"allocated":418086912,"data_stored":417919727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":16.706},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":9.9949999999999992}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.2539999999999996},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":16.768999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":7.2670000000000003},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":2.5710000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595024,"kb_used_data":284744,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93776816,"statfs":{"total":96636764160,"available":96027459584,"internally_reserved":0,"allocated":291577856,"data_stored":291410671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.1280000000000001},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.73599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":4.2000000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.56699999999999995},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.881}]}]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708616,"kb_used_data":402496,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93663224,"statfs":{"total":96636764160,"available":95911141376,"internally_reserved":0,"allocated":412155904,"data_stored":411988719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.66500000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.371},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":0.999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412835840,"data_stored":412835840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291917824,"data_stored":291917824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417910784,"data_stored":417910784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":416710656,"data_stored":416710656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:13.000 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:13.264 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 236 20 102461 102441 3.39034 1.96094 0.0514075 0.0460757 2024-09-18T11:48:13.549 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:13.549 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:13.571 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-18T11:48:11.914348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":831800692,"num_objects":101543,"num_object_clones":0,"num_object_copies":406172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101570,"num_write_kb":812442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2840,"num_bytes_recovered":23027712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101597,"ondisk_log_size":101597,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722584,"kb_used_data":1502648,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374764776,"statfs":{"total":386547056640,"available":383759130624,"internally_reserved":0,"allocated":1538711552,"data_stored":1538042812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,10,2,12,18,23,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29523968,"num_objects":3604,"num_object_clones":0,"num_object_copies":14416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3604,"num_write_kb":28832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":827392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004629"},"pg_stats":[{"pgid":"1.f","version":"33'6516","reported_seq":6559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.470328+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:10.470328+0000","last_peered":"2024-09-18T11:48:10.470328+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:10.470328+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6516,"log_dups_size":0,"ondisk_log_size":6516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53379072,"num_objects":6516,"num_object_clones":0,"num_object_copies":26064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6516,"num_write_kb":52128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6366","reported_seq":7872,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.821470+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:06.821470+0000","last_peered":"2024-09-18T11:48:06.821470+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:06.821470+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52114678,"num_objects":6362,"num_object_clones":0,"num_object_copies":25448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6364,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":722,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6313","reported_seq":6397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.575598+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:11.575598+0000","last_peered":"2024-09-18T11:48:11.575598+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:11.575598+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51690107,"num_objects":6310,"num_object_clones":0,"num_object_copies":25240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":50486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6296","reported_seq":6360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.860754+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:06.860754+0000","last_peered":"2024-09-18T11:48:06.860754+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:06.860754+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51541238,"num_objects":6292,"num_object_clones":0,"num_object_copies":25168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":50341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6342","reported_seq":6399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.452776+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:10.452776+0000","last_peered":"2024-09-18T11:48:10.452776+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:10.452776+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6342,"log_dups_size":0,"ondisk_log_size":6342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51926262,"num_objects":6339,"num_object_clones":0,"num_object_copies":25356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6340,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6292","reported_seq":6336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.516834+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:10.516834+0000","last_peered":"2024-09-18T11:48:10.516834+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:10.516834+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51544064,"num_objects":6292,"num_object_clones":0,"num_object_copies":25168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6292,"num_write_kb":50336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6364","reported_seq":6431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.723341+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:06.723341+0000","last_peered":"2024-09-18T11:48:06.723341+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:06.723341+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52081910,"num_objects":6358,"num_object_clones":0,"num_object_copies":25432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6362,"num_write_kb":50876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4224}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6258","reported_seq":6322,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.697904+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:06.697904+0000","last_peered":"2024-09-18T11:48:06.697904+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:06.697904+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51221750,"num_objects":6253,"num_object_clones":0,"num_object_copies":25012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6256,"num_write_kb":50036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6455","reported_seq":6513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.404188+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:10.404188+0000","last_peered":"2024-09-18T11:48:10.404188+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:10.404188+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52808172,"num_objects":6447,"num_object_clones":0,"num_object_copies":25788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6452,"num_write_kb":51593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4312}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6223","reported_seq":6309,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.493353+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:11.493353+0000","last_peered":"2024-09-18T11:48:11.493353+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:11.493353+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6223,"log_dups_size":0,"ondisk_log_size":6223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50959606,"num_objects":6221,"num_object_clones":0,"num_object_copies":24884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6221,"num_write_kb":49766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4246}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6410","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.518230+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:10.518230+0000","last_peered":"2024-09-18T11:48:10.518230+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:10.518230+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52465521,"num_objects":6405,"num_object_clones":0,"num_object_copies":25620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":51244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6403","reported_seq":6442,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.483749+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:10.483749+0000","last_peered":"2024-09-18T11:48:10.483749+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:10.483749+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52443771,"num_objects":6402,"num_object_clones":0,"num_object_copies":25608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6402,"num_write_kb":51215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4293}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6391","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.527235+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:11.527235+0000","last_peered":"2024-09-18T11:48:11.527235+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:11.527235+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52345467,"num_objects":6390,"num_object_clones":0,"num_object_copies":25560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6390,"num_write_kb":51119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4336}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6425","reported_seq":6476,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.570968+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:10.570968+0000","last_peered":"2024-09-18T11:48:10.570968+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:10.570968+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6425,"log_dups_size":0,"ondisk_log_size":6425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52598006,"num_objects":6421,"num_object_clones":0,"num_object_copies":25684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6423,"num_write_kb":51380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6345","reported_seq":10630,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:10.762388+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:10.762388+0000","last_peered":"2024-09-18T11:48:10.762388+0000","last_clean":"2024-09-18T11:48:10.762388+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:10.762388+0000","last_undegraded":"2024-09-18T11:48:10.762388+0000","last_fullsized":"2024-09-18T11:48:10.762388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6345,"log_dups_size":0,"ondisk_log_size":6345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51950838,"num_objects":6342,"num_object_clones":0,"num_object_copies":25368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6343,"num_write_kb":50741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6198","reported_seq":6262,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:06.863368+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:06.863368+0000","last_peered":"2024-09-18T11:48:06.863368+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:06.863368+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50730230,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":831800692,"num_objects":101543,"num_object_clones":0,"num_object_copies":406172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101570,"num_write_kb":812442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2840,"num_bytes_recovered":23027712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1539375104,"data_stored":1539375104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101597,"ondisk_log_size":101597,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713304,"kb_used_data":407120,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93658536,"statfs":{"total":96636764160,"available":95906340864,"internally_reserved":0,"allocated":416890880,"data_stored":416723695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,7,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":1.494},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.53}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.556},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.92700000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.44900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705640,"kb_used_data":408288,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93666200,"statfs":{"total":96636764160,"available":95914188800,"internally_reserved":0,"allocated":418086912,"data_stored":417919727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":16.706},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":9.9949999999999992}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.2539999999999996},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":16.768999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":7.2670000000000003},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":2.5710000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595024,"kb_used_data":284744,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93776816,"statfs":{"total":96636764160,"available":96027459584,"internally_reserved":0,"allocated":291577856,"data_stored":291410671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.1280000000000001},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.73599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":4.2000000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.56699999999999995},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.881}]}]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708616,"kb_used_data":402496,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93663224,"statfs":{"total":96636764160,"available":95911141376,"internally_reserved":0,"allocated":412155904,"data_stored":411988719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.66500000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":0.371},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":0.999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412835840,"data_stored":412835840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291917824,"data_stored":291917824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417910784,"data_stored":417910784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":416710656,"data_stored":416710656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:13.572 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:13.572 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6516", 'reported_seq': 6559, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:10.470328+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:10.470328+0000', 'last_peered': '2024-09-18T11:48:10.470328+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:10.470328+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6516, 'log_dups_size': 0, 'ondisk_log_size': 6516, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53379072, 'num_objects': 6516, 'num_object_clones': 0, 'num_object_copies': 26064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6516, 'num_write_kb': 52128, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4386}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:13.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:13.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6366", 'reported_seq': 7872, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.821470+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:06.821470+0000', 'last_peered': '2024-09-18T11:48:06.821470+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:06.821470+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6366, 'log_dups_size': 0, 'ondisk_log_size': 6366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52114678, 'num_objects': 6362, 'num_object_clones': 0, 'num_object_copies': 25448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6364, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 722, 'num_bytes_recovered': 5898240, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:13.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:13.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6313", 'reported_seq': 6397, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:11.575598+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:11.575598+0000', 'last_peered': '2024-09-18T11:48:11.575598+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:11.575598+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6313, 'log_dups_size': 0, 'ondisk_log_size': 6313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51690107, 'num_objects': 6310, 'num_object_clones': 0, 'num_object_copies': 25240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6312, 'num_write_kb': 50486, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4256}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:13.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:13.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6296", 'reported_seq': 6360, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.860754+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:06.860754+0000', 'last_peered': '2024-09-18T11:48:06.860754+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:06.860754+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6296, 'log_dups_size': 0, 'ondisk_log_size': 6296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51541238, 'num_objects': 6292, 'num_object_clones': 0, 'num_object_copies': 25168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6294, 'num_write_kb': 50341, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:13.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:13.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6342", 'reported_seq': 6399, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:10.452776+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:10.452776+0000', 'last_peered': '2024-09-18T11:48:10.452776+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:10.452776+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6342, 'log_dups_size': 0, 'ondisk_log_size': 6342, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51926262, 'num_objects': 6339, 'num_object_clones': 0, 'num_object_copies': 25356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6340, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4257}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:13.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:13.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6292", 'reported_seq': 6336, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:10.516834+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:10.516834+0000', 'last_peered': '2024-09-18T11:48:10.516834+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:10.516834+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6292, 'log_dups_size': 0, 'ondisk_log_size': 6292, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51544064, 'num_objects': 6292, 'num_object_clones': 0, 'num_object_copies': 25168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6292, 'num_write_kb': 50336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4138}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6364", 'reported_seq': 6431, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.723341+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:06.723341+0000', 'last_peered': '2024-09-18T11:48:06.723341+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:06.723341+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6364, 'log_dups_size': 0, 'ondisk_log_size': 6364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52081910, 'num_objects': 6358, 'num_object_clones': 0, 'num_object_copies': 25432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6362, 'num_write_kb': 50876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4224}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6258", 'reported_seq': 6322, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.697904+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:06.697904+0000', 'last_peered': '2024-09-18T11:48:06.697904+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:06.697904+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6258, 'log_dups_size': 0, 'ondisk_log_size': 6258, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51221750, 'num_objects': 6253, 'num_object_clones': 0, 'num_object_copies': 25012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6256, 'num_write_kb': 50036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4219}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6455", 'reported_seq': 6513, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:10.404188+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:10.404188+0000', 'last_peered': '2024-09-18T11:48:10.404188+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:10.404188+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6455, 'log_dups_size': 0, 'ondisk_log_size': 6455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52808172, 'num_objects': 6447, 'num_object_clones': 0, 'num_object_copies': 25788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6447, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6452, 'num_write_kb': 51593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4312}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6223", 'reported_seq': 6309, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:11.493353+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:11.493353+0000', 'last_peered': '2024-09-18T11:48:11.493353+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:11.493353+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6223, 'log_dups_size': 0, 'ondisk_log_size': 6223, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50959606, 'num_objects': 6221, 'num_object_clones': 0, 'num_object_copies': 24884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6221, 'num_write_kb': 49766, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4246}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6410", 'reported_seq': 6469, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:10.518230+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:10.518230+0000', 'last_peered': '2024-09-18T11:48:10.518230+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:10.518230+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6410, 'log_dups_size': 0, 'ondisk_log_size': 6410, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52465521, 'num_objects': 6405, 'num_object_clones': 0, 'num_object_copies': 25620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6407, 'num_write_kb': 51244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4342}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:13.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:13.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6403", 'reported_seq': 6442, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:10.483749+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:10.483749+0000', 'last_peered': '2024-09-18T11:48:10.483749+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:10.483749+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6403, 'log_dups_size': 0, 'ondisk_log_size': 6403, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52443771, 'num_objects': 6402, 'num_object_clones': 0, 'num_object_copies': 25608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6402, 'num_write_kb': 51215, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:13.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:13.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6391", 'reported_seq': 6469, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:11.527235+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:11.527235+0000', 'last_peered': '2024-09-18T11:48:11.527235+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:11.527235+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6391, 'log_dups_size': 0, 'ondisk_log_size': 6391, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52345467, 'num_objects': 6390, 'num_object_clones': 0, 'num_object_copies': 25560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6390, 'num_write_kb': 51119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4336}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:13.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:13.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6425", 'reported_seq': 6476, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:10.570968+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:10.570968+0000', 'last_peered': '2024-09-18T11:48:10.570968+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:10.570968+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6425, 'log_dups_size': 0, 'ondisk_log_size': 6425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52598006, 'num_objects': 6421, 'num_object_clones': 0, 'num_object_copies': 25684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6423, 'num_write_kb': 51380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4263}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:13.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:13.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6198", 'reported_seq': 6262, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:06.863368+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:06.863368+0000', 'last_peered': '2024-09-18T11:48:06.863368+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:06.863368+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6198, 'log_dups_size': 0, 'ondisk_log_size': 6198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50730230, 'num_objects': 6193, 'num_object_clones': 0, 'num_object_copies': 24772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6196, 'num_write_kb': 49556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:13.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:14.240 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:14.240 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30565/407732 objects degraded (7.496%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":292,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":101933,"data_bytes":834995572,"bytes_used":2797621248,"bytes_avail":383749435392,"bytes_total":386547056640,"degraded_objects":30565,"degraded_total":407732,"degraded_ratio":0.074963456388019586,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":68240,"recovering_keys_per_sec":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"write_bytes_sec":1842492,"read_op_per_sec":0,"write_op_per_sec":224},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:14.264 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 237 20 102701 102681 3.38394 1.875 0.104501 0.0461593 2024-09-18T11:48:14.267 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:14.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:14.851 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:14.852 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:14.870 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-18T11:48:13.915065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":834995572,"num_objects":101933,"num_object_clones":0,"num_object_copies":407732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101960,"num_write_kb":815562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":23429120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101987,"ondisk_log_size":101987,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2732052,"kb_used_data":1507956,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374755308,"statfs":{"total":386547056640,"available":383749435392,"internally_reserved":0,"allocated":1544146944,"data_stored":1543478204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,8,4,10,19,26,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22118400,"num_objects":2700,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2700,"num_write_kb":21600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004610"},"pg_stats":[{"pgid":"1.f","version":"33'6516","reported_seq":6559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.470328+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:10.470328+0000","last_peered":"2024-09-18T11:48:10.470328+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:10.470328+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6516,"log_dups_size":0,"ondisk_log_size":6516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53379072,"num_objects":6516,"num_object_clones":0,"num_object_copies":26064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6516,"num_write_kb":52128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6444","reported_seq":8047,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.878538+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:11.878538+0000","last_peered":"2024-09-18T11:48:11.878538+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:11.878538+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52753654,"num_objects":6440,"num_object_clones":0,"num_object_copies":25760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6442,"num_write_kb":51532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":771,"num_bytes_recovered":6299648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6313","reported_seq":6397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.575598+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:11.575598+0000","last_peered":"2024-09-18T11:48:11.575598+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:11.575598+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51690107,"num_objects":6310,"num_object_clones":0,"num_object_copies":25240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":50486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6371","reported_seq":6435,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.884541+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:11.884541+0000","last_peered":"2024-09-18T11:48:11.884541+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:11.884541+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52155638,"num_objects":6367,"num_object_clones":0,"num_object_copies":25468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6369,"num_write_kb":50941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4253}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6342","reported_seq":6399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.452776+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:10.452776+0000","last_peered":"2024-09-18T11:48:10.452776+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:10.452776+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6342,"log_dups_size":0,"ondisk_log_size":6342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51926262,"num_objects":6339,"num_object_clones":0,"num_object_copies":25356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6340,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6292","reported_seq":6336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.516834+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:10.516834+0000","last_peered":"2024-09-18T11:48:10.516834+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:10.516834+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51544064,"num_objects":6292,"num_object_clones":0,"num_object_copies":25168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6292,"num_write_kb":50336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6427","reported_seq":6494,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.807792+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:11.807792+0000","last_peered":"2024-09-18T11:48:11.807792+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:11.807792+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6427,"log_dups_size":0,"ondisk_log_size":6427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52598006,"num_objects":6421,"num_object_clones":0,"num_object_copies":25684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6425,"num_write_kb":51380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4287}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6354","reported_seq":6417,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.826453+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:11.826453+0000","last_peered":"2024-09-18T11:48:11.826453+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:11.826453+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6354,"log_dups_size":0,"ondisk_log_size":6354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52008182,"num_objects":6349,"num_object_clones":0,"num_object_copies":25396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6352,"num_write_kb":50804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4315}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6455","reported_seq":6513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.404188+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:10.404188+0000","last_peered":"2024-09-18T11:48:10.404188+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:10.404188+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52808172,"num_objects":6447,"num_object_clones":0,"num_object_copies":25788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6452,"num_write_kb":51593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4312}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6223","reported_seq":6309,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.493353+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:11.493353+0000","last_peered":"2024-09-18T11:48:11.493353+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:11.493353+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6223,"log_dups_size":0,"ondisk_log_size":6223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50959606,"num_objects":6221,"num_object_clones":0,"num_object_copies":24884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6221,"num_write_kb":49766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4246}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6410","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.518230+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:10.518230+0000","last_peered":"2024-09-18T11:48:10.518230+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:10.518230+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52465521,"num_objects":6405,"num_object_clones":0,"num_object_copies":25620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":51244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6403","reported_seq":6442,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.483749+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:10.483749+0000","last_peered":"2024-09-18T11:48:10.483749+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:10.483749+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52443771,"num_objects":6402,"num_object_clones":0,"num_object_copies":25608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6402,"num_write_kb":51215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4293}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6391","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.527235+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:11.527235+0000","last_peered":"2024-09-18T11:48:11.527235+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:11.527235+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52345467,"num_objects":6390,"num_object_clones":0,"num_object_copies":25560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6390,"num_write_kb":51119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4336}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6425","reported_seq":6476,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:10.570968+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:10.570968+0000","last_peered":"2024-09-18T11:48:10.570968+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:10.570968+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6425,"log_dups_size":0,"ondisk_log_size":6425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52598006,"num_objects":6421,"num_object_clones":0,"num_object_copies":25684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6423,"num_write_kb":51380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6345","reported_seq":10630,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:10.762388+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:10.762388+0000","last_peered":"2024-09-18T11:48:10.762388+0000","last_clean":"2024-09-18T11:48:10.762388+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:10.762388+0000","last_undegraded":"2024-09-18T11:48:10.762388+0000","last_fullsized":"2024-09-18T11:48:10.762388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6345,"log_dups_size":0,"ondisk_log_size":6345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51950838,"num_objects":6342,"num_object_clones":0,"num_object_copies":25368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6343,"num_write_kb":50741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6276","reported_seq":6340,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:11.648283+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:11.648283+0000","last_peered":"2024-09-18T11:48:11.648283+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:11.648283+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6276,"log_dups_size":0,"ondisk_log_size":6276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51369206,"num_objects":6271,"num_object_clones":0,"num_object_copies":25084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6274,"num_write_kb":50180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":834995572,"num_objects":101933,"num_object_clones":0,"num_object_copies":407732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101960,"num_write_kb":815562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":23429120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1544785920,"data_stored":1544785920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101987,"ondisk_log_size":101987,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713304,"kb_used_data":407120,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93658536,"statfs":{"total":96636764160,"available":95906340864,"internally_reserved":0,"allocated":416890880,"data_stored":416723695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,7,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":3.5009999999999999,"15min":3.0350000000000001},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":10.775,"5min":12.866,"15min":12.866},"last":1.494},{"interface":"front","average":{"1min":5.3099999999999996,"5min":3.8210000000000002,"15min":3.0569999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.807,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.53}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.556},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.92700000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":3.8530000000000002,"5min":2.0880000000000001,"15min":1.3540000000000001},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.487,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":5.1440000000000001,"5min":3.5720000000000001,"15min":2.9180000000000001},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.470000000000001,"5min":16.271000000000001,"15min":16.271000000000001},"last":0.44900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705640,"kb_used_data":408288,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93666200,"statfs":{"total":96636764160,"available":95914188800,"internally_reserved":0,"allocated":418086912,"data_stored":417919727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":16.706},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":9.9949999999999992}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.2539999999999996},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":16.768999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:47:10 2024","interfaces":[{"interface":"back","average":{"1min":4.3929999999999998,"5min":3.1299999999999999,"15min":2.6400000000000001},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.34,"5min":14.34,"15min":14.34},"last":7.2670000000000003},{"interface":"front","average":{"1min":2.6469999999999998,"5min":2.5289999999999999,"15min":1.976},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.442,"5min":16.673999999999999,"15min":16.673999999999999},"last":2.5710000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595024,"kb_used_data":284744,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93776816,"statfs":{"total":96636764160,"available":96027459584,"internally_reserved":0,"allocated":291577856,"data_stored":291410671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.1280000000000001},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.73599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":4.2000000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.56699999999999995},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.881}]}]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718084,"kb_used_data":407804,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93653756,"statfs":{"total":96636764160,"available":95901446144,"internally_reserved":0,"allocated":417591296,"data_stored":417424111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":4.5129999999999999},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":2.96}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.0990000000000002},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.173}]},{"osd":3,"last update":"Wed Sep 18 11:47:13 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.8100000000000001,"15min":2.5099999999999998},"min":{"1min":0.33900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":14.946999999999999,"5min":14.946999999999999,"15min":14.946999999999999},"last":12.595000000000001},{"interface":"front","average":{"1min":2.9460000000000002,"5min":2.327,"15min":1.7889999999999999},"min":{"1min":0.26900000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":2.931}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":418246656,"data_stored":418246656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291917824,"data_stored":291917824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417910784,"data_stored":417910784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":416710656,"data_stored":416710656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:15.267 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 238 20 102958 102938 3.37812 2.00781 0.123918 0.046239 2024-09-18T11:48:16.267 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout: 239 20 103235 103215 3.37304 2.16406 0.0475938 0.0463114 2024-09-18T11:48:17.267 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Total time run: 240.041 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Total writes made: 103501 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Write size: 8192 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Object size: 8192 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Bandwidth (MB/sec): 3.3686 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Stddev Bandwidth: 2.3636 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Max bandwidth (MB/sec): 13.8281 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Min bandwidth (MB/sec): 0 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Average IOPS: 431 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Stddev IOPS: 302.541 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Max IOPS: 1770 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Min IOPS: 0 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Average Latency(s): 0.046376 2024-09-18T11:48:17.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Stddev Latency(s): 0.0448009 2024-09-18T11:48:17.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Max latency(s): 2.42146 2024-09-18T11:48:17.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Min latency(s): 0.00272854 2024-09-18T11:48:17.279 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T11:48:17.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:18.392 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:18.393 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:18.410 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-18T11:48:17.916049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":845702516,"num_objects":103240,"num_object_clones":0,"num_object_copies":412960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103267,"num_write_kb":826018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2943,"num_bytes_recovered":23871488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103294,"ondisk_log_size":103294,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736376,"kb_used_data":1528920,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":374750984,"statfs":{"total":386547056640,"available":383745007616,"internally_reserved":0,"allocated":1565614080,"data_stored":1564945340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,7,7,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27254784,"num_objects":3327,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3327,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003916"},"pg_stats":[{"pgid":"1.f","version":"33'6604","reported_seq":6647,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.512023+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:15.512023+0000","last_peered":"2024-09-18T11:48:15.512023+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:15.512023+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54099968,"num_objects":6604,"num_object_clones":0,"num_object_copies":26416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":52832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6522","reported_seq":8234,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.935924+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:16.935924+0000","last_peered":"2024-09-18T11:48:16.935924+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:16.935924+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53392630,"num_objects":6518,"num_object_clones":0,"num_object_copies":26072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6520,"num_write_kb":52156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":825,"num_bytes_recovered":6742016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6397","reported_seq":6481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.561164+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:16.561164+0000","last_peered":"2024-09-18T11:48:16.561164+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:16.561164+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6397,"log_dups_size":0,"ondisk_log_size":6397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52378235,"num_objects":6394,"num_object_clones":0,"num_object_copies":25576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6396,"num_write_kb":51158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6458","reported_seq":6522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.852522+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:16.852522+0000","last_peered":"2024-09-18T11:48:16.852522+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:16.852522+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6458,"log_dups_size":0,"ondisk_log_size":6458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52868342,"num_objects":6454,"num_object_clones":0,"num_object_copies":25816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":51637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6416","reported_seq":6473,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.508838+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:15.508838+0000","last_peered":"2024-09-18T11:48:15.508838+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:15.508838+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52532470,"num_objects":6413,"num_object_clones":0,"num_object_copies":25652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6414,"num_write_kb":51309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6366","reported_seq":6411,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.571828+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:15.571828+0000","last_peered":"2024-09-18T11:48:15.571828+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:15.571828+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52150272,"num_objects":6366,"num_object_clones":0,"num_object_copies":25464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6366,"num_write_kb":50928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6506","reported_seq":6573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.819454+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:16.819454+0000","last_peered":"2024-09-18T11:48:16.819454+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:16.819454+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6506,"log_dups_size":0,"ondisk_log_size":6506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53245174,"num_objects":6500,"num_object_clones":0,"num_object_copies":26000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6504,"num_write_kb":52012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4366}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6437","reported_seq":6502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.938253+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:16.938253+0000","last_peered":"2024-09-18T11:48:16.938253+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:16.938253+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6437,"log_dups_size":0,"ondisk_log_size":6437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52688118,"num_objects":6432,"num_object_clones":0,"num_object_copies":25728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":51468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6554","reported_seq":6614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.554488+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:15.554488+0000","last_peered":"2024-09-18T11:48:15.554488+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:15.554488+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53619180,"num_objects":6546,"num_object_clones":0,"num_object_copies":26184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6551,"num_write_kb":52385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6290","reported_seq":6376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.512330+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:16.512330+0000","last_peered":"2024-09-18T11:48:16.512330+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:16.512330+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6290,"log_dups_size":0,"ondisk_log_size":6290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51508470,"num_objects":6288,"num_object_clones":0,"num_object_copies":25152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4313}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6494","reported_seq":6552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.602217+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:15.602217+0000","last_peered":"2024-09-18T11:48:15.602217+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:15.602217+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6494,"log_dups_size":0,"ondisk_log_size":6494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53153649,"num_objects":6489,"num_object_clones":0,"num_object_copies":25956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":51916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4426}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6490","reported_seq":6529,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.422362+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:15.422362+0000","last_peered":"2024-09-18T11:48:15.422362+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:15.422362+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53156475,"num_objects":6489,"num_object_clones":0,"num_object_copies":25956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6489,"num_write_kb":51911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4380}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6472","reported_seq":6550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.405138+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:16.405138+0000","last_peered":"2024-09-18T11:48:16.405138+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:16.405138+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53009019,"num_objects":6471,"num_object_clones":0,"num_object_copies":25884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":51767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4417}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6512","reported_seq":6561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.426673+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:15.426673+0000","last_peered":"2024-09-18T11:48:15.426673+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:15.426673+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6512,"log_dups_size":0,"ondisk_log_size":6512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53310710,"num_objects":6508,"num_object_clones":0,"num_object_copies":26032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6510,"num_write_kb":52076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6426","reported_seq":10711,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:15.754389+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:15.754389+0000","last_peered":"2024-09-18T11:48:15.754389+0000","last_clean":"2024-09-18T11:48:15.754389+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:15.754389+0000","last_undegraded":"2024-09-18T11:48:15.754389+0000","last_fullsized":"2024-09-18T11:48:15.754389+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52614390,"num_objects":6423,"num_object_clones":0,"num_object_copies":25692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":51389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6350","reported_seq":6414,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.755735+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:16.755735+0000","last_peered":"2024-09-18T11:48:16.755735+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:16.755735+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6350,"log_dups_size":0,"ondisk_log_size":6350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51975414,"num_objects":6345,"num_object_clones":0,"num_object_copies":25380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6348,"num_write_kb":50772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":845702516,"num_objects":103240,"num_object_clones":0,"num_object_copies":412960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103267,"num_write_kb":826018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2943,"num_bytes_recovered":23871488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1566560256,"data_stored":1566560256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103294,"ondisk_log_size":103294,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722748,"kb_used_data":412404,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93649092,"statfs":{"total":96636764160,"available":95896670208,"internally_reserved":0,"allocated":422301696,"data_stored":422134511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.33300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":16.824000000000002},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.4359999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714964,"kb_used_data":413452,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93656876,"statfs":{"total":96636764160,"available":95904641024,"internally_reserved":0,"allocated":423374848,"data_stored":423207663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.7789999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":18.361000000000001},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":2.3700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.30399999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.2230000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587960,"kb_used_data":290160,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93783880,"statfs":{"total":96636764160,"available":96034693120,"internally_reserved":0,"allocated":297123840,"data_stored":296956655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":14.877000000000001},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.24099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.48799999999999999},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":14.922000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710704,"kb_used_data":412904,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93661136,"statfs":{"total":96636764160,"available":95909003264,"internally_reserved":0,"allocated":422813696,"data_stored":422646511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":9.8100000000000005},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.7430000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.782}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":423665664,"data_stored":423665664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297504768,"data_stored":297504768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":423272448,"data_stored":423272448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":422117376,"data_stored":422117376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:18.412 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:18.965 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:18.966 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:18.996 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-18T11:48:17.916049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":845702516,"num_objects":103240,"num_object_clones":0,"num_object_copies":412960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103267,"num_write_kb":826018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2943,"num_bytes_recovered":23871488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103294,"ondisk_log_size":103294,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736376,"kb_used_data":1528920,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":374750984,"statfs":{"total":386547056640,"available":383745007616,"internally_reserved":0,"allocated":1565614080,"data_stored":1564945340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,7,7,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27254784,"num_objects":3327,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3327,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003916"},"pg_stats":[{"pgid":"1.f","version":"33'6604","reported_seq":6647,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.512023+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:15.512023+0000","last_peered":"2024-09-18T11:48:15.512023+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:15.512023+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54099968,"num_objects":6604,"num_object_clones":0,"num_object_copies":26416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":52832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6522","reported_seq":8234,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.935924+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:16.935924+0000","last_peered":"2024-09-18T11:48:16.935924+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:16.935924+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53392630,"num_objects":6518,"num_object_clones":0,"num_object_copies":26072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6520,"num_write_kb":52156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":825,"num_bytes_recovered":6742016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6397","reported_seq":6481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.561164+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:16.561164+0000","last_peered":"2024-09-18T11:48:16.561164+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:16.561164+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6397,"log_dups_size":0,"ondisk_log_size":6397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52378235,"num_objects":6394,"num_object_clones":0,"num_object_copies":25576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6396,"num_write_kb":51158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6458","reported_seq":6522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.852522+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:16.852522+0000","last_peered":"2024-09-18T11:48:16.852522+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:16.852522+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6458,"log_dups_size":0,"ondisk_log_size":6458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52868342,"num_objects":6454,"num_object_clones":0,"num_object_copies":25816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":51637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6416","reported_seq":6473,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.508838+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:15.508838+0000","last_peered":"2024-09-18T11:48:15.508838+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:15.508838+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52532470,"num_objects":6413,"num_object_clones":0,"num_object_copies":25652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6414,"num_write_kb":51309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6366","reported_seq":6411,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.571828+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:15.571828+0000","last_peered":"2024-09-18T11:48:15.571828+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:15.571828+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52150272,"num_objects":6366,"num_object_clones":0,"num_object_copies":25464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6366,"num_write_kb":50928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6506","reported_seq":6573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.819454+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:16.819454+0000","last_peered":"2024-09-18T11:48:16.819454+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:16.819454+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6506,"log_dups_size":0,"ondisk_log_size":6506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53245174,"num_objects":6500,"num_object_clones":0,"num_object_copies":26000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6504,"num_write_kb":52012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4366}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6437","reported_seq":6502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.938253+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:16.938253+0000","last_peered":"2024-09-18T11:48:16.938253+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:16.938253+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6437,"log_dups_size":0,"ondisk_log_size":6437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52688118,"num_objects":6432,"num_object_clones":0,"num_object_copies":25728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":51468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6554","reported_seq":6614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.554488+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:15.554488+0000","last_peered":"2024-09-18T11:48:15.554488+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:15.554488+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53619180,"num_objects":6546,"num_object_clones":0,"num_object_copies":26184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6551,"num_write_kb":52385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6290","reported_seq":6376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.512330+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:16.512330+0000","last_peered":"2024-09-18T11:48:16.512330+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:16.512330+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6290,"log_dups_size":0,"ondisk_log_size":6290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51508470,"num_objects":6288,"num_object_clones":0,"num_object_copies":25152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4313}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6494","reported_seq":6552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.602217+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:15.602217+0000","last_peered":"2024-09-18T11:48:15.602217+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:15.602217+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6494,"log_dups_size":0,"ondisk_log_size":6494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53153649,"num_objects":6489,"num_object_clones":0,"num_object_copies":25956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":51916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4426}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6490","reported_seq":6529,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.422362+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:15.422362+0000","last_peered":"2024-09-18T11:48:15.422362+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:15.422362+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53156475,"num_objects":6489,"num_object_clones":0,"num_object_copies":25956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6489,"num_write_kb":51911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4380}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6472","reported_seq":6550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.405138+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:16.405138+0000","last_peered":"2024-09-18T11:48:16.405138+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:16.405138+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53009019,"num_objects":6471,"num_object_clones":0,"num_object_copies":25884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":51767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4417}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6512","reported_seq":6561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.426673+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:15.426673+0000","last_peered":"2024-09-18T11:48:15.426673+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:15.426673+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6512,"log_dups_size":0,"ondisk_log_size":6512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53310710,"num_objects":6508,"num_object_clones":0,"num_object_copies":26032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6510,"num_write_kb":52076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6426","reported_seq":10711,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:15.754389+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:15.754389+0000","last_peered":"2024-09-18T11:48:15.754389+0000","last_clean":"2024-09-18T11:48:15.754389+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:15.754389+0000","last_undegraded":"2024-09-18T11:48:15.754389+0000","last_fullsized":"2024-09-18T11:48:15.754389+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52614390,"num_objects":6423,"num_object_clones":0,"num_object_copies":25692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":51389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6350","reported_seq":6414,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.755735+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:16.755735+0000","last_peered":"2024-09-18T11:48:16.755735+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:16.755735+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6350,"log_dups_size":0,"ondisk_log_size":6350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51975414,"num_objects":6345,"num_object_clones":0,"num_object_copies":25380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6348,"num_write_kb":50772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":845702516,"num_objects":103240,"num_object_clones":0,"num_object_copies":412960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103267,"num_write_kb":826018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2943,"num_bytes_recovered":23871488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1566560256,"data_stored":1566560256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103294,"ondisk_log_size":103294,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722748,"kb_used_data":412404,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93649092,"statfs":{"total":96636764160,"available":95896670208,"internally_reserved":0,"allocated":422301696,"data_stored":422134511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.33300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":16.824000000000002},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.4359999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714964,"kb_used_data":413452,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93656876,"statfs":{"total":96636764160,"available":95904641024,"internally_reserved":0,"allocated":423374848,"data_stored":423207663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.7789999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":18.361000000000001},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":2.3700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.30399999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.2230000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587960,"kb_used_data":290160,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93783880,"statfs":{"total":96636764160,"available":96034693120,"internally_reserved":0,"allocated":297123840,"data_stored":296956655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":14.877000000000001},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.24099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.48799999999999999},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":14.922000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710704,"kb_used_data":412904,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93661136,"statfs":{"total":96636764160,"available":95909003264,"internally_reserved":0,"allocated":422813696,"data_stored":422646511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":9.8100000000000005},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.7430000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.782}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":423665664,"data_stored":423665664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297504768,"data_stored":297504768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":423272448,"data_stored":423272448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":422117376,"data_stored":422117376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:18.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:18.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6604", 'reported_seq': 6647, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:15.512023+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:15.512023+0000', 'last_peered': '2024-09-18T11:48:15.512023+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:15.512023+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6604, 'log_dups_size': 0, 'ondisk_log_size': 6604, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 54099968, 'num_objects': 6604, 'num_object_clones': 0, 'num_object_copies': 26416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6604, 'num_write_kb': 52832, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:18.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:18.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6522", 'reported_seq': 8234, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:16.935924+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:16.935924+0000', 'last_peered': '2024-09-18T11:48:16.935924+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:16.935924+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6522, 'log_dups_size': 0, 'ondisk_log_size': 6522, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53392630, 'num_objects': 6518, 'num_object_clones': 0, 'num_object_copies': 26072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6520, 'num_write_kb': 52156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 825, 'num_bytes_recovered': 6742016, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5239}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:18.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6397", 'reported_seq': 6481, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:16.561164+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:16.561164+0000', 'last_peered': '2024-09-18T11:48:16.561164+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:16.561164+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6397, 'log_dups_size': 0, 'ondisk_log_size': 6397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52378235, 'num_objects': 6394, 'num_object_clones': 0, 'num_object_copies': 25576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6396, 'num_write_kb': 51158, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4340}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6458", 'reported_seq': 6522, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:16.852522+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:16.852522+0000', 'last_peered': '2024-09-18T11:48:16.852522+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:16.852522+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6458, 'log_dups_size': 0, 'ondisk_log_size': 6458, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52868342, 'num_objects': 6454, 'num_object_clones': 0, 'num_object_copies': 25816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6456, 'num_write_kb': 51637, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6416", 'reported_seq': 6473, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:15.508838+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:15.508838+0000', 'last_peered': '2024-09-18T11:48:15.508838+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:15.508838+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6416, 'log_dups_size': 0, 'ondisk_log_size': 6416, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52532470, 'num_objects': 6413, 'num_object_clones': 0, 'num_object_copies': 25652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6414, 'num_write_kb': 51309, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4331}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6366", 'reported_seq': 6411, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:15.571828+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:15.571828+0000', 'last_peered': '2024-09-18T11:48:15.571828+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:15.571828+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6366, 'log_dups_size': 0, 'ondisk_log_size': 6366, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52150272, 'num_objects': 6366, 'num_object_clones': 0, 'num_object_copies': 25464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6366, 'num_write_kb': 50928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4212}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6506", 'reported_seq': 6573, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:16.819454+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:16.819454+0000', 'last_peered': '2024-09-18T11:48:16.819454+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:16.819454+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6506, 'log_dups_size': 0, 'ondisk_log_size': 6506, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53245174, 'num_objects': 6500, 'num_object_clones': 0, 'num_object_copies': 26000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6500, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6504, 'num_write_kb': 52012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4366}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:18.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6437", 'reported_seq': 6502, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:16.938253+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:16.938253+0000', 'last_peered': '2024-09-18T11:48:16.938253+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:16.938253+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6437, 'log_dups_size': 0, 'ondisk_log_size': 6437, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52688118, 'num_objects': 6432, 'num_object_clones': 0, 'num_object_copies': 25728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6435, 'num_write_kb': 51468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4398}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6554", 'reported_seq': 6614, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:15.554488+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:15.554488+0000', 'last_peered': '2024-09-18T11:48:15.554488+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:15.554488+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6554, 'log_dups_size': 0, 'ondisk_log_size': 6554, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53619180, 'num_objects': 6546, 'num_object_clones': 0, 'num_object_copies': 26184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6551, 'num_write_kb': 52385, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4411}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6290", 'reported_seq': 6376, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:16.512330+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:16.512330+0000', 'last_peered': '2024-09-18T11:48:16.512330+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:16.512330+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6290, 'log_dups_size': 0, 'ondisk_log_size': 6290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51508470, 'num_objects': 6288, 'num_object_clones': 0, 'num_object_copies': 25152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6288, 'num_write_kb': 50302, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4313}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6494", 'reported_seq': 6552, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:15.602217+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:15.602217+0000', 'last_peered': '2024-09-18T11:48:15.602217+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:15.602217+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6494, 'log_dups_size': 0, 'ondisk_log_size': 6494, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53153649, 'num_objects': 6489, 'num_object_clones': 0, 'num_object_copies': 25956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6491, 'num_write_kb': 51916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4426}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6490", 'reported_seq': 6529, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:15.422362+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:15.422362+0000', 'last_peered': '2024-09-18T11:48:15.422362+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:15.422362+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6490, 'log_dups_size': 0, 'ondisk_log_size': 6490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53156475, 'num_objects': 6489, 'num_object_clones': 0, 'num_object_copies': 25956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6489, 'num_write_kb': 51911, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4380}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6472", 'reported_seq': 6550, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:16.405138+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:16.405138+0000', 'last_peered': '2024-09-18T11:48:16.405138+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:16.405138+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6472, 'log_dups_size': 0, 'ondisk_log_size': 6472, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53009019, 'num_objects': 6471, 'num_object_clones': 0, 'num_object_copies': 25884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6471, 'num_write_kb': 51767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4417}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:18.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:19.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6512", 'reported_seq': 6561, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:15.426673+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:15.426673+0000', 'last_peered': '2024-09-18T11:48:15.426673+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:15.426673+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6512, 'log_dups_size': 0, 'ondisk_log_size': 6512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53310710, 'num_objects': 6508, 'num_object_clones': 0, 'num_object_copies': 26032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6510, 'num_write_kb': 52076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4350}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:19.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:19.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6350", 'reported_seq': 6414, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:16.755735+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:16.755735+0000', 'last_peered': '2024-09-18T11:48:16.755735+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:16.755735+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6350, 'log_dups_size': 0, 'ondisk_log_size': 6350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51975414, 'num_objects': 6345, 'num_object_clones': 0, 'num_object_copies': 25380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6348, 'num_write_kb': 50772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:19.000 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:19.679 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:19.679 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30511/412960 objects degraded (7.388%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":297,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":103240,"data_bytes":845702516,"bytes_used":2802049024,"bytes_avail":383745007616,"bytes_total":386547056640,"degraded_objects":30511,"degraded_total":412960,"degraded_ratio":0.073883669120495932,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105096,"recovering_keys_per_sec":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"write_bytes_sec":2270491,"read_op_per_sec":0,"write_op_per_sec":277},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:19.715 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:19.715 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:20.281 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:20.281 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:20.312 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-18T11:48:19.916703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":845702516,"num_objects":103240,"num_object_clones":0,"num_object_copies":412960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103267,"num_write_kb":826018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2943,"num_bytes_recovered":23871488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103294,"ondisk_log_size":103294,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736376,"kb_used_data":1528920,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":374750984,"statfs":{"total":386547056640,"available":383745007616,"internally_reserved":0,"allocated":1565614080,"data_stored":1564945340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,7,7,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21585920,"num_objects":2635,"num_object_clones":0,"num_object_copies":10540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2635,"num_write_kb":21080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003746"},"pg_stats":[{"pgid":"1.f","version":"33'6604","reported_seq":6647,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.512023+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:15.512023+0000","last_peered":"2024-09-18T11:48:15.512023+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:15.512023+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54099968,"num_objects":6604,"num_object_clones":0,"num_object_copies":26416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":52832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6522","reported_seq":8234,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.935924+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:16.935924+0000","last_peered":"2024-09-18T11:48:16.935924+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:16.935924+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53392630,"num_objects":6518,"num_object_clones":0,"num_object_copies":26072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6520,"num_write_kb":52156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":825,"num_bytes_recovered":6742016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6397","reported_seq":6481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.561164+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:16.561164+0000","last_peered":"2024-09-18T11:48:16.561164+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:16.561164+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6397,"log_dups_size":0,"ondisk_log_size":6397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52378235,"num_objects":6394,"num_object_clones":0,"num_object_copies":25576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6396,"num_write_kb":51158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6458","reported_seq":6522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.852522+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:16.852522+0000","last_peered":"2024-09-18T11:48:16.852522+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:16.852522+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6458,"log_dups_size":0,"ondisk_log_size":6458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52868342,"num_objects":6454,"num_object_clones":0,"num_object_copies":25816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":51637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6416","reported_seq":6473,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.508838+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:15.508838+0000","last_peered":"2024-09-18T11:48:15.508838+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:15.508838+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52532470,"num_objects":6413,"num_object_clones":0,"num_object_copies":25652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6414,"num_write_kb":51309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6366","reported_seq":6411,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.571828+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:15.571828+0000","last_peered":"2024-09-18T11:48:15.571828+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:15.571828+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52150272,"num_objects":6366,"num_object_clones":0,"num_object_copies":25464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6366,"num_write_kb":50928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6506","reported_seq":6573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.819454+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:16.819454+0000","last_peered":"2024-09-18T11:48:16.819454+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:16.819454+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6506,"log_dups_size":0,"ondisk_log_size":6506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53245174,"num_objects":6500,"num_object_clones":0,"num_object_copies":26000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6504,"num_write_kb":52012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4366}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6437","reported_seq":6502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.938253+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:16.938253+0000","last_peered":"2024-09-18T11:48:16.938253+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:16.938253+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6437,"log_dups_size":0,"ondisk_log_size":6437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52688118,"num_objects":6432,"num_object_clones":0,"num_object_copies":25728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":51468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6554","reported_seq":6614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.554488+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:15.554488+0000","last_peered":"2024-09-18T11:48:15.554488+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:15.554488+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53619180,"num_objects":6546,"num_object_clones":0,"num_object_copies":26184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6551,"num_write_kb":52385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6290","reported_seq":6376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.512330+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:16.512330+0000","last_peered":"2024-09-18T11:48:16.512330+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:16.512330+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6290,"log_dups_size":0,"ondisk_log_size":6290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51508470,"num_objects":6288,"num_object_clones":0,"num_object_copies":25152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4313}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6494","reported_seq":6552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.602217+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:15.602217+0000","last_peered":"2024-09-18T11:48:15.602217+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:15.602217+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6494,"log_dups_size":0,"ondisk_log_size":6494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53153649,"num_objects":6489,"num_object_clones":0,"num_object_copies":25956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":51916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4426}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6490","reported_seq":6529,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.422362+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:15.422362+0000","last_peered":"2024-09-18T11:48:15.422362+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:15.422362+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53156475,"num_objects":6489,"num_object_clones":0,"num_object_copies":25956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6489,"num_write_kb":51911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4380}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6472","reported_seq":6550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.405138+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:16.405138+0000","last_peered":"2024-09-18T11:48:16.405138+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:16.405138+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53009019,"num_objects":6471,"num_object_clones":0,"num_object_copies":25884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":51767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4417}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6512","reported_seq":6561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:15.426673+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:15.426673+0000","last_peered":"2024-09-18T11:48:15.426673+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:15.426673+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6512,"log_dups_size":0,"ondisk_log_size":6512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53310710,"num_objects":6508,"num_object_clones":0,"num_object_copies":26032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6510,"num_write_kb":52076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6426","reported_seq":10711,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:15.754389+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:15.754389+0000","last_peered":"2024-09-18T11:48:15.754389+0000","last_clean":"2024-09-18T11:48:15.754389+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:15.754389+0000","last_undegraded":"2024-09-18T11:48:15.754389+0000","last_fullsized":"2024-09-18T11:48:15.754389+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52614390,"num_objects":6423,"num_object_clones":0,"num_object_copies":25692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":51389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6350","reported_seq":6414,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.755735+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:16.755735+0000","last_peered":"2024-09-18T11:48:16.755735+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:16.755735+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6350,"log_dups_size":0,"ondisk_log_size":6350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51975414,"num_objects":6345,"num_object_clones":0,"num_object_copies":25380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6348,"num_write_kb":50772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":845702516,"num_objects":103240,"num_object_clones":0,"num_object_copies":412960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103267,"num_write_kb":826018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2943,"num_bytes_recovered":23871488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1566560256,"data_stored":1566560256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103294,"ondisk_log_size":103294,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722748,"kb_used_data":412404,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93649092,"statfs":{"total":96636764160,"available":95896670208,"internally_reserved":0,"allocated":422301696,"data_stored":422134511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.33300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":16.824000000000002},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.4359999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714964,"kb_used_data":413452,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93656876,"statfs":{"total":96636764160,"available":95904641024,"internally_reserved":0,"allocated":423374848,"data_stored":423207663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.7789999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":18.361000000000001},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":2.3700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.30399999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.2230000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587960,"kb_used_data":290160,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93783880,"statfs":{"total":96636764160,"available":96034693120,"internally_reserved":0,"allocated":297123840,"data_stored":296956655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":14.877000000000001},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.24099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.48799999999999999},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":14.922000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710704,"kb_used_data":412904,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93661136,"statfs":{"total":96636764160,"available":95909003264,"internally_reserved":0,"allocated":422813696,"data_stored":422646511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":9.8100000000000005},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.7430000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.782}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":423665664,"data_stored":423665664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297504768,"data_stored":297504768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":423272448,"data_stored":423272448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":422117376,"data_stored":422117376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:23.314 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:23.856 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:23.893 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-18T11:48:21.920798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":842417548,"num_objects":102840,"num_object_clones":0,"num_object_copies":411360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102840,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104189,"num_write_kb":828099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2946,"num_bytes_recovered":23871488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104216,"ondisk_log_size":104216,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729596,"kb_used_data":1526280,"kb_used_omap":14,"kb_used_meta":1203249,"kb_avail":374757764,"statfs":{"total":386547056640,"available":383751950336,"internally_reserved":0,"allocated":1562910720,"data_stored":1562241980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232127003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,7,9,8,3,0,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18300952,"num_objects":2235,"num_object_clones":0,"num_object_copies":8940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2235,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3557,"num_write_kb":23161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007149"},"pg_stats":[{"pgid":"1.f","version":"33'6669","reported_seq":6711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.522551+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:20.522551+0000","last_peered":"2024-09-18T11:48:20.522551+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:20.522551+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6669,"log_dups_size":0,"ondisk_log_size":6669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53911552,"num_objects":6581,"num_object_clones":0,"num_object_copies":26324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6669,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6522","reported_seq":8234,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.935924+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:16.935924+0000","last_peered":"2024-09-18T11:48:16.935924+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:16.935924+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53392630,"num_objects":6518,"num_object_clones":0,"num_object_copies":26072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6520,"num_write_kb":52156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":825,"num_bytes_recovered":6742016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6461","reported_seq":6546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.567961+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:21.567961+0000","last_peered":"2024-09-18T11:48:21.567961+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:21.567961+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6461,"log_dups_size":0,"ondisk_log_size":6461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51993235,"num_objects":6348,"num_object_clones":0,"num_object_copies":25392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6348,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6460,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6458","reported_seq":6522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.852522+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:16.852522+0000","last_peered":"2024-09-18T11:48:16.852522+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:16.852522+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6458,"log_dups_size":0,"ondisk_log_size":6458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52868342,"num_objects":6454,"num_object_clones":0,"num_object_copies":25816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":51637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6495","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.525474+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:20.525474+0000","last_peered":"2024-09-18T11:48:20.525474+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:20.525474+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6495,"log_dups_size":0,"ondisk_log_size":6495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52425974,"num_objects":6400,"num_object_clones":0,"num_object_copies":25600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6493,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4319}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6460","reported_seq":6505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.574780+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:20.574780+0000","last_peered":"2024-09-18T11:48:20.574780+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:20.574780+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51822592,"num_objects":6326,"num_object_clones":0,"num_object_copies":25304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6506","reported_seq":6573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.819454+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:16.819454+0000","last_peered":"2024-09-18T11:48:16.819454+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:16.819454+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6506,"log_dups_size":0,"ondisk_log_size":6506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53245174,"num_objects":6500,"num_object_clones":0,"num_object_copies":26000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6504,"num_write_kb":52012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4366}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6437","reported_seq":6502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.938253+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:16.938253+0000","last_peered":"2024-09-18T11:48:16.938253+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:16.938253+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6437,"log_dups_size":0,"ondisk_log_size":6437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52688118,"num_objects":6432,"num_object_clones":0,"num_object_copies":25728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":51468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6649","reported_seq":6709,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.601053+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:20.601053+0000","last_peered":"2024-09-18T11:48:20.601053+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:20.601053+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6649,"log_dups_size":0,"ondisk_log_size":6649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53463532,"num_objects":6527,"num_object_clones":0,"num_object_copies":26108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6646,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6375","reported_seq":6461,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.577111+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:21.577111+0000","last_peered":"2024-09-18T11:48:21.577111+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:21.577111+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6375,"log_dups_size":0,"ondisk_log_size":6375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50943222,"num_objects":6219,"num_object_clones":0,"num_object_copies":24876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6373,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4244}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6576","reported_seq":6639,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.525259+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:20.525259+0000","last_peered":"2024-09-18T11:48:20.525259+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:20.525259+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52891505,"num_objects":6457,"num_object_clones":0,"num_object_copies":25828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6573,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6567","reported_seq":6606,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.521138+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:20.521138+0000","last_peered":"2024-09-18T11:48:20.521138+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:20.521138+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52951675,"num_objects":6464,"num_object_clones":0,"num_object_copies":25856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4355}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6567","reported_seq":6645,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.480545+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:21.480545+0000","last_peered":"2024-09-18T11:48:21.480545+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:21.480545+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52427387,"num_objects":6400,"num_object_clones":0,"num_object_copies":25600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4346}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6608","reported_seq":6658,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.644591+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:20.644591+0000","last_peered":"2024-09-18T11:48:20.644591+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:20.644591+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6608,"log_dups_size":0,"ondisk_log_size":6608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53097718,"num_objects":6482,"num_object_clones":0,"num_object_copies":25928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4324}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6516","reported_seq":10801,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:20.679161+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:20.679161+0000","last_peered":"2024-09-18T11:48:20.679161+0000","last_clean":"2024-09-18T11:48:20.679161+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:20.679161+0000","last_undegraded":"2024-09-18T11:48:20.679161+0000","last_fullsized":"2024-09-18T11:48:20.679161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6516,"log_dups_size":0,"ondisk_log_size":6516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52319478,"num_objects":6387,"num_object_clones":0,"num_object_copies":25548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6514,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6350","reported_seq":6414,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:16.755735+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:16.755735+0000","last_peered":"2024-09-18T11:48:16.755735+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:16.755735+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6350,"log_dups_size":0,"ondisk_log_size":6350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51975414,"num_objects":6345,"num_object_clones":0,"num_object_copies":25380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6348,"num_write_kb":50772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":842417548,"num_objects":102840,"num_object_clones":0,"num_object_copies":411360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102840,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104189,"num_write_kb":828099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2946,"num_bytes_recovered":23871488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1562447872,"data_stored":1562447872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104216,"ondisk_log_size":104216,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708964,"kb_used_data":411080,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93662876,"statfs":{"total":96636764160,"available":95910785024,"internally_reserved":0,"allocated":420945920,"data_stored":420778735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,5,0,0,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.2199999999999998},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.2519999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":7.2850000000000001},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":8.1310000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":6.3419999999999996},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.1620000000000008}]}]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715724,"kb_used_data":410052,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93656116,"statfs":{"total":96636764160,"available":95903862784,"internally_reserved":0,"allocated":419893248,"data_stored":419726063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":5.6630000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.3059999999999992},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":8.9199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.48699999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":5.6159999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594204,"kb_used_data":292244,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93777636,"statfs":{"total":96636764160,"available":96028299264,"internally_reserved":0,"allocated":299257856,"data_stored":299090671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.24299999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":6.5129999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":3.9249999999999998},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.6829999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":2.6339999999999999},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.9850000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710704,"kb_used_data":412904,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93661136,"statfs":{"total":96636764160,"available":95909003264,"internally_reserved":0,"allocated":422813696,"data_stored":422646511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":9.8100000000000005},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.7430000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.782}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":423665664,"data_stored":423665664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298938368,"data_stored":298938368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":419373056,"data_stored":419373056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":420470784,"data_stored":420470784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:23.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:24.425 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:24.425 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:24.448 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-18T11:48:23.923153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":839738764,"num_objects":102513,"num_object_clones":0,"num_object_copies":410052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104574,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3015,"num_bytes_recovered":24428544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104601,"ondisk_log_size":104601,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731332,"kb_used_data":1523852,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":374756028,"statfs":{"total":386547056640,"available":383750172672,"internally_reserved":0,"allocated":1560424448,"data_stored":1559755708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,8,8,9,2,0,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7938072,"num_objects":970,"num_object_clones":0,"num_object_copies":3880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3004,"num_write_kb":15889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008804"},"pg_stats":[{"pgid":"1.f","version":"33'6669","reported_seq":6711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.522551+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:20.522551+0000","last_peered":"2024-09-18T11:48:20.522551+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:20.522551+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6669,"log_dups_size":0,"ondisk_log_size":6669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53911552,"num_objects":6581,"num_object_clones":0,"num_object_copies":26324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6669,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6591","reported_seq":8439,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.926688+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:21.926688+0000","last_peered":"2024-09-18T11:48:21.926688+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:21.926688+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6591,"log_dups_size":0,"ondisk_log_size":6591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52958454,"num_objects":6465,"num_object_clones":0,"num_object_copies":25860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6589,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":893,"num_bytes_recovered":7299072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6461","reported_seq":6546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.567961+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:21.567961+0000","last_peered":"2024-09-18T11:48:21.567961+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:21.567961+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6461,"log_dups_size":0,"ondisk_log_size":6461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51993235,"num_objects":6348,"num_object_clones":0,"num_object_copies":25392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6348,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6460,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6530","reported_seq":6594,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.848782+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:21.848782+0000","last_peered":"2024-09-18T11:48:21.848782+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:21.848782+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52393206,"num_objects":6396,"num_object_clones":0,"num_object_copies":25584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6495","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.525474+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:20.525474+0000","last_peered":"2024-09-18T11:48:20.525474+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:20.525474+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6495,"log_dups_size":0,"ondisk_log_size":6495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52425974,"num_objects":6400,"num_object_clones":0,"num_object_copies":25600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6493,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4319}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6460","reported_seq":6505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.574780+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:20.574780+0000","last_peered":"2024-09-18T11:48:20.574780+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:20.574780+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51822592,"num_objects":6326,"num_object_clones":0,"num_object_copies":25304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6585","reported_seq":6652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.818190+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:21.818190+0000","last_peered":"2024-09-18T11:48:21.818190+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:21.818190+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6585,"log_dups_size":0,"ondisk_log_size":6585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52679926,"num_objects":6431,"num_object_clones":0,"num_object_copies":25724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6583,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4297}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6505","reported_seq":6570,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.801270+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:21.801270+0000","last_peered":"2024-09-18T11:48:21.801270+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:21.801270+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6505,"log_dups_size":0,"ondisk_log_size":6505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52180214,"num_objects":6370,"num_object_clones":0,"num_object_copies":25480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6503,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4336}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6649","reported_seq":6709,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.601053+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:20.601053+0000","last_peered":"2024-09-18T11:48:20.601053+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:20.601053+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6649,"log_dups_size":0,"ondisk_log_size":6649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53463532,"num_objects":6527,"num_object_clones":0,"num_object_copies":26108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6646,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6375","reported_seq":6461,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.577111+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:21.577111+0000","last_peered":"2024-09-18T11:48:21.577111+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:21.577111+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6375,"log_dups_size":0,"ondisk_log_size":6375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50943222,"num_objects":6219,"num_object_clones":0,"num_object_copies":24876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6373,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4244}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6576","reported_seq":6639,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.525259+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:20.525259+0000","last_peered":"2024-09-18T11:48:20.525259+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:20.525259+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52891505,"num_objects":6457,"num_object_clones":0,"num_object_copies":25828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6573,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6567","reported_seq":6606,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.521138+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:20.521138+0000","last_peered":"2024-09-18T11:48:20.521138+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:20.521138+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52951675,"num_objects":6464,"num_object_clones":0,"num_object_copies":25856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4355}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6567","reported_seq":6645,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.480545+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:21.480545+0000","last_peered":"2024-09-18T11:48:21.480545+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:21.480545+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52427387,"num_objects":6400,"num_object_clones":0,"num_object_copies":25600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4346}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6608","reported_seq":6658,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.644591+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:20.644591+0000","last_peered":"2024-09-18T11:48:20.644591+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:20.644591+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6608,"log_dups_size":0,"ondisk_log_size":6608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53097718,"num_objects":6482,"num_object_clones":0,"num_object_copies":25928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4324}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6516","reported_seq":10801,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:20.679161+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:20.679161+0000","last_peered":"2024-09-18T11:48:20.679161+0000","last_clean":"2024-09-18T11:48:20.679161+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:20.679161+0000","last_undegraded":"2024-09-18T11:48:20.679161+0000","last_fullsized":"2024-09-18T11:48:20.679161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6516,"log_dups_size":0,"ondisk_log_size":6516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52319478,"num_objects":6387,"num_object_clones":0,"num_object_copies":25548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6514,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6447","reported_seq":6513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.939643+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:21.939643+0000","last_peered":"2024-09-18T11:48:21.939643+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:21.939643+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51279094,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6445,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":839738764,"num_objects":102513,"num_object_clones":0,"num_object_copies":410052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104574,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3015,"num_bytes_recovered":24428544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1557774336,"data_stored":1557774336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104601,"ondisk_log_size":104601,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708964,"kb_used_data":411080,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93662876,"statfs":{"total":96636764160,"available":95910785024,"internally_reserved":0,"allocated":420945920,"data_stored":420778735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,5,0,0,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.2199999999999998},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.2519999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":7.2850000000000001},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":8.1310000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":6.3419999999999996},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.1620000000000008}]}]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715724,"kb_used_data":410052,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93656116,"statfs":{"total":96636764160,"available":95903862784,"internally_reserved":0,"allocated":419893248,"data_stored":419726063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":5.6630000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.3059999999999992},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":8.9199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.48699999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":5.6159999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594204,"kb_used_data":292244,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93777636,"statfs":{"total":96636764160,"available":96028299264,"internally_reserved":0,"allocated":299257856,"data_stored":299090671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.24299999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":6.5129999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":3.9249999999999998},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.6829999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":2.6339999999999999},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.9850000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712440,"kb_used_data":410476,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93659400,"statfs":{"total":96636764160,"available":95907225600,"internally_reserved":0,"allocated":420327424,"data_stored":420160239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":4.0140000000000002},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":8.0079999999999991}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.7570000000000001},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":9.0489999999999995},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":8.0449999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":418992128,"data_stored":418992128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298938368,"data_stored":298938368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":419373056,"data_stored":419373056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":420470784,"data_stored":420470784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:24.450 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:24.450 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6669", 'reported_seq': 6711, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:20.522551+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:20.522551+0000', 'last_peered': '2024-09-18T11:48:20.522551+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:20.522551+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6669, 'log_dups_size': 0, 'ondisk_log_size': 6669, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53911552, 'num_objects': 6581, 'num_object_clones': 0, 'num_object_copies': 26324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6669, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4451}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:24.450 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:24.450 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6591", 'reported_seq': 8439, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:21.926688+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:21.926688+0000', 'last_peered': '2024-09-18T11:48:21.926688+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:21.926688+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6591, 'log_dups_size': 0, 'ondisk_log_size': 6591, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52958454, 'num_objects': 6465, 'num_object_clones': 0, 'num_object_copies': 25860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6589, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 893, 'num_bytes_recovered': 7299072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5254}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:24.450 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:24.450 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6461", 'reported_seq': 6546, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:21.567961+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:21.567961+0000', 'last_peered': '2024-09-18T11:48:21.567961+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:21.567961+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6461, 'log_dups_size': 0, 'ondisk_log_size': 6461, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51993235, 'num_objects': 6348, 'num_object_clones': 0, 'num_object_copies': 25392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6348, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6460, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4294}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:24.450 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:24.450 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6530", 'reported_seq': 6594, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:21.848782+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:21.848782+0000', 'last_peered': '2024-09-18T11:48:21.848782+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:21.848782+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6530, 'log_dups_size': 0, 'ondisk_log_size': 6530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52393206, 'num_objects': 6396, 'num_object_clones': 0, 'num_object_copies': 25584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6396, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6528, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4282}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:24.450 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:24.450 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6495", 'reported_seq': 6554, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:20.525474+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:20.525474+0000', 'last_peered': '2024-09-18T11:48:20.525474+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:20.525474+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6495, 'log_dups_size': 0, 'ondisk_log_size': 6495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52425974, 'num_objects': 6400, 'num_object_clones': 0, 'num_object_copies': 25600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6493, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4319}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:24.451 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:24.451 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6460", 'reported_seq': 6505, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:20.574780+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:20.574780+0000', 'last_peered': '2024-09-18T11:48:20.574780+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:20.574780+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6460, 'log_dups_size': 0, 'ondisk_log_size': 6460, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51822592, 'num_objects': 6326, 'num_object_clones': 0, 'num_object_copies': 25304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6460, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4172}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:24.451 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:24.451 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6585", 'reported_seq': 6652, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:21.818190+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:21.818190+0000', 'last_peered': '2024-09-18T11:48:21.818190+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:21.818190+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6585, 'log_dups_size': 0, 'ondisk_log_size': 6585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52679926, 'num_objects': 6431, 'num_object_clones': 0, 'num_object_copies': 25724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6583, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4297}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:24.451 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:24.451 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6505", 'reported_seq': 6570, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:21.801270+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:21.801270+0000', 'last_peered': '2024-09-18T11:48:21.801270+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:21.801270+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6505, 'log_dups_size': 0, 'ondisk_log_size': 6505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52180214, 'num_objects': 6370, 'num_object_clones': 0, 'num_object_copies': 25480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6503, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4336}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:24.451 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:24.451 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6649", 'reported_seq': 6709, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:20.601053+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:20.601053+0000', 'last_peered': '2024-09-18T11:48:20.601053+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:20.601053+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6649, 'log_dups_size': 0, 'ondisk_log_size': 6649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53463532, 'num_objects': 6527, 'num_object_clones': 0, 'num_object_copies': 26108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6646, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4392}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:24.451 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:24.451 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6375", 'reported_seq': 6461, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:21.577111+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:21.577111+0000', 'last_peered': '2024-09-18T11:48:21.577111+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:21.577111+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6375, 'log_dups_size': 0, 'ondisk_log_size': 6375, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50943222, 'num_objects': 6219, 'num_object_clones': 0, 'num_object_copies': 24876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6373, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4244}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:24.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:24.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6576", 'reported_seq': 6639, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:20.525259+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:20.525259+0000', 'last_peered': '2024-09-18T11:48:20.525259+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:20.525259+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6576, 'log_dups_size': 0, 'ondisk_log_size': 6576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52891505, 'num_objects': 6457, 'num_object_clones': 0, 'num_object_copies': 25828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2064, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6573, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4396}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:24.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:24.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6567", 'reported_seq': 6606, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:20.521138+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:20.521138+0000', 'last_peered': '2024-09-18T11:48:20.521138+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:20.521138+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6567, 'log_dups_size': 0, 'ondisk_log_size': 6567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52951675, 'num_objects': 6464, 'num_object_clones': 0, 'num_object_copies': 25856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6566, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4355}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:24.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:24.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6567", 'reported_seq': 6645, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:21.480545+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:21.480545+0000', 'last_peered': '2024-09-18T11:48:21.480545+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:21.480545+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6567, 'log_dups_size': 0, 'ondisk_log_size': 6567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52427387, 'num_objects': 6400, 'num_object_clones': 0, 'num_object_copies': 25600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6566, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:24.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:24.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6608", 'reported_seq': 6658, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:20.644591+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:20.644591+0000', 'last_peered': '2024-09-18T11:48:20.644591+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:20.644591+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6608, 'log_dups_size': 0, 'ondisk_log_size': 6608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53097718, 'num_objects': 6482, 'num_object_clones': 0, 'num_object_copies': 25928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6606, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4324}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:24.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:24.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6447", 'reported_seq': 6513, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:21.939643+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:21.939643+0000', 'last_peered': '2024-09-18T11:48:21.939643+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:21.939643+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6447, 'log_dups_size': 0, 'ondisk_log_size': 6447, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51279094, 'num_objects': 6260, 'num_object_clones': 0, 'num_object_copies': 25040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2014, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6445, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4248}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:24.453 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:25.093 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:25.093 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30439/410052 objects degraded (7.423%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":303,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":102513,"data_bytes":839738764,"bytes_used":2796883968,"bytes_avail":383750172672,"bytes_total":386547056640,"degraded_objects":30439,"degraded_total":410052,"degraded_ratio":0.074232048618224036,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":116650,"recovering_keys_per_sec":0,"num_objects_recovered":175,"num_bytes_recovered":1400832,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":1354867,"read_op_per_sec":0,"write_op_per_sec":250},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:25.117 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:25.117 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:25.703 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:25.704 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:25.723 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-18T11:48:23.923153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":839738764,"num_objects":102513,"num_object_clones":0,"num_object_copies":410052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104574,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3015,"num_bytes_recovered":24428544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104601,"ondisk_log_size":104601,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731332,"kb_used_data":1523852,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":374756028,"statfs":{"total":386547056640,"available":383750172672,"internally_reserved":0,"allocated":1560424448,"data_stored":1559755708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,8,8,9,2,0,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7938072,"num_objects":970,"num_object_clones":0,"num_object_copies":3880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3004,"num_write_kb":15889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008804"},"pg_stats":[{"pgid":"1.f","version":"33'6669","reported_seq":6711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.522551+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:20.522551+0000","last_peered":"2024-09-18T11:48:20.522551+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:20.522551+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6669,"log_dups_size":0,"ondisk_log_size":6669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53911552,"num_objects":6581,"num_object_clones":0,"num_object_copies":26324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6669,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6591","reported_seq":8439,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.926688+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:21.926688+0000","last_peered":"2024-09-18T11:48:21.926688+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:21.926688+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6591,"log_dups_size":0,"ondisk_log_size":6591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52958454,"num_objects":6465,"num_object_clones":0,"num_object_copies":25860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6589,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":893,"num_bytes_recovered":7299072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6461","reported_seq":6546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.567961+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:21.567961+0000","last_peered":"2024-09-18T11:48:21.567961+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:21.567961+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6461,"log_dups_size":0,"ondisk_log_size":6461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51993235,"num_objects":6348,"num_object_clones":0,"num_object_copies":25392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6348,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6460,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6530","reported_seq":6594,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.848782+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:21.848782+0000","last_peered":"2024-09-18T11:48:21.848782+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:21.848782+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52393206,"num_objects":6396,"num_object_clones":0,"num_object_copies":25584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6495","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.525474+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:20.525474+0000","last_peered":"2024-09-18T11:48:20.525474+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:20.525474+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6495,"log_dups_size":0,"ondisk_log_size":6495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52425974,"num_objects":6400,"num_object_clones":0,"num_object_copies":25600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6493,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4319}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6460","reported_seq":6505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.574780+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:20.574780+0000","last_peered":"2024-09-18T11:48:20.574780+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:20.574780+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51822592,"num_objects":6326,"num_object_clones":0,"num_object_copies":25304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6585","reported_seq":6652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.818190+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:21.818190+0000","last_peered":"2024-09-18T11:48:21.818190+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:21.818190+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6585,"log_dups_size":0,"ondisk_log_size":6585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52679926,"num_objects":6431,"num_object_clones":0,"num_object_copies":25724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6583,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4297}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6505","reported_seq":6570,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.801270+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:21.801270+0000","last_peered":"2024-09-18T11:48:21.801270+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:21.801270+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6505,"log_dups_size":0,"ondisk_log_size":6505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52180214,"num_objects":6370,"num_object_clones":0,"num_object_copies":25480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6503,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4336}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6649","reported_seq":6709,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.601053+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:20.601053+0000","last_peered":"2024-09-18T11:48:20.601053+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:20.601053+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6649,"log_dups_size":0,"ondisk_log_size":6649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53463532,"num_objects":6527,"num_object_clones":0,"num_object_copies":26108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6646,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6375","reported_seq":6461,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.577111+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:21.577111+0000","last_peered":"2024-09-18T11:48:21.577111+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:21.577111+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6375,"log_dups_size":0,"ondisk_log_size":6375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50943222,"num_objects":6219,"num_object_clones":0,"num_object_copies":24876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6373,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4244}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6576","reported_seq":6639,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.525259+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:20.525259+0000","last_peered":"2024-09-18T11:48:20.525259+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:20.525259+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52891505,"num_objects":6457,"num_object_clones":0,"num_object_copies":25828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6573,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6567","reported_seq":6606,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.521138+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:20.521138+0000","last_peered":"2024-09-18T11:48:20.521138+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:20.521138+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52951675,"num_objects":6464,"num_object_clones":0,"num_object_copies":25856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4355}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6567","reported_seq":6645,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.480545+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:21.480545+0000","last_peered":"2024-09-18T11:48:21.480545+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:21.480545+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52427387,"num_objects":6400,"num_object_clones":0,"num_object_copies":25600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4346}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6608","reported_seq":6658,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:20.644591+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:20.644591+0000","last_peered":"2024-09-18T11:48:20.644591+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:20.644591+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6608,"log_dups_size":0,"ondisk_log_size":6608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53097718,"num_objects":6482,"num_object_clones":0,"num_object_copies":25928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4324}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6516","reported_seq":10801,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:20.679161+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:20.679161+0000","last_peered":"2024-09-18T11:48:20.679161+0000","last_clean":"2024-09-18T11:48:20.679161+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:20.679161+0000","last_undegraded":"2024-09-18T11:48:20.679161+0000","last_fullsized":"2024-09-18T11:48:20.679161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6516,"log_dups_size":0,"ondisk_log_size":6516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52319478,"num_objects":6387,"num_object_clones":0,"num_object_copies":25548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6514,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6447","reported_seq":6513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:21.939643+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:21.939643+0000","last_peered":"2024-09-18T11:48:21.939643+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:21.939643+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51279094,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6445,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":839738764,"num_objects":102513,"num_object_clones":0,"num_object_copies":410052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104574,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3015,"num_bytes_recovered":24428544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1557774336,"data_stored":1557774336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104601,"ondisk_log_size":104601,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708964,"kb_used_data":411080,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93662876,"statfs":{"total":96636764160,"available":95910785024,"internally_reserved":0,"allocated":420945920,"data_stored":420778735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,5,0,0,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.2199999999999998},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.2519999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":7.2850000000000001},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":8.1310000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":6.3419999999999996},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.1620000000000008}]}]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715724,"kb_used_data":410052,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93656116,"statfs":{"total":96636764160,"available":95903862784,"internally_reserved":0,"allocated":419893248,"data_stored":419726063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":5.6630000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.3059999999999992},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":8.9199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.48699999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":5.6159999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594204,"kb_used_data":292244,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93777636,"statfs":{"total":96636764160,"available":96028299264,"internally_reserved":0,"allocated":299257856,"data_stored":299090671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.24299999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":6.5129999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":3.9249999999999998},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.6829999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":2.6339999999999999},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.9850000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712440,"kb_used_data":410476,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93659400,"statfs":{"total":96636764160,"available":95907225600,"internally_reserved":0,"allocated":420327424,"data_stored":420160239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":4.0140000000000002},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":8.0079999999999991}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.7570000000000001},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":9.0489999999999995},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":8.0449999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":418992128,"data_stored":418992128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298938368,"data_stored":298938368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":419373056,"data_stored":419373056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":420470784,"data_stored":420470784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:28.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:29.245 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:29.246 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:29.272 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-18T11:48:27.924055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":829556108,"num_objects":101270,"num_object_clones":0,"num_object_copies":405080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101270,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105817,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3090,"num_bytes_recovered":25042944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105844,"ondisk_log_size":105844,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731852,"kb_used_data":1507744,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374755508,"statfs":{"total":386547056640,"available":383749640192,"internally_reserved":0,"allocated":1543929856,"data_stored":1543261116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,1,16,12,9,2,0,0,0,0,0,1,7],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10960872,"num_objects":-1337,"num_object_clones":0,"num_object_copies":-5348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3183,"num_write_kb":7377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1613824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008502"},"pg_stats":[{"pgid":"1.f","version":"33'6745","reported_seq":6788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.606033+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:25.606033+0000","last_peered":"2024-09-18T11:48:25.606033+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:25.606033+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53288960,"num_objects":6505,"num_object_clones":0,"num_object_copies":26020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6745,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6661","reported_seq":8659,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.996691+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:26.996691+0000","last_peered":"2024-09-18T11:48:26.996691+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:26.996691+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6661,"log_dups_size":0,"ondisk_log_size":6661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52385014,"num_objects":6395,"num_object_clones":0,"num_object_copies":25580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6659,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":7913472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5259}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6541","reported_seq":6626,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.526526+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:26.526526+0000","last_peered":"2024-09-18T11:48:26.526526+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:26.526526+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6541,"log_dups_size":0,"ondisk_log_size":6541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51337875,"num_objects":6268,"num_object_clones":0,"num_object_copies":25072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6540,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6608","reported_seq":6672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.915120+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:26.915120+0000","last_peered":"2024-09-18T11:48:26.915120+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:26.915120+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6608,"log_dups_size":0,"ondisk_log_size":6608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51754230,"num_objects":6318,"num_object_clones":0,"num_object_copies":25272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6580","reported_seq":6639,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.636595+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:25.636595+0000","last_peered":"2024-09-18T11:48:25.636595+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:25.636595+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6580,"log_dups_size":0,"ondisk_log_size":6580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51729654,"num_objects":6315,"num_object_clones":0,"num_object_copies":25260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6578,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6534","reported_seq":6579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.413282+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:25.413282+0000","last_peered":"2024-09-18T11:48:25.413282+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:25.413282+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51216384,"num_objects":6252,"num_object_clones":0,"num_object_copies":25008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6660","reported_seq":6727,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.919821+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:26.919821+0000","last_peered":"2024-09-18T11:48:26.919821+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:26.919821+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52065526,"num_objects":6356,"num_object_clones":0,"num_object_copies":25424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6658,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4222}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6576","reported_seq":6641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.913656+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:26.913656+0000","last_peered":"2024-09-18T11:48:26.913656+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:26.913656+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51598582,"num_objects":6299,"num_object_clones":0,"num_object_copies":25196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6574,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6727","reported_seq":6787,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.658916+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:25.658916+0000","last_peered":"2024-09-18T11:48:25.658916+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:25.658916+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52824556,"num_objects":6449,"num_object_clones":0,"num_object_copies":25796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6459","reported_seq":6545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.574355+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:26.574355+0000","last_peered":"2024-09-18T11:48:26.574355+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:26.574355+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6459,"log_dups_size":0,"ondisk_log_size":6459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50255094,"num_objects":6135,"num_object_clones":0,"num_object_copies":24540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6457,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6663","reported_seq":6726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.568939+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:25.568939+0000","last_peered":"2024-09-18T11:48:25.568939+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:25.568939+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52178801,"num_objects":6370,"num_object_clones":0,"num_object_copies":25480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6660,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6641","reported_seq":6679,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.620623+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:25.620623+0000","last_peered":"2024-09-18T11:48:25.620623+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:25.620623+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52345467,"num_objects":6390,"num_object_clones":0,"num_object_copies":25560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6640,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4281}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6641","reported_seq":6719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.526479+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:26.526479+0000","last_peered":"2024-09-18T11:48:26.526479+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:26.526479+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51821179,"num_objects":6326,"num_object_clones":0,"num_object_copies":25304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6640,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4272}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6694","reported_seq":6745,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.602953+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:25.602953+0000","last_peered":"2024-09-18T11:48:25.602953+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:25.602953+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52393206,"num_objects":6396,"num_object_clones":0,"num_object_copies":25584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4238}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6601","reported_seq":10886,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:25.698379+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:25.698379+0000","last_peered":"2024-09-18T11:48:25.698379+0000","last_clean":"2024-09-18T11:48:25.698379+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:25.698379+0000","last_undegraded":"2024-09-18T11:48:25.698379+0000","last_fullsized":"2024-09-18T11:48:25.698379+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6601,"log_dups_size":0,"ondisk_log_size":6601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51623158,"num_objects":6302,"num_object_clones":0,"num_object_copies":25208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6599,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6513","reported_seq":6579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:27.000572+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:27.000572+0000","last_peered":"2024-09-18T11:48:27.000572+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:27.000572+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6513,"log_dups_size":0,"ondisk_log_size":6513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50738422,"num_objects":6194,"num_object_clones":0,"num_object_copies":24776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":829556108,"num_objects":101270,"num_object_clones":0,"num_object_copies":405080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101270,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105817,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3090,"num_bytes_recovered":25042944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1542180864,"data_stored":1542180864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105844,"ondisk_log_size":105844,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708192,"kb_used_data":406168,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93663648,"statfs":{"total":96636764160,"available":95911575552,"internally_reserved":0,"allocated":415916032,"data_stored":415748847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,7,2,0,0,0,0,0,0,0,7],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.1340000000000003},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.1619999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":8.4600000000000009},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":14.118}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":19.850999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":19.167999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714980,"kb_used_data":405148,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93656860,"statfs":{"total":96636764160,"available":95904624640,"internally_reserved":0,"allocated":414871552,"data_stored":414704367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,4,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":4.0049999999999999},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.89300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":4.6989999999999998},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":0.81599999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.91800000000000004},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.83499999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598020,"kb_used_data":291892,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93773820,"statfs":{"total":96636764160,"available":96024391680,"internally_reserved":0,"allocated":298897408,"data_stored":298730223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":2.7999999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":13.847},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.2290000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":5.7709999999999999},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":13.888999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710660,"kb_used_data":404536,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93661180,"statfs":{"total":96636764160,"available":95909048320,"internally_reserved":0,"allocated":414244864,"data_stored":414077679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":1.325},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":3.6379999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.542},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.5170000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.0659999999999998},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":14.198}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":413937664,"data_stored":413937664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298606592,"data_stored":298606592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":414318592,"data_stored":414318592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":415318016,"data_stored":415318016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:29.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:29.802 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:29.803 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:29.825 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-18T11:48:27.924055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":829556108,"num_objects":101270,"num_object_clones":0,"num_object_copies":405080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101270,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105817,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3090,"num_bytes_recovered":25042944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105844,"ondisk_log_size":105844,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731852,"kb_used_data":1507744,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374755508,"statfs":{"total":386547056640,"available":383749640192,"internally_reserved":0,"allocated":1543929856,"data_stored":1543261116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,1,16,12,9,2,0,0,0,0,0,1,7],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10960872,"num_objects":-1337,"num_object_clones":0,"num_object_copies":-5348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3183,"num_write_kb":7377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1613824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008502"},"pg_stats":[{"pgid":"1.f","version":"33'6745","reported_seq":6788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.606033+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:25.606033+0000","last_peered":"2024-09-18T11:48:25.606033+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:25.606033+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53288960,"num_objects":6505,"num_object_clones":0,"num_object_copies":26020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6745,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6661","reported_seq":8659,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.996691+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:26.996691+0000","last_peered":"2024-09-18T11:48:26.996691+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:26.996691+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6661,"log_dups_size":0,"ondisk_log_size":6661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52385014,"num_objects":6395,"num_object_clones":0,"num_object_copies":25580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6659,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":7913472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5259}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6541","reported_seq":6626,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.526526+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:26.526526+0000","last_peered":"2024-09-18T11:48:26.526526+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:26.526526+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6541,"log_dups_size":0,"ondisk_log_size":6541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51337875,"num_objects":6268,"num_object_clones":0,"num_object_copies":25072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6540,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6608","reported_seq":6672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.915120+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:26.915120+0000","last_peered":"2024-09-18T11:48:26.915120+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:26.915120+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6608,"log_dups_size":0,"ondisk_log_size":6608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51754230,"num_objects":6318,"num_object_clones":0,"num_object_copies":25272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6580","reported_seq":6639,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.636595+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:25.636595+0000","last_peered":"2024-09-18T11:48:25.636595+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:25.636595+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6580,"log_dups_size":0,"ondisk_log_size":6580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51729654,"num_objects":6315,"num_object_clones":0,"num_object_copies":25260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6578,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6534","reported_seq":6579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.413282+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:25.413282+0000","last_peered":"2024-09-18T11:48:25.413282+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:25.413282+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51216384,"num_objects":6252,"num_object_clones":0,"num_object_copies":25008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6660","reported_seq":6727,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.919821+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:26.919821+0000","last_peered":"2024-09-18T11:48:26.919821+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:26.919821+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52065526,"num_objects":6356,"num_object_clones":0,"num_object_copies":25424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6658,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4222}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6576","reported_seq":6641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.913656+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:26.913656+0000","last_peered":"2024-09-18T11:48:26.913656+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:26.913656+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51598582,"num_objects":6299,"num_object_clones":0,"num_object_copies":25196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6574,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6727","reported_seq":6787,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.658916+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:25.658916+0000","last_peered":"2024-09-18T11:48:25.658916+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:25.658916+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52824556,"num_objects":6449,"num_object_clones":0,"num_object_copies":25796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6459","reported_seq":6545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.574355+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:26.574355+0000","last_peered":"2024-09-18T11:48:26.574355+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:26.574355+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6459,"log_dups_size":0,"ondisk_log_size":6459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50255094,"num_objects":6135,"num_object_clones":0,"num_object_copies":24540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6457,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6663","reported_seq":6726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.568939+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:25.568939+0000","last_peered":"2024-09-18T11:48:25.568939+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:25.568939+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52178801,"num_objects":6370,"num_object_clones":0,"num_object_copies":25480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6660,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6641","reported_seq":6679,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.620623+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:25.620623+0000","last_peered":"2024-09-18T11:48:25.620623+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:25.620623+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52345467,"num_objects":6390,"num_object_clones":0,"num_object_copies":25560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6640,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4281}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6641","reported_seq":6719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.526479+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:26.526479+0000","last_peered":"2024-09-18T11:48:26.526479+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:26.526479+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51821179,"num_objects":6326,"num_object_clones":0,"num_object_copies":25304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6640,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4272}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6694","reported_seq":6745,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.602953+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:25.602953+0000","last_peered":"2024-09-18T11:48:25.602953+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:25.602953+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52393206,"num_objects":6396,"num_object_clones":0,"num_object_copies":25584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4238}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6601","reported_seq":10886,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:25.698379+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:25.698379+0000","last_peered":"2024-09-18T11:48:25.698379+0000","last_clean":"2024-09-18T11:48:25.698379+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:25.698379+0000","last_undegraded":"2024-09-18T11:48:25.698379+0000","last_fullsized":"2024-09-18T11:48:25.698379+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6601,"log_dups_size":0,"ondisk_log_size":6601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51623158,"num_objects":6302,"num_object_clones":0,"num_object_copies":25208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6599,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6513","reported_seq":6579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:27.000572+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:27.000572+0000","last_peered":"2024-09-18T11:48:27.000572+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:27.000572+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6513,"log_dups_size":0,"ondisk_log_size":6513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50738422,"num_objects":6194,"num_object_clones":0,"num_object_copies":24776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":829556108,"num_objects":101270,"num_object_clones":0,"num_object_copies":405080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101270,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105817,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3090,"num_bytes_recovered":25042944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1542180864,"data_stored":1542180864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105844,"ondisk_log_size":105844,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708192,"kb_used_data":406168,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93663648,"statfs":{"total":96636764160,"available":95911575552,"internally_reserved":0,"allocated":415916032,"data_stored":415748847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,7,2,0,0,0,0,0,0,0,7],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.1340000000000003},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.1619999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":8.4600000000000009},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":14.118}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":19.850999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":19.167999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714980,"kb_used_data":405148,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93656860,"statfs":{"total":96636764160,"available":95904624640,"internally_reserved":0,"allocated":414871552,"data_stored":414704367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,4,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":4.0049999999999999},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.89300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":4.6989999999999998},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":0.81599999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.91800000000000004},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.83499999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598020,"kb_used_data":291892,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93773820,"statfs":{"total":96636764160,"available":96024391680,"internally_reserved":0,"allocated":298897408,"data_stored":298730223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":2.7999999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":13.847},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.2290000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":5.7709999999999999},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":13.888999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710660,"kb_used_data":404536,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93661180,"statfs":{"total":96636764160,"available":95909048320,"internally_reserved":0,"allocated":414244864,"data_stored":414077679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":1.325},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":3.6379999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.542},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.5170000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.0659999999999998},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":14.198}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":413937664,"data_stored":413937664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298606592,"data_stored":298606592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":414318592,"data_stored":414318592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":415318016,"data_stored":415318016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:29.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:29.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6745", 'reported_seq': 6788, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:25.606033+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:25.606033+0000', 'last_peered': '2024-09-18T11:48:25.606033+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:25.606033+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6745, 'log_dups_size': 0, 'ondisk_log_size': 6745, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53288960, 'num_objects': 6505, 'num_object_clones': 0, 'num_object_copies': 26020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6505, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6745, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4375}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:29.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:29.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6661", 'reported_seq': 8659, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:26.996691+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:26.996691+0000', 'last_peered': '2024-09-18T11:48:26.996691+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:26.996691+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6661, 'log_dups_size': 0, 'ondisk_log_size': 6661, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52385014, 'num_objects': 6395, 'num_object_clones': 0, 'num_object_copies': 25580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6659, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 968, 'num_bytes_recovered': 7913472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5259}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:29.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:29.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6541", 'reported_seq': 6626, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:26.526526+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:26.526526+0000', 'last_peered': '2024-09-18T11:48:26.526526+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:26.526526+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6541, 'log_dups_size': 0, 'ondisk_log_size': 6541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51337875, 'num_objects': 6268, 'num_object_clones': 0, 'num_object_copies': 25072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6268, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6540, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4214}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:29.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:29.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6608", 'reported_seq': 6672, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:26.915120+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:26.915120+0000', 'last_peered': '2024-09-18T11:48:26.915120+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:26.915120+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6608, 'log_dups_size': 0, 'ondisk_log_size': 6608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51754230, 'num_objects': 6318, 'num_object_clones': 0, 'num_object_copies': 25272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6606, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4204}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:29.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:29.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6580", 'reported_seq': 6639, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:25.636595+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:25.636595+0000', 'last_peered': '2024-09-18T11:48:25.636595+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:25.636595+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6580, 'log_dups_size': 0, 'ondisk_log_size': 6580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51729654, 'num_objects': 6315, 'num_object_clones': 0, 'num_object_copies': 25260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6578, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4234}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:29.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:29.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6534", 'reported_seq': 6579, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:25.413282+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:25.413282+0000', 'last_peered': '2024-09-18T11:48:25.413282+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:25.413282+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6534, 'log_dups_size': 0, 'ondisk_log_size': 6534, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51216384, 'num_objects': 6252, 'num_object_clones': 0, 'num_object_copies': 25008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6534, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4098}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:29.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:29.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6660", 'reported_seq': 6727, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:26.919821+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:26.919821+0000', 'last_peered': '2024-09-18T11:48:26.919821+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:26.919821+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6660, 'log_dups_size': 0, 'ondisk_log_size': 6660, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52065526, 'num_objects': 6356, 'num_object_clones': 0, 'num_object_copies': 25424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6658, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4222}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:29.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:29.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6576", 'reported_seq': 6641, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:26.913656+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:26.913656+0000', 'last_peered': '2024-09-18T11:48:26.913656+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:26.913656+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6576, 'log_dups_size': 0, 'ondisk_log_size': 6576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51598582, 'num_objects': 6299, 'num_object_clones': 0, 'num_object_copies': 25196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6574, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4265}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:29.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:29.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6727", 'reported_seq': 6787, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:25.658916+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:25.658916+0000', 'last_peered': '2024-09-18T11:48:25.658916+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:25.658916+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6727, 'log_dups_size': 0, 'ondisk_log_size': 6727, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52824556, 'num_objects': 6449, 'num_object_clones': 0, 'num_object_copies': 25796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6724, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4314}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:29.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:29.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6459", 'reported_seq': 6545, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:26.574355+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:26.574355+0000', 'last_peered': '2024-09-18T11:48:26.574355+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:26.574355+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6459, 'log_dups_size': 0, 'ondisk_log_size': 6459, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50255094, 'num_objects': 6135, 'num_object_clones': 0, 'num_object_copies': 24540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6457, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4160}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:29.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:29.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6663", 'reported_seq': 6726, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:25.568939+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:25.568939+0000', 'last_peered': '2024-09-18T11:48:25.568939+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:25.568939+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6663, 'log_dups_size': 0, 'ondisk_log_size': 6663, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52178801, 'num_objects': 6370, 'num_object_clones': 0, 'num_object_copies': 25480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2064, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6660, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:29.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:29.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6641", 'reported_seq': 6679, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:25.620623+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:25.620623+0000', 'last_peered': '2024-09-18T11:48:25.620623+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:25.620623+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6641, 'log_dups_size': 0, 'ondisk_log_size': 6641, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52345467, 'num_objects': 6390, 'num_object_clones': 0, 'num_object_copies': 25560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6640, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:29.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:29.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6641", 'reported_seq': 6719, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:26.526479+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:26.526479+0000', 'last_peered': '2024-09-18T11:48:26.526479+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:26.526479+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6641, 'log_dups_size': 0, 'ondisk_log_size': 6641, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51821179, 'num_objects': 6326, 'num_object_clones': 0, 'num_object_copies': 25304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6640, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:29.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:29.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6694", 'reported_seq': 6745, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:25.602953+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:25.602953+0000', 'last_peered': '2024-09-18T11:48:25.602953+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:25.602953+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6694, 'log_dups_size': 0, 'ondisk_log_size': 6694, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52393206, 'num_objects': 6396, 'num_object_clones': 0, 'num_object_copies': 25584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6396, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6692, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:29.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:29.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6513", 'reported_seq': 6579, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:27.000572+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:27.000572+0000', 'last_peered': '2024-09-18T11:48:27.000572+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:27.000572+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6513, 'log_dups_size': 0, 'ondisk_log_size': 6513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50738422, 'num_objects': 6194, 'num_object_clones': 0, 'num_object_copies': 24776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2014, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6511, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4182}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:29.831 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:30.481 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30364/405080 objects degraded (7.496%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":308,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":101270,"data_bytes":829556108,"bytes_used":2797416448,"bytes_avail":383749640192,"bytes_total":386547056640,"degraded_objects":30364,"degraded_total":405080,"degraded_ratio":0.074958032981139525,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":97550,"recovering_keys_per_sec":0,"num_objects_recovered":147,"num_bytes_recovered":1171456,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":197233,"read_op_per_sec":0,"write_op_per_sec":212},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:30.507 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:30.507 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:31.078 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:31.080 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:31.108 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-18T11:48:29.924705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":829556108,"num_objects":101270,"num_object_clones":0,"num_object_copies":405080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101270,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105817,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3090,"num_bytes_recovered":25042944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105844,"ondisk_log_size":105844,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731852,"kb_used_data":1507744,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374755508,"statfs":{"total":386547056640,"available":383749640192,"internally_reserved":0,"allocated":1543929856,"data_stored":1543261116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,1,16,12,9,2,0,0,0,0,0,1,7],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16146408,"num_objects":-1970,"num_object_clones":0,"num_object_copies":-7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2550,"num_write_kb":2313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1171456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008655"},"pg_stats":[{"pgid":"1.f","version":"33'6745","reported_seq":6788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.606033+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:25.606033+0000","last_peered":"2024-09-18T11:48:25.606033+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:25.606033+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53288960,"num_objects":6505,"num_object_clones":0,"num_object_copies":26020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6745,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6661","reported_seq":8659,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.996691+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:26.996691+0000","last_peered":"2024-09-18T11:48:26.996691+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:26.996691+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6661,"log_dups_size":0,"ondisk_log_size":6661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52385014,"num_objects":6395,"num_object_clones":0,"num_object_copies":25580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6659,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":7913472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5259}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6541","reported_seq":6626,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.526526+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:26.526526+0000","last_peered":"2024-09-18T11:48:26.526526+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:26.526526+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6541,"log_dups_size":0,"ondisk_log_size":6541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51337875,"num_objects":6268,"num_object_clones":0,"num_object_copies":25072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6540,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6608","reported_seq":6672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.915120+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:26.915120+0000","last_peered":"2024-09-18T11:48:26.915120+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:26.915120+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6608,"log_dups_size":0,"ondisk_log_size":6608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51754230,"num_objects":6318,"num_object_clones":0,"num_object_copies":25272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6580","reported_seq":6639,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.636595+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:25.636595+0000","last_peered":"2024-09-18T11:48:25.636595+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:25.636595+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6580,"log_dups_size":0,"ondisk_log_size":6580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51729654,"num_objects":6315,"num_object_clones":0,"num_object_copies":25260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6578,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6534","reported_seq":6579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.413282+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:25.413282+0000","last_peered":"2024-09-18T11:48:25.413282+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:25.413282+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51216384,"num_objects":6252,"num_object_clones":0,"num_object_copies":25008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6660","reported_seq":6727,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.919821+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:26.919821+0000","last_peered":"2024-09-18T11:48:26.919821+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:26.919821+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52065526,"num_objects":6356,"num_object_clones":0,"num_object_copies":25424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6658,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4222}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6576","reported_seq":6641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.913656+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:26.913656+0000","last_peered":"2024-09-18T11:48:26.913656+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:26.913656+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51598582,"num_objects":6299,"num_object_clones":0,"num_object_copies":25196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6574,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6727","reported_seq":6787,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.658916+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:25.658916+0000","last_peered":"2024-09-18T11:48:25.658916+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:25.658916+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52824556,"num_objects":6449,"num_object_clones":0,"num_object_copies":25796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6459","reported_seq":6545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.574355+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:26.574355+0000","last_peered":"2024-09-18T11:48:26.574355+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:26.574355+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6459,"log_dups_size":0,"ondisk_log_size":6459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50255094,"num_objects":6135,"num_object_clones":0,"num_object_copies":24540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6457,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6663","reported_seq":6726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.568939+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:25.568939+0000","last_peered":"2024-09-18T11:48:25.568939+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:25.568939+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52178801,"num_objects":6370,"num_object_clones":0,"num_object_copies":25480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6660,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6641","reported_seq":6679,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.620623+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:25.620623+0000","last_peered":"2024-09-18T11:48:25.620623+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:25.620623+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52345467,"num_objects":6390,"num_object_clones":0,"num_object_copies":25560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6640,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4281}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6641","reported_seq":6719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:26.526479+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:26.526479+0000","last_peered":"2024-09-18T11:48:26.526479+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:26.526479+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51821179,"num_objects":6326,"num_object_clones":0,"num_object_copies":25304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6640,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4272}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6694","reported_seq":6745,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:25.602953+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:25.602953+0000","last_peered":"2024-09-18T11:48:25.602953+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:25.602953+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52393206,"num_objects":6396,"num_object_clones":0,"num_object_copies":25584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4238}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6601","reported_seq":10886,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:25.698379+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:25.698379+0000","last_peered":"2024-09-18T11:48:25.698379+0000","last_clean":"2024-09-18T11:48:25.698379+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:25.698379+0000","last_undegraded":"2024-09-18T11:48:25.698379+0000","last_fullsized":"2024-09-18T11:48:25.698379+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6601,"log_dups_size":0,"ondisk_log_size":6601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51623158,"num_objects":6302,"num_object_clones":0,"num_object_copies":25208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6599,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6513","reported_seq":6579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:27.000572+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:27.000572+0000","last_peered":"2024-09-18T11:48:27.000572+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:27.000572+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6513,"log_dups_size":0,"ondisk_log_size":6513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50738422,"num_objects":6194,"num_object_clones":0,"num_object_copies":24776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":829556108,"num_objects":101270,"num_object_clones":0,"num_object_copies":405080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101270,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105817,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3090,"num_bytes_recovered":25042944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1542180864,"data_stored":1542180864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105844,"ondisk_log_size":105844,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708192,"kb_used_data":406168,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93663648,"statfs":{"total":96636764160,"available":95911575552,"internally_reserved":0,"allocated":415916032,"data_stored":415748847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,7,2,0,0,0,0,0,0,0,7],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.1340000000000003},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.1619999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":8.4600000000000009},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":14.118}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":19.850999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":19.167999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714980,"kb_used_data":405148,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93656860,"statfs":{"total":96636764160,"available":95904624640,"internally_reserved":0,"allocated":414871552,"data_stored":414704367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,4,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":4.0049999999999999},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.89300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":4.6989999999999998},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":0.81599999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.91800000000000004},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.83499999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598020,"kb_used_data":291892,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93773820,"statfs":{"total":96636764160,"available":96024391680,"internally_reserved":0,"allocated":298897408,"data_stored":298730223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":2.7999999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":13.847},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.2290000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":5.7709999999999999},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":13.888999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710660,"kb_used_data":404536,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93661180,"statfs":{"total":96636764160,"available":95909048320,"internally_reserved":0,"allocated":414244864,"data_stored":414077679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":1.325},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":3.6379999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.542},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.5170000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.0659999999999998},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":14.198}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":413937664,"data_stored":413937664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298606592,"data_stored":298606592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":414318592,"data_stored":414318592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":415318016,"data_stored":415318016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:34.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:34.624 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:34.650 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-18T11:48:33.925539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":819570060,"num_objects":100051,"num_object_clones":0,"num_object_copies":400204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107036,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3223,"num_bytes_recovered":25862144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107063,"ondisk_log_size":107063,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725488,"kb_used_data":1493064,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":374761872,"statfs":{"total":386547056640,"available":383756156928,"internally_reserved":0,"allocated":1528897536,"data_stored":1528228796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,3,10,9,2,3,0,0,0,0,0,4,0,8],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22847488,"num_objects":-2789,"num_object_clones":0,"num_object_copies":-11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2847,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":1990656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004740"},"pg_stats":[{"pgid":"1.f","version":"33'6833","reported_seq":6886,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.666659+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:30.666659+0000","last_peered":"2024-09-18T11:48:30.666659+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:30.666659+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52568064,"num_objects":6417,"num_object_clones":0,"num_object_copies":25668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6833,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4292}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6735","reported_seq":8937,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:32.009834+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:32.009834+0000","last_peered":"2024-09-18T11:48:32.009834+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:32.009834+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6735,"log_dups_size":0,"ondisk_log_size":6735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51778806,"num_objects":6321,"num_object_clones":0,"num_object_copies":25284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":8732672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5285}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6622","reported_seq":6707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.615799+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:31.615799+0000","last_peered":"2024-09-18T11:48:31.615799+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:31.615799+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6622,"log_dups_size":0,"ondisk_log_size":6622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50674323,"num_objects":6187,"num_object_clones":0,"num_object_copies":24748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6187,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6621,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4133}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6673","reported_seq":6737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.881089+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:31.881089+0000","last_peered":"2024-09-18T11:48:31.881089+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:31.881089+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51221750,"num_objects":6253,"num_object_clones":0,"num_object_copies":25012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4139}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6658","reported_seq":6721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.670245+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:30.670245+0000","last_peered":"2024-09-18T11:48:30.670245+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:30.670245+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6658,"log_dups_size":0,"ondisk_log_size":6658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51090678,"num_objects":6237,"num_object_clones":0,"num_object_copies":24948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6656,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6612","reported_seq":6659,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.662360+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:30.662360+0000","last_peered":"2024-09-18T11:48:30.662360+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:30.662360+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6612,"log_dups_size":0,"ondisk_log_size":6612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50577408,"num_objects":6174,"num_object_clones":0,"num_object_copies":24696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6612,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6753","reported_seq":6820,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.950576+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:31.950576+0000","last_peered":"2024-09-18T11:48:31.950576+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:31.950576+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6753,"log_dups_size":0,"ondisk_log_size":6753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51303670,"num_objects":6263,"num_object_clones":0,"num_object_copies":25052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6751,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6645","reported_seq":6709,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:32.001160+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:32.001160+0000","last_peered":"2024-09-18T11:48:32.001160+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:32.001160+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51033334,"num_objects":6230,"num_object_clones":0,"num_object_copies":24920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4196}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6805","reported_seq":6877,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.621909+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:30.621909+0000","last_peered":"2024-09-18T11:48:30.621909+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:30.621909+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6805,"log_dups_size":0,"ondisk_log_size":6805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52185580,"num_objects":6371,"num_object_clones":0,"num_object_copies":25484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6802,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6539","reported_seq":6625,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.653060+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:31.653060+0000","last_peered":"2024-09-18T11:48:31.653060+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:31.653060+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6539,"log_dups_size":0,"ondisk_log_size":6539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49599734,"num_objects":6055,"num_object_clones":0,"num_object_copies":24220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6537,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6727","reported_seq":6794,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.712051+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:30.712051+0000","last_peered":"2024-09-18T11:48:30.712051+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:30.712051+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51654513,"num_objects":6306,"num_object_clones":0,"num_object_copies":25224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6699","reported_seq":6746,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.605423+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:30.605423+0000","last_peered":"2024-09-18T11:48:30.605423+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:30.605423+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6699,"log_dups_size":0,"ondisk_log_size":6699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51870331,"num_objects":6332,"num_object_clones":0,"num_object_copies":25328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6698,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4227}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6711","reported_seq":6789,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.506880+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:31.506880+0000","last_peered":"2024-09-18T11:48:31.506880+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:31.506880+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51247739,"num_objects":6256,"num_object_clones":0,"num_object_copies":25024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6710,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6778","reported_seq":6849,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.476370+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:30.476370+0000","last_peered":"2024-09-18T11:48:30.476370+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:30.476370+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51705078,"num_objects":6312,"num_object_clones":0,"num_object_copies":25248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4164}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6687","reported_seq":10972,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:30.753513+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:30.753513+0000","last_peered":"2024-09-18T11:48:30.753513+0000","last_clean":"2024-09-18T11:48:30.753513+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:30.753513+0000","last_undegraded":"2024-09-18T11:48:30.753513+0000","last_fullsized":"2024-09-18T11:48:30.753513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6687,"log_dups_size":0,"ondisk_log_size":6687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50918646,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6586","reported_seq":6652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.916073+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:31.916073+0000","last_peered":"2024-09-18T11:48:31.916073+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:31.916073+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6586,"log_dups_size":0,"ondisk_log_size":6586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50140406,"num_objects":6121,"num_object_clones":0,"num_object_copies":24484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":819570060,"num_objects":100051,"num_object_clones":0,"num_object_copies":400204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107036,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3223,"num_bytes_recovered":25862144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1527066624,"data_stored":1527066624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107063,"ondisk_log_size":107063,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707580,"kb_used_data":401396,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93664260,"statfs":{"total":96636764160,"available":95912202240,"internally_reserved":0,"allocated":411029504,"data_stored":410862319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,3,1,3,0,0,0,0,0,0,0,7],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.35299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":2.056},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":10.247999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":2.7309999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":10.305}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697648,"kb_used_data":400296,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93674192,"statfs":{"total":96636764160,"available":95922372608,"internally_reserved":0,"allocated":409903104,"data_stored":409735919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":23.731999999999999},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":10.779}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.0979999999999999},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":7.1680000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":7.2119999999999997},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.72799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606128,"kb_used_data":291680,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93765712,"statfs":{"total":96636764160,"available":96016089088,"internally_reserved":0,"allocated":298680320,"data_stored":298513135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.0049999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":3.5819999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.0449999999999999},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.7400000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.52800000000000002},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.2610000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714132,"kb_used_data":399692,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93657708,"statfs":{"total":96636764160,"available":95905492992,"internally_reserved":0,"allocated":409284608,"data_stored":409117423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,6,1,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":3.0920000000000001},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":1.881}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.036},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.3119999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":1.907}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":408997888,"data_stored":408997888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298352640,"data_stored":298352640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409387008,"data_stored":409387008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":410329088,"data_stored":410329088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:34.651 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:35.227 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:35.227 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:35.252 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-18T11:48:33.925539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":819570060,"num_objects":100051,"num_object_clones":0,"num_object_copies":400204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107036,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3223,"num_bytes_recovered":25862144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107063,"ondisk_log_size":107063,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725488,"kb_used_data":1493064,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":374761872,"statfs":{"total":386547056640,"available":383756156928,"internally_reserved":0,"allocated":1528897536,"data_stored":1528228796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,3,10,9,2,3,0,0,0,0,0,4,0,8],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22847488,"num_objects":-2789,"num_object_clones":0,"num_object_copies":-11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2847,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":1990656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004740"},"pg_stats":[{"pgid":"1.f","version":"33'6833","reported_seq":6886,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.666659+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:30.666659+0000","last_peered":"2024-09-18T11:48:30.666659+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:30.666659+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52568064,"num_objects":6417,"num_object_clones":0,"num_object_copies":25668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6833,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4292}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6735","reported_seq":8937,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:32.009834+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:32.009834+0000","last_peered":"2024-09-18T11:48:32.009834+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:32.009834+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6735,"log_dups_size":0,"ondisk_log_size":6735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51778806,"num_objects":6321,"num_object_clones":0,"num_object_copies":25284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":8732672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5285}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6622","reported_seq":6707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.615799+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:31.615799+0000","last_peered":"2024-09-18T11:48:31.615799+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:31.615799+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6622,"log_dups_size":0,"ondisk_log_size":6622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50674323,"num_objects":6187,"num_object_clones":0,"num_object_copies":24748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6187,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6621,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4133}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6673","reported_seq":6737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.881089+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:31.881089+0000","last_peered":"2024-09-18T11:48:31.881089+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:31.881089+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51221750,"num_objects":6253,"num_object_clones":0,"num_object_copies":25012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4139}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6658","reported_seq":6721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.670245+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:30.670245+0000","last_peered":"2024-09-18T11:48:30.670245+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:30.670245+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6658,"log_dups_size":0,"ondisk_log_size":6658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51090678,"num_objects":6237,"num_object_clones":0,"num_object_copies":24948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6656,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6612","reported_seq":6659,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.662360+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:30.662360+0000","last_peered":"2024-09-18T11:48:30.662360+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:30.662360+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6612,"log_dups_size":0,"ondisk_log_size":6612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50577408,"num_objects":6174,"num_object_clones":0,"num_object_copies":24696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6612,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6753","reported_seq":6820,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.950576+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:31.950576+0000","last_peered":"2024-09-18T11:48:31.950576+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:31.950576+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6753,"log_dups_size":0,"ondisk_log_size":6753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51303670,"num_objects":6263,"num_object_clones":0,"num_object_copies":25052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6751,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6645","reported_seq":6709,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:32.001160+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:32.001160+0000","last_peered":"2024-09-18T11:48:32.001160+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:32.001160+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51033334,"num_objects":6230,"num_object_clones":0,"num_object_copies":24920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4196}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6805","reported_seq":6877,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.621909+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:30.621909+0000","last_peered":"2024-09-18T11:48:30.621909+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:30.621909+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6805,"log_dups_size":0,"ondisk_log_size":6805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52185580,"num_objects":6371,"num_object_clones":0,"num_object_copies":25484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6802,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6539","reported_seq":6625,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.653060+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:31.653060+0000","last_peered":"2024-09-18T11:48:31.653060+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:31.653060+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6539,"log_dups_size":0,"ondisk_log_size":6539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49599734,"num_objects":6055,"num_object_clones":0,"num_object_copies":24220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6537,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6727","reported_seq":6794,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.712051+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:30.712051+0000","last_peered":"2024-09-18T11:48:30.712051+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:30.712051+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51654513,"num_objects":6306,"num_object_clones":0,"num_object_copies":25224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6699","reported_seq":6746,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.605423+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:30.605423+0000","last_peered":"2024-09-18T11:48:30.605423+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:30.605423+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6699,"log_dups_size":0,"ondisk_log_size":6699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51870331,"num_objects":6332,"num_object_clones":0,"num_object_copies":25328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6698,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4227}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6711","reported_seq":6789,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.506880+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:31.506880+0000","last_peered":"2024-09-18T11:48:31.506880+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:31.506880+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51247739,"num_objects":6256,"num_object_clones":0,"num_object_copies":25024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6710,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6778","reported_seq":6849,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:30.476370+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:30.476370+0000","last_peered":"2024-09-18T11:48:30.476370+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:30.476370+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51705078,"num_objects":6312,"num_object_clones":0,"num_object_copies":25248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4164}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6687","reported_seq":10972,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:30.753513+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:30.753513+0000","last_peered":"2024-09-18T11:48:30.753513+0000","last_clean":"2024-09-18T11:48:30.753513+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:30.753513+0000","last_undegraded":"2024-09-18T11:48:30.753513+0000","last_fullsized":"2024-09-18T11:48:30.753513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6687,"log_dups_size":0,"ondisk_log_size":6687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50918646,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6586","reported_seq":6652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.916073+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:31.916073+0000","last_peered":"2024-09-18T11:48:31.916073+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:31.916073+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6586,"log_dups_size":0,"ondisk_log_size":6586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50140406,"num_objects":6121,"num_object_clones":0,"num_object_copies":24484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":819570060,"num_objects":100051,"num_object_clones":0,"num_object_copies":400204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107036,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3223,"num_bytes_recovered":25862144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1527066624,"data_stored":1527066624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107063,"ondisk_log_size":107063,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707580,"kb_used_data":401396,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93664260,"statfs":{"total":96636764160,"available":95912202240,"internally_reserved":0,"allocated":411029504,"data_stored":410862319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,3,1,3,0,0,0,0,0,0,0,7],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.35299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":2.056},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":10.247999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":2.7309999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":10.305}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697648,"kb_used_data":400296,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93674192,"statfs":{"total":96636764160,"available":95922372608,"internally_reserved":0,"allocated":409903104,"data_stored":409735919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":23.731999999999999},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":10.779}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.0979999999999999},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":7.1680000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":7.2119999999999997},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.72799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606128,"kb_used_data":291680,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93765712,"statfs":{"total":96636764160,"available":96016089088,"internally_reserved":0,"allocated":298680320,"data_stored":298513135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.0049999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":3.5819999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.0449999999999999},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.7400000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.52800000000000002},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.2610000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714132,"kb_used_data":399692,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93657708,"statfs":{"total":96636764160,"available":95905492992,"internally_reserved":0,"allocated":409284608,"data_stored":409117423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,6,1,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":3.0920000000000001},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":1.881}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.036},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.3119999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":1.907}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":408997888,"data_stored":408997888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298352640,"data_stored":298352640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409387008,"data_stored":409387008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":410329088,"data_stored":410329088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:35.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:35.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6833", 'reported_seq': 6886, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:30.666659+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:30.666659+0000', 'last_peered': '2024-09-18T11:48:30.666659+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:30.666659+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6833, 'log_dups_size': 0, 'ondisk_log_size': 6833, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52568064, 'num_objects': 6417, 'num_object_clones': 0, 'num_object_copies': 25668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6833, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4292}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:35.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:35.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6735", 'reported_seq': 8937, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:32.009834+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:32.009834+0000', 'last_peered': '2024-09-18T11:48:32.009834+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:32.009834+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6735, 'log_dups_size': 0, 'ondisk_log_size': 6735, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51778806, 'num_objects': 6321, 'num_object_clones': 0, 'num_object_copies': 25284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6733, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1070, 'num_bytes_recovered': 8732672, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5285}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:35.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:35.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6622", 'reported_seq': 6707, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:31.615799+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:31.615799+0000', 'last_peered': '2024-09-18T11:48:31.615799+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:31.615799+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6622, 'log_dups_size': 0, 'ondisk_log_size': 6622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50674323, 'num_objects': 6187, 'num_object_clones': 0, 'num_object_copies': 24748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6187, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6621, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4133}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:35.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:35.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6673", 'reported_seq': 6737, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:31.881089+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:31.881089+0000', 'last_peered': '2024-09-18T11:48:31.881089+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:31.881089+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6673, 'log_dups_size': 0, 'ondisk_log_size': 6673, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51221750, 'num_objects': 6253, 'num_object_clones': 0, 'num_object_copies': 25012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6671, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4139}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:35.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:35.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6658", 'reported_seq': 6721, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:30.670245+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:30.670245+0000', 'last_peered': '2024-09-18T11:48:30.670245+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:30.670245+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6658, 'log_dups_size': 0, 'ondisk_log_size': 6658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51090678, 'num_objects': 6237, 'num_object_clones': 0, 'num_object_copies': 24948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6656, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4158}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:35.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:35.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6612", 'reported_seq': 6659, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:30.662360+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:30.662360+0000', 'last_peered': '2024-09-18T11:48:30.662360+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:30.662360+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6612, 'log_dups_size': 0, 'ondisk_log_size': 6612, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50577408, 'num_objects': 6174, 'num_object_clones': 0, 'num_object_copies': 24696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2152, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6612, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4022}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:35.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:35.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6753", 'reported_seq': 6820, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:31.950576+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:31.950576+0000', 'last_peered': '2024-09-18T11:48:31.950576+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:31.950576+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6753, 'log_dups_size': 0, 'ondisk_log_size': 6753, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51303670, 'num_objects': 6263, 'num_object_clones': 0, 'num_object_copies': 25052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6751, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4129}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:35.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:35.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6645", 'reported_seq': 6709, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:32.001160+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:32.001160+0000', 'last_peered': '2024-09-18T11:48:32.001160+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:32.001160+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6645, 'log_dups_size': 0, 'ondisk_log_size': 6645, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51033334, 'num_objects': 6230, 'num_object_clones': 0, 'num_object_copies': 24920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6643, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:35.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:35.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6805", 'reported_seq': 6877, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:30.621909+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:30.621909+0000', 'last_peered': '2024-09-18T11:48:30.621909+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:30.621909+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6805, 'log_dups_size': 0, 'ondisk_log_size': 6805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52185580, 'num_objects': 6371, 'num_object_clones': 0, 'num_object_copies': 25484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6802, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4242}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6539", 'reported_seq': 6625, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:31.653060+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:31.653060+0000', 'last_peered': '2024-09-18T11:48:31.653060+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:31.653060+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6539, 'log_dups_size': 0, 'ondisk_log_size': 6539, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49599734, 'num_objects': 6055, 'num_object_clones': 0, 'num_object_copies': 24220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6537, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4080}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6727", 'reported_seq': 6794, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:30.712051+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:30.712051+0000', 'last_peered': '2024-09-18T11:48:30.712051+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:30.712051+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6727, 'log_dups_size': 0, 'ondisk_log_size': 6727, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51654513, 'num_objects': 6306, 'num_object_clones': 0, 'num_object_copies': 25224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2062, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6724, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4247}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6699", 'reported_seq': 6746, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:30.605423+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:30.605423+0000', 'last_peered': '2024-09-18T11:48:30.605423+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:30.605423+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6699, 'log_dups_size': 0, 'ondisk_log_size': 6699, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51870331, 'num_objects': 6332, 'num_object_clones': 0, 'num_object_copies': 25328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6698, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4227}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6711", 'reported_seq': 6789, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:31.506880+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:31.506880+0000', 'last_peered': '2024-09-18T11:48:31.506880+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:31.506880+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6711, 'log_dups_size': 0, 'ondisk_log_size': 6711, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51247739, 'num_objects': 6256, 'num_object_clones': 0, 'num_object_copies': 25024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6710, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6778", 'reported_seq': 6849, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:30.476370+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:30.476370+0000', 'last_peered': '2024-09-18T11:48:30.476370+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:30.476370+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6778, 'log_dups_size': 0, 'ondisk_log_size': 6778, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51705078, 'num_objects': 6312, 'num_object_clones': 0, 'num_object_copies': 25248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2150, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6776, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4164}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:35.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:35.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6586", 'reported_seq': 6652, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:31.916073+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:31.916073+0000', 'last_peered': '2024-09-18T11:48:31.916073+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:31.916073+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6586, 'log_dups_size': 0, 'ondisk_log_size': 6586, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50140406, 'num_objects': 6121, 'num_object_clones': 0, 'num_object_copies': 24484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2014, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6584, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:35.257 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:35.985 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:35.986 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30233/400204 objects degraded (7.554%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":314,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":100051,"data_bytes":819570060,"bytes_used":2790899712,"bytes_avail":383756156928,"bytes_total":386547056640,"degraded_objects":30233,"degraded_total":400204,"degraded_ratio":0.075543972573987261,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":165822,"recovering_keys_per_sec":0,"num_objects_recovered":277,"num_bytes_recovered":1990656,"num_keys_recovered":0,"write_bytes_sec":19789,"read_op_per_sec":0,"write_op_per_sec":237},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:36.024 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:36.024 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:36.433 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:48:36.431+0000 7fda77fff640 0 --2- 172.21.15.71:0/3406495695 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fda7806dc40 0x7fda78124a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T11:48:36.763 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:36.763 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:36.799 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-18T11:48:35.926006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":814990732,"num_objects":99492,"num_object_clones":0,"num_object_copies":397968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107595,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3235,"num_bytes_recovered":25862144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107622,"ondisk_log_size":107622,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2724672,"kb_used_data":1488088,"kb_used_omap":14,"kb_used_meta":1236529,"kb_avail":374762688,"statfs":{"total":386547056640,"available":383756992512,"internally_reserved":0,"allocated":1523802112,"data_stored":1523133372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1266205723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,1,13,6,4,4,0,0,0,0,5,4,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24748032,"num_objects":-3021,"num_object_clones":0,"num_object_copies":-12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3021,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002852"},"pg_stats":[{"pgid":"1.f","version":"33'6908","reported_seq":6965,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.675035+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:35.675035+0000","last_peered":"2024-09-18T11:48:35.675035+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:35.675035+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51953664,"num_objects":6342,"num_object_clones":0,"num_object_copies":25368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6735","reported_seq":8937,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:32.009834+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:32.009834+0000","last_peered":"2024-09-18T11:48:32.009834+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:32.009834+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6735,"log_dups_size":0,"ondisk_log_size":6735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51778806,"num_objects":6321,"num_object_clones":0,"num_object_copies":25284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":8732672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5285}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6622","reported_seq":6707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.615799+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:31.615799+0000","last_peered":"2024-09-18T11:48:31.615799+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:31.615799+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6622,"log_dups_size":0,"ondisk_log_size":6622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50674323,"num_objects":6187,"num_object_clones":0,"num_object_copies":24748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6187,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6621,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4133}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6673","reported_seq":6737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.881089+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:31.881089+0000","last_peered":"2024-09-18T11:48:31.881089+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:31.881089+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51221750,"num_objects":6253,"num_object_clones":0,"num_object_copies":25012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4139}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6740","reported_seq":6805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.723505+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:35.723505+0000","last_peered":"2024-09-18T11:48:35.723505+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:35.723505+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50418934,"num_objects":6155,"num_object_clones":0,"num_object_copies":24620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6691","reported_seq":6742,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.720479+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:35.720479+0000","last_peered":"2024-09-18T11:48:35.720479+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:35.720479+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6691,"log_dups_size":0,"ondisk_log_size":6691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49930240,"num_objects":6095,"num_object_clones":0,"num_object_copies":24380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6691,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6753","reported_seq":6820,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.950576+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:31.950576+0000","last_peered":"2024-09-18T11:48:31.950576+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:31.950576+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6753,"log_dups_size":0,"ondisk_log_size":6753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51303670,"num_objects":6263,"num_object_clones":0,"num_object_copies":25052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6751,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6645","reported_seq":6709,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:32.001160+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:32.001160+0000","last_peered":"2024-09-18T11:48:32.001160+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:32.001160+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51033334,"num_objects":6230,"num_object_clones":0,"num_object_copies":24920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4196}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6890","reported_seq":6962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.620030+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:35.620030+0000","last_peered":"2024-09-18T11:48:35.620030+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:35.620030+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6890,"log_dups_size":0,"ondisk_log_size":6890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51489260,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4157}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6539","reported_seq":6625,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.653060+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:31.653060+0000","last_peered":"2024-09-18T11:48:31.653060+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:31.653060+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6539,"log_dups_size":0,"ondisk_log_size":6539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49599734,"num_objects":6055,"num_object_clones":0,"num_object_copies":24220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6537,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6808","reported_seq":6875,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.678455+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:35.678455+0000","last_peered":"2024-09-18T11:48:35.678455+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:35.678455+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50990961,"num_objects":6225,"num_object_clones":0,"num_object_copies":24900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6805,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6782","reported_seq":6835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.596831+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:35.596831+0000","last_peered":"2024-09-18T11:48:35.596831+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:35.596831+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51190395,"num_objects":6249,"num_object_clones":0,"num_object_copies":24996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6781,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6711","reported_seq":6789,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.506880+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:31.506880+0000","last_peered":"2024-09-18T11:48:31.506880+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:31.506880+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51247739,"num_objects":6256,"num_object_clones":0,"num_object_copies":25024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6710,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6852","reported_seq":6933,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.200325+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:35.200325+0000","last_peered":"2024-09-18T11:48:35.200325+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:35.200325+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6852,"log_dups_size":0,"ondisk_log_size":6852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51098870,"num_objects":6238,"num_object_clones":0,"num_object_copies":24952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6850,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4095}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6687","reported_seq":10972,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:30.753513+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:30.753513+0000","last_peered":"2024-09-18T11:48:30.753513+0000","last_clean":"2024-09-18T11:48:30.753513+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:30.753513+0000","last_undegraded":"2024-09-18T11:48:30.753513+0000","last_fullsized":"2024-09-18T11:48:30.753513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6687,"log_dups_size":0,"ondisk_log_size":6687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50918646,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6586","reported_seq":6652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:31.916073+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:31.916073+0000","last_peered":"2024-09-18T11:48:31.916073+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:31.916073+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6586,"log_dups_size":0,"ondisk_log_size":6586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50140406,"num_objects":6121,"num_object_clones":0,"num_object_copies":24484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":814990732,"num_objects":99492,"num_object_clones":0,"num_object_copies":397968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107595,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3235,"num_bytes_recovered":25862144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1521922048,"data_stored":1521922048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107622,"ondisk_log_size":107622,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706764,"kb_used_data":396420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93665076,"statfs":{"total":96636764160,"available":95913037824,"internally_reserved":0,"allocated":405934080,"data_stored":405766895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,7,0,3,4,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":4.2380000000000004},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.2640000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":9.7439999999999998},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":8.4649999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":4.0949999999999998},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":6.7279999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697648,"kb_used_data":400296,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93674192,"statfs":{"total":96636764160,"available":95922372608,"internally_reserved":0,"allocated":409903104,"data_stored":409735919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":23.731999999999999},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":10.779}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.0979999999999999},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":7.1680000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":7.2119999999999997},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.72799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606128,"kb_used_data":291680,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93765712,"statfs":{"total":96636764160,"available":96016089088,"internally_reserved":0,"allocated":298680320,"data_stored":298513135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.0049999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":3.5819999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.0449999999999999},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.7400000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.52800000000000002},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.2610000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714132,"kb_used_data":399692,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93657708,"statfs":{"total":96636764160,"available":95905492992,"internally_reserved":0,"allocated":409284608,"data_stored":409117423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,6,1,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":3.0920000000000001},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":1.881}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.036},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.3119999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":1.907}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":408997888,"data_stored":408997888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298352640,"data_stored":298352640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409387008,"data_stored":409387008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405184512,"data_stored":405184512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:39.801 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:40.329 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:40.329 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:40.351 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-18T11:48:39.926965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":809452940,"num_objects":98816,"num_object_clones":0,"num_object_copies":395264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108271,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3311,"num_bytes_recovered":26476544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108298,"ondisk_log_size":108298,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694520,"kb_used_data":1478740,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374792840,"statfs":{"total":386547056640,"available":383787868160,"internally_reserved":0,"allocated":1514229760,"data_stored":1513561020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,1,14,6,9,5,0,0,0,0,6,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20103168,"num_objects":-2454,"num_object_clones":0,"num_object_copies":-9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002909"},"pg_stats":[{"pgid":"1.f","version":"33'6908","reported_seq":6965,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.675035+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:35.675035+0000","last_peered":"2024-09-18T11:48:35.675035+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:35.675035+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51953664,"num_objects":6342,"num_object_clones":0,"num_object_copies":25368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6813","reported_seq":9165,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:37.043974+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:37.043974+0000","last_peered":"2024-09-18T11:48:37.043974+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:37.043974+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6813,"log_dups_size":0,"ondisk_log_size":6813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51139830,"num_objects":6243,"num_object_clones":0,"num_object_copies":24972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6811,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1145,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6709","reported_seq":6794,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.677520+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:36.677520+0000","last_peered":"2024-09-18T11:48:36.677520+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:36.677520+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6709,"log_dups_size":0,"ondisk_log_size":6709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49961619,"num_objects":6100,"num_object_clones":0,"num_object_copies":24400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6708,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4046}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6746","reported_seq":6810,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.985319+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:36.985319+0000","last_peered":"2024-09-18T11:48:36.985319+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:36.985319+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6746,"log_dups_size":0,"ondisk_log_size":6746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50623734,"num_objects":6180,"num_object_clones":0,"num_object_copies":24720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6744,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6740","reported_seq":6805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.723505+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:35.723505+0000","last_peered":"2024-09-18T11:48:35.723505+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:35.723505+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50418934,"num_objects":6155,"num_object_clones":0,"num_object_copies":24620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6691","reported_seq":6742,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.720479+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:35.720479+0000","last_peered":"2024-09-18T11:48:35.720479+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:35.720479+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6691,"log_dups_size":0,"ondisk_log_size":6691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49930240,"num_objects":6095,"num_object_clones":0,"num_object_copies":24380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6691,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6818","reported_seq":6885,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.988079+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:36.988079+0000","last_peered":"2024-09-18T11:48:36.988079+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:36.988079+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50771190,"num_objects":6198,"num_object_clones":0,"num_object_copies":24792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4064}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6718","reported_seq":6783,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.867383+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:36.867383+0000","last_peered":"2024-09-18T11:48:36.867383+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:36.867383+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50435318,"num_objects":6157,"num_object_clones":0,"num_object_copies":24628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4123}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6890","reported_seq":6962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.620030+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:35.620030+0000","last_peered":"2024-09-18T11:48:35.620030+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:35.620030+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6890,"log_dups_size":0,"ondisk_log_size":6890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51489260,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4157}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6612","reported_seq":6700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.688277+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:36.688277+0000","last_peered":"2024-09-18T11:48:36.688277+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:36.688277+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6612,"log_dups_size":0,"ondisk_log_size":6612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49001718,"num_objects":5982,"num_object_clones":0,"num_object_copies":23928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6610,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4008}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6808","reported_seq":6875,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.678455+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:35.678455+0000","last_peered":"2024-09-18T11:48:35.678455+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:35.678455+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50990961,"num_objects":6225,"num_object_clones":0,"num_object_copies":24900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6805,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6782","reported_seq":6835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.596831+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:35.596831+0000","last_peered":"2024-09-18T11:48:35.596831+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:35.596831+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51190395,"num_objects":6249,"num_object_clones":0,"num_object_copies":24996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6781,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6785","reported_seq":6863,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.597875+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:36.597875+0000","last_peered":"2024-09-18T11:48:36.597875+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:36.597875+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6785,"log_dups_size":0,"ondisk_log_size":6785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50641531,"num_objects":6182,"num_object_clones":0,"num_object_copies":24728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6852","reported_seq":6933,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.200325+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:35.200325+0000","last_peered":"2024-09-18T11:48:35.200325+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:35.200325+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6852,"log_dups_size":0,"ondisk_log_size":6852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51098870,"num_objects":6238,"num_object_clones":0,"num_object_copies":24952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6850,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4095}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6764","reported_seq":11049,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:35.718024+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:35.718024+0000","last_peered":"2024-09-18T11:48:35.718024+0000","last_clean":"2024-09-18T11:48:35.718024+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:35.718024+0000","last_undegraded":"2024-09-18T11:48:35.718024+0000","last_fullsized":"2024-09-18T11:48:35.718024+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6764,"log_dups_size":0,"ondisk_log_size":6764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50287862,"num_objects":6139,"num_object_clones":0,"num_object_copies":24556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6762,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6662","reported_seq":6728,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.738112+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:36.738112+0000","last_peered":"2024-09-18T11:48:36.738112+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:36.738112+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6662,"log_dups_size":0,"ondisk_log_size":6662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49517814,"num_objects":6045,"num_object_clones":0,"num_object_copies":24180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6660,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4033}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":809452940,"num_objects":98816,"num_object_clones":0,"num_object_copies":395264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108271,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3311,"num_bytes_recovered":26476544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1511391232,"data_stored":1511391232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108298,"ondisk_log_size":108298,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706764,"kb_used_data":396420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93665076,"statfs":{"total":96636764160,"available":95913037824,"internally_reserved":0,"allocated":405934080,"data_stored":405766895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,7,0,3,4,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":4.2380000000000004},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.2640000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":9.7439999999999998},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":8.4649999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":4.0949999999999998},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":6.7279999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696796,"kb_used_data":395280,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93675044,"statfs":{"total":96636764160,"available":95923245056,"internally_reserved":0,"allocated":404766720,"data_stored":404599535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.51600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.6830000000000001},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.53600000000000003},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.2649999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593324,"kb_used_data":291364,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93778516,"statfs":{"total":96636764160,"available":96029200384,"internally_reserved":0,"allocated":298356736,"data_stored":298189551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.9390000000000001},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":4.2249999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.7050000000000001},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.931}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":4.1959999999999997},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.6340000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697636,"kb_used_data":395676,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93674204,"statfs":{"total":96636764160,"available":95922384896,"internally_reserved":0,"allocated":405172224,"data_stored":405005039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,5,1,2,1,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":1.615},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.629}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.294},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.65500000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":8.0690000000000008}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":403922944,"data_stored":403922944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298057728,"data_stored":298057728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404226048,"data_stored":404226048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405184512,"data_stored":405184512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:40.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:40.908 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:40.909 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:40.944 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-18T11:48:39.926965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":809452940,"num_objects":98816,"num_object_clones":0,"num_object_copies":395264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108271,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3311,"num_bytes_recovered":26476544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108298,"ondisk_log_size":108298,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694520,"kb_used_data":1478740,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374792840,"statfs":{"total":386547056640,"available":383787868160,"internally_reserved":0,"allocated":1514229760,"data_stored":1513561020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,1,14,6,9,5,0,0,0,0,6,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20103168,"num_objects":-2454,"num_object_clones":0,"num_object_copies":-9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002909"},"pg_stats":[{"pgid":"1.f","version":"33'6908","reported_seq":6965,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.675035+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:35.675035+0000","last_peered":"2024-09-18T11:48:35.675035+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:35.675035+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51953664,"num_objects":6342,"num_object_clones":0,"num_object_copies":25368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6813","reported_seq":9165,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:37.043974+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:37.043974+0000","last_peered":"2024-09-18T11:48:37.043974+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:37.043974+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6813,"log_dups_size":0,"ondisk_log_size":6813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51139830,"num_objects":6243,"num_object_clones":0,"num_object_copies":24972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6811,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1145,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6709","reported_seq":6794,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.677520+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:36.677520+0000","last_peered":"2024-09-18T11:48:36.677520+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:36.677520+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6709,"log_dups_size":0,"ondisk_log_size":6709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49961619,"num_objects":6100,"num_object_clones":0,"num_object_copies":24400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6708,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4046}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6746","reported_seq":6810,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.985319+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:36.985319+0000","last_peered":"2024-09-18T11:48:36.985319+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:36.985319+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6746,"log_dups_size":0,"ondisk_log_size":6746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50623734,"num_objects":6180,"num_object_clones":0,"num_object_copies":24720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6744,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6740","reported_seq":6805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.723505+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:35.723505+0000","last_peered":"2024-09-18T11:48:35.723505+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:35.723505+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50418934,"num_objects":6155,"num_object_clones":0,"num_object_copies":24620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6691","reported_seq":6742,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.720479+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:35.720479+0000","last_peered":"2024-09-18T11:48:35.720479+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:35.720479+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6691,"log_dups_size":0,"ondisk_log_size":6691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49930240,"num_objects":6095,"num_object_clones":0,"num_object_copies":24380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6691,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6818","reported_seq":6885,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.988079+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:36.988079+0000","last_peered":"2024-09-18T11:48:36.988079+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:36.988079+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50771190,"num_objects":6198,"num_object_clones":0,"num_object_copies":24792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4064}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6718","reported_seq":6783,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.867383+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:36.867383+0000","last_peered":"2024-09-18T11:48:36.867383+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:36.867383+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50435318,"num_objects":6157,"num_object_clones":0,"num_object_copies":24628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4123}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6890","reported_seq":6962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.620030+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:35.620030+0000","last_peered":"2024-09-18T11:48:35.620030+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:35.620030+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6890,"log_dups_size":0,"ondisk_log_size":6890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51489260,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4157}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6612","reported_seq":6700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.688277+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:36.688277+0000","last_peered":"2024-09-18T11:48:36.688277+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:36.688277+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6612,"log_dups_size":0,"ondisk_log_size":6612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49001718,"num_objects":5982,"num_object_clones":0,"num_object_copies":23928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6610,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4008}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6808","reported_seq":6875,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.678455+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:35.678455+0000","last_peered":"2024-09-18T11:48:35.678455+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:35.678455+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50990961,"num_objects":6225,"num_object_clones":0,"num_object_copies":24900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6805,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6782","reported_seq":6835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.596831+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:35.596831+0000","last_peered":"2024-09-18T11:48:35.596831+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:35.596831+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51190395,"num_objects":6249,"num_object_clones":0,"num_object_copies":24996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6781,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6785","reported_seq":6863,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.597875+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:36.597875+0000","last_peered":"2024-09-18T11:48:36.597875+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:36.597875+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6785,"log_dups_size":0,"ondisk_log_size":6785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50641531,"num_objects":6182,"num_object_clones":0,"num_object_copies":24728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6852","reported_seq":6933,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:35.200325+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:35.200325+0000","last_peered":"2024-09-18T11:48:35.200325+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:35.200325+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6852,"log_dups_size":0,"ondisk_log_size":6852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51098870,"num_objects":6238,"num_object_clones":0,"num_object_copies":24952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6850,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4095}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6764","reported_seq":11049,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:35.718024+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:35.718024+0000","last_peered":"2024-09-18T11:48:35.718024+0000","last_clean":"2024-09-18T11:48:35.718024+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:35.718024+0000","last_undegraded":"2024-09-18T11:48:35.718024+0000","last_fullsized":"2024-09-18T11:48:35.718024+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6764,"log_dups_size":0,"ondisk_log_size":6764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50287862,"num_objects":6139,"num_object_clones":0,"num_object_copies":24556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6762,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6662","reported_seq":6728,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.738112+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:36.738112+0000","last_peered":"2024-09-18T11:48:36.738112+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:36.738112+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6662,"log_dups_size":0,"ondisk_log_size":6662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49517814,"num_objects":6045,"num_object_clones":0,"num_object_copies":24180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6660,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4033}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":809452940,"num_objects":98816,"num_object_clones":0,"num_object_copies":395264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108271,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3311,"num_bytes_recovered":26476544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1511391232,"data_stored":1511391232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108298,"ondisk_log_size":108298,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706764,"kb_used_data":396420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93665076,"statfs":{"total":96636764160,"available":95913037824,"internally_reserved":0,"allocated":405934080,"data_stored":405766895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,7,0,3,4,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":4.2380000000000004},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.2640000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":9.7439999999999998},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":8.4649999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":4.0949999999999998},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":6.7279999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696796,"kb_used_data":395280,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93675044,"statfs":{"total":96636764160,"available":95923245056,"internally_reserved":0,"allocated":404766720,"data_stored":404599535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.51600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.6830000000000001},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.53600000000000003},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.2649999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593324,"kb_used_data":291364,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93778516,"statfs":{"total":96636764160,"available":96029200384,"internally_reserved":0,"allocated":298356736,"data_stored":298189551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.9390000000000001},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":4.2249999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.7050000000000001},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.931}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":4.1959999999999997},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.6340000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697636,"kb_used_data":395676,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93674204,"statfs":{"total":96636764160,"available":95922384896,"internally_reserved":0,"allocated":405172224,"data_stored":405005039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,5,1,2,1,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":1.615},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.629}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.294},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.65500000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":8.0690000000000008}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":403922944,"data_stored":403922944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298057728,"data_stored":298057728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404226048,"data_stored":404226048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405184512,"data_stored":405184512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:40.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:40.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6908", 'reported_seq': 6965, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:35.675035+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:35.675035+0000', 'last_peered': '2024-09-18T11:48:35.675035+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:35.675035+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6908, 'log_dups_size': 0, 'ondisk_log_size': 6908, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51953664, 'num_objects': 6342, 'num_object_clones': 0, 'num_object_copies': 25368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6908, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6813", 'reported_seq': 9165, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:37.043974+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:37.043974+0000', 'last_peered': '2024-09-18T11:48:37.043974+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:37.043974+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6813, 'log_dups_size': 0, 'ondisk_log_size': 6813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51139830, 'num_objects': 6243, 'num_object_clones': 0, 'num_object_copies': 24972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 961, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6243, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6811, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1145, 'num_bytes_recovered': 9347072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5282}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6709", 'reported_seq': 6794, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:36.677520+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:36.677520+0000', 'last_peered': '2024-09-18T11:48:36.677520+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:36.677520+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6709, 'log_dups_size': 0, 'ondisk_log_size': 6709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49961619, 'num_objects': 6100, 'num_object_clones': 0, 'num_object_copies': 24400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6100, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6708, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4046}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6746", 'reported_seq': 6810, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:36.985319+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:36.985319+0000', 'last_peered': '2024-09-18T11:48:36.985319+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:36.985319+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6746, 'log_dups_size': 0, 'ondisk_log_size': 6746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50623734, 'num_objects': 6180, 'num_object_clones': 0, 'num_object_copies': 24720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6744, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4066}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6740", 'reported_seq': 6805, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:35.723505+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:35.723505+0000', 'last_peered': '2024-09-18T11:48:35.723505+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:35.723505+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6740, 'log_dups_size': 0, 'ondisk_log_size': 6740, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50418934, 'num_objects': 6155, 'num_object_clones': 0, 'num_object_copies': 24620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2080, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6155, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6738, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4077}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6691", 'reported_seq': 6742, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:35.720479+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:35.720479+0000', 'last_peered': '2024-09-18T11:48:35.720479+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:35.720479+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6691, 'log_dups_size': 0, 'ondisk_log_size': 6691, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49930240, 'num_objects': 6095, 'num_object_clones': 0, 'num_object_copies': 24380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2151, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6095, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6691, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3944}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:40.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6818", 'reported_seq': 6885, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:36.988079+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:36.988079+0000', 'last_peered': '2024-09-18T11:48:36.988079+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:36.988079+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6818, 'log_dups_size': 0, 'ondisk_log_size': 6818, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50771190, 'num_objects': 6198, 'num_object_clones': 0, 'num_object_copies': 24792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6816, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4064}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6718", 'reported_seq': 6783, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:36.867383+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:36.867383+0000', 'last_peered': '2024-09-18T11:48:36.867383+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:36.867383+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6718, 'log_dups_size': 0, 'ondisk_log_size': 6718, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50435318, 'num_objects': 6157, 'num_object_clones': 0, 'num_object_copies': 24628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6716, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4123}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6890", 'reported_seq': 6962, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:35.620030+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:35.620030+0000', 'last_peered': '2024-09-18T11:48:35.620030+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:35.620030+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6890, 'log_dups_size': 0, 'ondisk_log_size': 6890, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51489260, 'num_objects': 6286, 'num_object_clones': 0, 'num_object_copies': 25144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6887, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4157}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6612", 'reported_seq': 6700, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:36.688277+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:36.688277+0000', 'last_peered': '2024-09-18T11:48:36.688277+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:36.688277+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6612, 'log_dups_size': 0, 'ondisk_log_size': 6612, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49001718, 'num_objects': 5982, 'num_object_clones': 0, 'num_object_copies': 23928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6610, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4008}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6808", 'reported_seq': 6875, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:35.678455+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:35.678455+0000', 'last_peered': '2024-09-18T11:48:35.678455+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:35.678455+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6808, 'log_dups_size': 0, 'ondisk_log_size': 6808, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50990961, 'num_objects': 6225, 'num_object_clones': 0, 'num_object_copies': 24900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2062, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6225, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6805, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4166}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:40.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6782", 'reported_seq': 6835, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:35.596831+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:35.596831+0000', 'last_peered': '2024-09-18T11:48:35.596831+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:35.596831+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6782, 'log_dups_size': 0, 'ondisk_log_size': 6782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51190395, 'num_objects': 6249, 'num_object_clones': 0, 'num_object_copies': 24996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6781, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:40.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:40.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6785", 'reported_seq': 6863, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:36.597875+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:36.597875+0000', 'last_peered': '2024-09-18T11:48:36.597875+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:36.597875+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6785, 'log_dups_size': 0, 'ondisk_log_size': 6785, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50641531, 'num_objects': 6182, 'num_object_clones': 0, 'num_object_copies': 24728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6784, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:40.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:40.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6852", 'reported_seq': 6933, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:35.200325+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:35.200325+0000', 'last_peered': '2024-09-18T11:48:35.200325+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:35.200325+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6852, 'log_dups_size': 0, 'ondisk_log_size': 6852, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51098870, 'num_objects': 6238, 'num_object_clones': 0, 'num_object_copies': 24952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6850, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4095}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:40.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:40.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6662", 'reported_seq': 6728, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:36.738112+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:36.738112+0000', 'last_peered': '2024-09-18T11:48:36.738112+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:36.738112+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6662, 'log_dups_size': 0, 'ondisk_log_size': 6662, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49517814, 'num_objects': 6045, 'num_object_clones': 0, 'num_object_copies': 24180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2014, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6045, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6660, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4033}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:40.948 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:41.591 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:41.592 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30145/395264 objects degraded (7.627%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":319,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":98816,"data_bytes":809452940,"bytes_used":2759188480,"bytes_avail":383787868160,"bytes_total":386547056640,"degraded_objects":30145,"degraded_total":395264,"degraded_ratio":0.076265483322538866,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":119437,"recovering_keys_per_sec":0,"num_objects_recovered":221,"num_bytes_recovered":1433600,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":204},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:41.612 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:41.613 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:42.170 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:42.171 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:42.191 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-18T11:48:41.928567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":802334092,"num_objects":97947,"num_object_clones":0,"num_object_copies":391788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109140,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3311,"num_bytes_recovered":26476544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109167,"ondisk_log_size":109167,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680168,"kb_used_data":1468552,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374807192,"statfs":{"total":386547056640,"available":383802564608,"internally_reserved":0,"allocated":1503797248,"data_stored":1503128508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,7,7,9,5,1,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27222016,"num_objects":-3323,"num_object_clones":0,"num_object_copies":-13292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3323,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003862"},"pg_stats":[{"pgid":"1.f","version":"33'6995","reported_seq":7052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.805852+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:40.805852+0000","last_peered":"2024-09-18T11:48:40.805852+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:40.805852+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6995,"log_dups_size":0,"ondisk_log_size":6995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51240960,"num_objects":6255,"num_object_clones":0,"num_object_copies":25020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6995,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6813","reported_seq":9165,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:37.043974+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:37.043974+0000","last_peered":"2024-09-18T11:48:37.043974+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:37.043974+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6813,"log_dups_size":0,"ondisk_log_size":6813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51139830,"num_objects":6243,"num_object_clones":0,"num_object_copies":24972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6811,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1145,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6782","reported_seq":6867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.674540+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:41.674540+0000","last_peered":"2024-09-18T11:48:41.674540+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:41.674540+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49363603,"num_objects":6027,"num_object_clones":0,"num_object_copies":24108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6781,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6746","reported_seq":6810,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.985319+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:36.985319+0000","last_peered":"2024-09-18T11:48:36.985319+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:36.985319+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6746,"log_dups_size":0,"ondisk_log_size":6746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50623734,"num_objects":6180,"num_object_clones":0,"num_object_copies":24720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6744,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6814","reported_seq":6879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.754944+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:40.754944+0000","last_peered":"2024-09-18T11:48:40.754944+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:40.754944+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6814,"log_dups_size":0,"ondisk_log_size":6814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49812726,"num_objects":6081,"num_object_clones":0,"num_object_copies":24324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6812,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6782","reported_seq":6833,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.552184+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:40.552184+0000","last_peered":"2024-09-18T11:48:40.552184+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:40.552184+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49184768,"num_objects":6004,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6818","reported_seq":6885,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.988079+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:36.988079+0000","last_peered":"2024-09-18T11:48:36.988079+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:36.988079+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50771190,"num_objects":6198,"num_object_clones":0,"num_object_copies":24792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4064}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6718","reported_seq":6783,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.867383+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:36.867383+0000","last_peered":"2024-09-18T11:48:36.867383+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:36.867383+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50435318,"num_objects":6157,"num_object_clones":0,"num_object_copies":24628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4123}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6985","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.827198+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:40.827198+0000","last_peered":"2024-09-18T11:48:40.827198+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:40.827198+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6985,"log_dups_size":0,"ondisk_log_size":6985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50711020,"num_objects":6191,"num_object_clones":0,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6982,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6682","reported_seq":6770,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.518538+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:41.518538+0000","last_peered":"2024-09-18T11:48:41.518538+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:41.518538+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6682,"log_dups_size":0,"ondisk_log_size":6682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48428278,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3938}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6883","reported_seq":6950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.760336+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:40.760336+0000","last_peered":"2024-09-18T11:48:40.760336+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:40.760336+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6883,"log_dups_size":0,"ondisk_log_size":6883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50376561,"num_objects":6150,"num_object_clones":0,"num_object_copies":24600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6880,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6849","reported_seq":6902,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.652228+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:40.652228+0000","last_peered":"2024-09-18T11:48:40.652228+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:40.652228+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6849,"log_dups_size":0,"ondisk_log_size":6849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50641531,"num_objects":6182,"num_object_clones":0,"num_object_copies":24728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4080}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6854","reported_seq":6932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.640765+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:41.640765+0000","last_peered":"2024-09-18T11:48:41.640765+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:41.640765+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50076283,"num_objects":6113,"num_object_clones":0,"num_object_copies":24452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6934","reported_seq":7015,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.693160+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:40.693160+0000","last_peered":"2024-09-18T11:48:40.693160+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:40.693160+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6934,"log_dups_size":0,"ondisk_log_size":6934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50427126,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6932,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4013}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6850","reported_seq":11135,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:40.764841+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:40.764841+0000","last_peered":"2024-09-18T11:48:40.764841+0000","last_clean":"2024-09-18T11:48:40.764841+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:40.764841+0000","last_undegraded":"2024-09-18T11:48:40.764841+0000","last_fullsized":"2024-09-18T11:48:40.764841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49583350,"num_objects":6053,"num_object_clones":0,"num_object_copies":24212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6662","reported_seq":6728,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:36.738112+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:36.738112+0000","last_peered":"2024-09-18T11:48:36.738112+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:36.738112+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6662,"log_dups_size":0,"ondisk_log_size":6662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49517814,"num_objects":6045,"num_object_clones":0,"num_object_copies":24180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6660,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4033}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":802334092,"num_objects":97947,"num_object_clones":0,"num_object_copies":391788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109140,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3311,"num_bytes_recovered":26476544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1501085696,"data_stored":1501085696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109167,"ondisk_log_size":109167,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689244,"kb_used_data":391380,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93682596,"statfs":{"total":96636764160,"available":95930978304,"internally_reserved":0,"allocated":400773120,"data_stored":400605935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,3,1,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":6.3949999999999996},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":2.77}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":7.4080000000000004},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.923}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696000,"kb_used_data":390328,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93675840,"statfs":{"total":96636764160,"available":95924060160,"internally_reserved":0,"allocated":399695872,"data_stored":399528687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":6.8289999999999997},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.95599999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":4.5609999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.81899999999999995},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.65100000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597288,"kb_used_data":291168,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93774552,"statfs":{"total":96636764160,"available":96025141248,"internally_reserved":0,"allocated":298156032,"data_stored":297988847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.29699999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.19400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.22600000000000001},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.252}]}]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697636,"kb_used_data":395676,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93674204,"statfs":{"total":96636764160,"available":95922384896,"internally_reserved":0,"allocated":405172224,"data_stored":405005039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,5,1,2,1,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":1.615},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.629}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.294},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.65500000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":8.0690000000000008}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":403922944,"data_stored":403922944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297877504,"data_stored":297877504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399216640,"data_stored":399216640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":400068608,"data_stored":400068608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:45.194 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:45.749 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:45.749 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:45.775 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-09-18T11:48:43.929743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":799212940,"num_objects":97566,"num_object_clones":0,"num_object_copies":390264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3420,"num_bytes_recovered":27369472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109548,"ondisk_log_size":109548,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2679340,"kb_used_data":1463556,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374808020,"statfs":{"total":386547056640,"available":383803412480,"internally_reserved":0,"allocated":1498681344,"data_stored":1498012604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,6,7,4,1,0,0,0,0,0,2,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23420928,"num_objects":-2859,"num_object_clones":0,"num_object_copies":-11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2326528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004671"},"pg_stats":[{"pgid":"1.f","version":"33'6995","reported_seq":7052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.805852+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:40.805852+0000","last_peered":"2024-09-18T11:48:40.805852+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:40.805852+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6995,"log_dups_size":0,"ondisk_log_size":6995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51240960,"num_objects":6255,"num_object_clones":0,"num_object_copies":25020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6995,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6877","reported_seq":9447,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:42.033434+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:42.033434+0000","last_peered":"2024-09-18T11:48:42.033434+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:42.033434+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50615542,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1254,"num_bytes_recovered":10240000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6782","reported_seq":6867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.674540+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:41.674540+0000","last_peered":"2024-09-18T11:48:41.674540+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:41.674540+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49363603,"num_objects":6027,"num_object_clones":0,"num_object_copies":24108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6781,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6823","reported_seq":6887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.850294+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:41.850294+0000","last_peered":"2024-09-18T11:48:41.850294+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:41.850294+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6823,"log_dups_size":0,"ondisk_log_size":6823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49992950,"num_objects":6103,"num_object_clones":0,"num_object_copies":24412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6821,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3989}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6814","reported_seq":6879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.754944+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:40.754944+0000","last_peered":"2024-09-18T11:48:40.754944+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:40.754944+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6814,"log_dups_size":0,"ondisk_log_size":6814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49812726,"num_objects":6081,"num_object_clones":0,"num_object_copies":24324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6812,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6782","reported_seq":6833,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.552184+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:40.552184+0000","last_peered":"2024-09-18T11:48:40.552184+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:40.552184+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49184768,"num_objects":6004,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6912","reported_seq":6979,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.976100+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:41.976100+0000","last_peered":"2024-09-18T11:48:41.976100+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:41.976100+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50001142,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3970}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6799","reported_seq":6864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.908102+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:41.908102+0000","last_peered":"2024-09-18T11:48:41.908102+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:41.908102+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6799,"log_dups_size":0,"ondisk_log_size":6799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49771766,"num_objects":6076,"num_object_clones":0,"num_object_copies":24304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6985","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.827198+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:40.827198+0000","last_peered":"2024-09-18T11:48:40.827198+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:40.827198+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6985,"log_dups_size":0,"ondisk_log_size":6985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50711020,"num_objects":6191,"num_object_clones":0,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6982,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6682","reported_seq":6770,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.518538+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:41.518538+0000","last_peered":"2024-09-18T11:48:41.518538+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:41.518538+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6682,"log_dups_size":0,"ondisk_log_size":6682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48428278,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3938}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6883","reported_seq":6950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.760336+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:40.760336+0000","last_peered":"2024-09-18T11:48:40.760336+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:40.760336+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6883,"log_dups_size":0,"ondisk_log_size":6883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50376561,"num_objects":6150,"num_object_clones":0,"num_object_copies":24600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6880,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6849","reported_seq":6902,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.652228+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:40.652228+0000","last_peered":"2024-09-18T11:48:40.652228+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:40.652228+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6849,"log_dups_size":0,"ondisk_log_size":6849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50641531,"num_objects":6182,"num_object_clones":0,"num_object_copies":24728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4080}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6854","reported_seq":6932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.640765+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:41.640765+0000","last_peered":"2024-09-18T11:48:41.640765+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:41.640765+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50076283,"num_objects":6113,"num_object_clones":0,"num_object_copies":24452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6934","reported_seq":7015,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:40.693160+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:40.693160+0000","last_peered":"2024-09-18T11:48:40.693160+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:40.693160+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6934,"log_dups_size":0,"ondisk_log_size":6934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50427126,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6932,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4013}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6850","reported_seq":11135,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:40.764841+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:40.764841+0000","last_peered":"2024-09-18T11:48:40.764841+0000","last_clean":"2024-09-18T11:48:40.764841+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:40.764841+0000","last_undegraded":"2024-09-18T11:48:40.764841+0000","last_fullsized":"2024-09-18T11:48:40.764841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49583350,"num_objects":6053,"num_object_clones":0,"num_object_copies":24212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6727","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.662459+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:41.662459+0000","last_peered":"2024-09-18T11:48:41.662459+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:41.662459+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985334,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6725,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":799212940,"num_objects":97566,"num_object_clones":0,"num_object_copies":390264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3420,"num_bytes_recovered":27369472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1496039424,"data_stored":1496039424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109548,"ondisk_log_size":109548,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689244,"kb_used_data":391380,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93682596,"statfs":{"total":96636764160,"available":95930978304,"internally_reserved":0,"allocated":400773120,"data_stored":400605935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,3,1,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":6.3949999999999996},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":2.77}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":7.4080000000000004},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.923}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696000,"kb_used_data":390328,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93675840,"statfs":{"total":96636764160,"available":95924060160,"internally_reserved":0,"allocated":399695872,"data_stored":399528687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":6.8289999999999997},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.95599999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":4.5609999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.81899999999999995},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.65100000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597288,"kb_used_data":291168,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93774552,"statfs":{"total":96636764160,"available":96025141248,"internally_reserved":0,"allocated":298156032,"data_stored":297988847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.29699999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.19400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.22600000000000001},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.252}]}]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696808,"kb_used_data":390680,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93675032,"statfs":{"total":96636764160,"available":95923232768,"internally_reserved":0,"allocated":400056320,"data_stored":399889135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.19},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.246}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.27700000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":398876672,"data_stored":398876672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297877504,"data_stored":297877504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399216640,"data_stored":399216640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":400068608,"data_stored":400068608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:45.776 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:46.338 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:46.338 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:46.360 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-18T11:48:45.930775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":794510732,"num_objects":96992,"num_object_clones":0,"num_object_copies":387968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110095,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3420,"num_bytes_recovered":27369472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110122,"ondisk_log_size":110122,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2678272,"kb_used_data":1458320,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374809088,"statfs":{"total":386547056640,"available":383804506112,"internally_reserved":0,"allocated":1493319680,"data_stored":1492650940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,8,2,5,6,0,0,0,0,0,0,2,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25059328,"num_objects":-3059,"num_object_clones":0,"num_object_copies":-12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3059,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005236"},"pg_stats":[{"pgid":"1.f","version":"33'7055","reported_seq":7111,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.790489+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:45.790489+0000","last_peered":"2024-09-18T11:48:45.790489+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:45.790489+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50749440,"num_objects":6195,"num_object_clones":0,"num_object_copies":24780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4072}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6877","reported_seq":9447,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:42.033434+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:42.033434+0000","last_peered":"2024-09-18T11:48:42.033434+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:42.033434+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50615542,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1254,"num_bytes_recovered":10240000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6782","reported_seq":6867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.674540+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:41.674540+0000","last_peered":"2024-09-18T11:48:41.674540+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:41.674540+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49363603,"num_objects":6027,"num_object_clones":0,"num_object_copies":24108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6781,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6823","reported_seq":6887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.850294+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:41.850294+0000","last_peered":"2024-09-18T11:48:41.850294+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:41.850294+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6823,"log_dups_size":0,"ondisk_log_size":6823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49992950,"num_objects":6103,"num_object_clones":0,"num_object_copies":24412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6821,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3989}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6897","reported_seq":6962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.775226+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:45.775226+0000","last_peered":"2024-09-18T11:48:45.775226+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:45.775226+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49132790,"num_objects":5998,"num_object_clones":0,"num_object_copies":23992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6895,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6862","reported_seq":6913,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.709392+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:45.709392+0000","last_peered":"2024-09-18T11:48:45.709392+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:45.709392+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6862,"log_dups_size":0,"ondisk_log_size":6862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48529408,"num_objects":5924,"num_object_clones":0,"num_object_copies":23696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6862,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6912","reported_seq":6979,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.976100+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:41.976100+0000","last_peered":"2024-09-18T11:48:41.976100+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:41.976100+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50001142,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3970}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6799","reported_seq":6864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.908102+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:41.908102+0000","last_peered":"2024-09-18T11:48:41.908102+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:41.908102+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6799,"log_dups_size":0,"ondisk_log_size":6799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49771766,"num_objects":6076,"num_object_clones":0,"num_object_copies":24304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7063","reported_seq":7135,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.655962+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:45.655962+0000","last_peered":"2024-09-18T11:48:45.655962+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:45.655962+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7063,"log_dups_size":0,"ondisk_log_size":7063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50072044,"num_objects":6113,"num_object_clones":0,"num_object_copies":24452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6682","reported_seq":6770,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.518538+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:41.518538+0000","last_peered":"2024-09-18T11:48:41.518538+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:41.518538+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6682,"log_dups_size":0,"ondisk_log_size":6682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48428278,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3938}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6972","reported_seq":7039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.611284+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:45.611284+0000","last_peered":"2024-09-18T11:48:45.611284+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:45.611284+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6972,"log_dups_size":0,"ondisk_log_size":6972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49647473,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6969,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6938","reported_seq":6991,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.701407+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:45.701407+0000","last_peered":"2024-09-18T11:48:45.701407+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:45.701407+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49912443,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6937,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3991}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6854","reported_seq":6932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.640765+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:41.640765+0000","last_peered":"2024-09-18T11:48:41.640765+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:41.640765+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50076283,"num_objects":6113,"num_object_clones":0,"num_object_copies":24452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7029","reported_seq":7110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.806206+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:45.806206+0000","last_peered":"2024-09-18T11:48:45.806206+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:45.806206+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7029,"log_dups_size":0,"ondisk_log_size":7029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49648886,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7027,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3918}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6850","reported_seq":11135,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:40.764841+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:40.764841+0000","last_peered":"2024-09-18T11:48:40.764841+0000","last_clean":"2024-09-18T11:48:40.764841+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:40.764841+0000","last_undegraded":"2024-09-18T11:48:40.764841+0000","last_fullsized":"2024-09-18T11:48:40.764841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49583350,"num_objects":6053,"num_object_clones":0,"num_object_copies":24212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6727","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.662459+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:41.662459+0000","last_peered":"2024-09-18T11:48:41.662459+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:41.662459+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985334,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6725,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":794510732,"num_objects":96992,"num_object_clones":0,"num_object_copies":387968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110095,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3420,"num_bytes_recovered":27369472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1490755584,"data_stored":1490755584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110122,"ondisk_log_size":110122,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688176,"kb_used_data":386144,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93683664,"statfs":{"total":96636764160,"available":95932071936,"internally_reserved":0,"allocated":395411456,"data_stored":395244271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,0,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.8809999999999998},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.9060000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":10.887},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":2.706}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":0.20499999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":4.6760000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696000,"kb_used_data":390328,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93675840,"statfs":{"total":96636764160,"available":95924060160,"internally_reserved":0,"allocated":399695872,"data_stored":399528687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":6.8289999999999997},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.95599999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":4.5609999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.81899999999999995},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.65100000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597288,"kb_used_data":291168,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93774552,"statfs":{"total":96636764160,"available":96025141248,"internally_reserved":0,"allocated":298156032,"data_stored":297988847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.29699999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.19400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.22600000000000001},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.252}]}]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696808,"kb_used_data":390680,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93675032,"statfs":{"total":96636764160,"available":95923232768,"internally_reserved":0,"allocated":400056320,"data_stored":399889135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.19},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.246}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.27700000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":398876672,"data_stored":398876672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297877504,"data_stored":297877504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399216640,"data_stored":399216640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394784768,"data_stored":394784768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:46.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:46.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7055", 'reported_seq': 7111, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.790489+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:45.790489+0000', 'last_peered': '2024-09-18T11:48:45.790489+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:45.790489+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7055, 'log_dups_size': 0, 'ondisk_log_size': 7055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50749440, 'num_objects': 6195, 'num_object_clones': 0, 'num_object_copies': 24780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7055, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4072}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:46.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:46.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6877", 'reported_seq': 9447, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:42.033434+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:42.033434+0000', 'last_peered': '2024-09-18T11:48:42.033434+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:42.033434+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6877, 'log_dups_size': 0, 'ondisk_log_size': 6877, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50615542, 'num_objects': 6179, 'num_object_clones': 0, 'num_object_copies': 24716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 852, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6875, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1254, 'num_bytes_recovered': 10240000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5327}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:46.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:46.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6782", 'reported_seq': 6867, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:41.674540+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:41.674540+0000', 'last_peered': '2024-09-18T11:48:41.674540+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:41.674540+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6782, 'log_dups_size': 0, 'ondisk_log_size': 6782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49363603, 'num_objects': 6027, 'num_object_clones': 0, 'num_object_copies': 24108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6027, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6781, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3973}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:46.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:46.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6823", 'reported_seq': 6887, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:41.850294+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:41.850294+0000', 'last_peered': '2024-09-18T11:48:41.850294+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:41.850294+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6823, 'log_dups_size': 0, 'ondisk_log_size': 6823, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49992950, 'num_objects': 6103, 'num_object_clones': 0, 'num_object_copies': 24412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6821, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3989}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:46.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:46.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6897", 'reported_seq': 6962, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.775226+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:45.775226+0000', 'last_peered': '2024-09-18T11:48:45.775226+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:45.775226+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6897, 'log_dups_size': 0, 'ondisk_log_size': 6897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49132790, 'num_objects': 5998, 'num_object_clones': 0, 'num_object_copies': 23992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2080, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5998, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6895, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3920}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:46.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:46.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6862", 'reported_seq': 6913, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.709392+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:45.709392+0000', 'last_peered': '2024-09-18T11:48:45.709392+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:45.709392+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6862, 'log_dups_size': 0, 'ondisk_log_size': 6862, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48529408, 'num_objects': 5924, 'num_object_clones': 0, 'num_object_copies': 23696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2151, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6862, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3773}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:46.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:46.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6912", 'reported_seq': 6979, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:41.976100+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:41.976100+0000', 'last_peered': '2024-09-18T11:48:41.976100+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:41.976100+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6912, 'log_dups_size': 0, 'ondisk_log_size': 6912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50001142, 'num_objects': 6104, 'num_object_clones': 0, 'num_object_copies': 24416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6910, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3970}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:46.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:46.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6799", 'reported_seq': 6864, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:41.908102+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:41.908102+0000', 'last_peered': '2024-09-18T11:48:41.908102+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:41.908102+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6799, 'log_dups_size': 0, 'ondisk_log_size': 6799, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49771766, 'num_objects': 6076, 'num_object_clones': 0, 'num_object_copies': 24304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6797, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:46.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:46.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7063", 'reported_seq': 7135, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.655962+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:45.655962+0000', 'last_peered': '2024-09-18T11:48:45.655962+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:45.655962+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7063, 'log_dups_size': 0, 'ondisk_log_size': 7063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50072044, 'num_objects': 6113, 'num_object_clones': 0, 'num_object_copies': 24452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7060, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3984}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:46.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:46.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6682", 'reported_seq': 6770, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:41.518538+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:41.518538+0000', 'last_peered': '2024-09-18T11:48:41.518538+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:41.518538+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6682, 'log_dups_size': 0, 'ondisk_log_size': 6682, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48428278, 'num_objects': 5912, 'num_object_clones': 0, 'num_object_copies': 23648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6680, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3938}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:46.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:46.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6972", 'reported_seq': 7039, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.611284+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:45.611284+0000', 'last_peered': '2024-09-18T11:48:45.611284+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:45.611284+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6972, 'log_dups_size': 0, 'ondisk_log_size': 6972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49647473, 'num_objects': 6061, 'num_object_clones': 0, 'num_object_copies': 24244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2062, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6969, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4002}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:46.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:46.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6938", 'reported_seq': 6991, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.701407+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:45.701407+0000', 'last_peered': '2024-09-18T11:48:45.701407+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:45.701407+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6938, 'log_dups_size': 0, 'ondisk_log_size': 6938, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49912443, 'num_objects': 6093, 'num_object_clones': 0, 'num_object_copies': 24372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6937, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3991}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:46.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:46.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6854", 'reported_seq': 6932, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:41.640765+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:41.640765+0000', 'last_peered': '2024-09-18T11:48:41.640765+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:41.640765+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6854, 'log_dups_size': 0, 'ondisk_log_size': 6854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50076283, 'num_objects': 6113, 'num_object_clones': 0, 'num_object_copies': 24452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6853, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:46.365 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:46.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7029", 'reported_seq': 7110, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.806206+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:45.806206+0000', 'last_peered': '2024-09-18T11:48:45.806206+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:45.806206+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7029, 'log_dups_size': 0, 'ondisk_log_size': 7029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49648886, 'num_objects': 6061, 'num_object_clones': 0, 'num_object_copies': 24244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7027, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3918}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:46.365 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:46.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6727", 'reported_seq': 6793, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:41.662459+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:41.662459+0000', 'last_peered': '2024-09-18T11:48:41.662459+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:41.662459+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6727, 'log_dups_size': 0, 'ondisk_log_size': 6727, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48985334, 'num_objects': 5980, 'num_object_clones': 0, 'num_object_copies': 23920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2014, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6725, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3968}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:46.365 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:47.064 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:47.064 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30036/387968 objects degraded (7.742%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":325,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":96992,"data_bytes":794510732,"bytes_used":2742550528,"bytes_avail":383804506112,"bytes_total":386547056640,"degraded_objects":30036,"degraded_total":387968,"degraded_ratio":0.077418756186077201,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":125555,"recovering_keys_per_sec":0,"num_objects_recovered":197,"num_bytes_recovered":1507328,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":254},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:47.092 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:47.093 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:47.666 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:47.667 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:47.695 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-18T11:48:45.930775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":794510732,"num_objects":96992,"num_object_clones":0,"num_object_copies":387968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110095,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3420,"num_bytes_recovered":27369472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110122,"ondisk_log_size":110122,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2678272,"kb_used_data":1458320,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374809088,"statfs":{"total":386547056640,"available":383804506112,"internally_reserved":0,"allocated":1493319680,"data_stored":1492650940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,8,2,5,6,0,0,0,0,0,0,2,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25059328,"num_objects":-3059,"num_object_clones":0,"num_object_copies":-12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3059,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005236"},"pg_stats":[{"pgid":"1.f","version":"33'7055","reported_seq":7111,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.790489+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:45.790489+0000","last_peered":"2024-09-18T11:48:45.790489+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:45.790489+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50749440,"num_objects":6195,"num_object_clones":0,"num_object_copies":24780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4072}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6877","reported_seq":9447,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:42.033434+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:42.033434+0000","last_peered":"2024-09-18T11:48:42.033434+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:42.033434+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50615542,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1254,"num_bytes_recovered":10240000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6782","reported_seq":6867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.674540+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:41.674540+0000","last_peered":"2024-09-18T11:48:41.674540+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:41.674540+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49363603,"num_objects":6027,"num_object_clones":0,"num_object_copies":24108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6781,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6823","reported_seq":6887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.850294+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:41.850294+0000","last_peered":"2024-09-18T11:48:41.850294+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:41.850294+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6823,"log_dups_size":0,"ondisk_log_size":6823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49992950,"num_objects":6103,"num_object_clones":0,"num_object_copies":24412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6821,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3989}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6897","reported_seq":6962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.775226+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:45.775226+0000","last_peered":"2024-09-18T11:48:45.775226+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:45.775226+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49132790,"num_objects":5998,"num_object_clones":0,"num_object_copies":23992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6895,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6862","reported_seq":6913,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.709392+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:45.709392+0000","last_peered":"2024-09-18T11:48:45.709392+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:45.709392+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6862,"log_dups_size":0,"ondisk_log_size":6862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48529408,"num_objects":5924,"num_object_clones":0,"num_object_copies":23696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6862,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6912","reported_seq":6979,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.976100+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:41.976100+0000","last_peered":"2024-09-18T11:48:41.976100+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:41.976100+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50001142,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3970}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6799","reported_seq":6864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.908102+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:41.908102+0000","last_peered":"2024-09-18T11:48:41.908102+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:41.908102+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6799,"log_dups_size":0,"ondisk_log_size":6799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49771766,"num_objects":6076,"num_object_clones":0,"num_object_copies":24304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7063","reported_seq":7135,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.655962+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:45.655962+0000","last_peered":"2024-09-18T11:48:45.655962+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:45.655962+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7063,"log_dups_size":0,"ondisk_log_size":7063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50072044,"num_objects":6113,"num_object_clones":0,"num_object_copies":24452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6682","reported_seq":6770,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.518538+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:41.518538+0000","last_peered":"2024-09-18T11:48:41.518538+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:41.518538+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6682,"log_dups_size":0,"ondisk_log_size":6682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48428278,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3938}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6972","reported_seq":7039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.611284+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:45.611284+0000","last_peered":"2024-09-18T11:48:45.611284+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:45.611284+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6972,"log_dups_size":0,"ondisk_log_size":6972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49647473,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6969,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6938","reported_seq":6991,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.701407+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:45.701407+0000","last_peered":"2024-09-18T11:48:45.701407+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:45.701407+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49912443,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6937,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3991}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6854","reported_seq":6932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.640765+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:41.640765+0000","last_peered":"2024-09-18T11:48:41.640765+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:41.640765+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50076283,"num_objects":6113,"num_object_clones":0,"num_object_copies":24452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7029","reported_seq":7110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.806206+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:45.806206+0000","last_peered":"2024-09-18T11:48:45.806206+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:45.806206+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7029,"log_dups_size":0,"ondisk_log_size":7029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49648886,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7027,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3918}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6850","reported_seq":11135,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:40.764841+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:40.764841+0000","last_peered":"2024-09-18T11:48:40.764841+0000","last_clean":"2024-09-18T11:48:40.764841+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:40.764841+0000","last_undegraded":"2024-09-18T11:48:40.764841+0000","last_fullsized":"2024-09-18T11:48:40.764841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49583350,"num_objects":6053,"num_object_clones":0,"num_object_copies":24212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6727","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:41.662459+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:41.662459+0000","last_peered":"2024-09-18T11:48:41.662459+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:41.662459+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985334,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6725,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":794510732,"num_objects":96992,"num_object_clones":0,"num_object_copies":387968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110095,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3420,"num_bytes_recovered":27369472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1490755584,"data_stored":1490755584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110122,"ondisk_log_size":110122,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688176,"kb_used_data":386144,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93683664,"statfs":{"total":96636764160,"available":95932071936,"internally_reserved":0,"allocated":395411456,"data_stored":395244271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,0,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.8809999999999998},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.9060000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":10.887},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":2.706}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":0.20499999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":4.6760000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696000,"kb_used_data":390328,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93675840,"statfs":{"total":96636764160,"available":95924060160,"internally_reserved":0,"allocated":399695872,"data_stored":399528687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":6.8289999999999997},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.95599999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":4.5609999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.81899999999999995},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.65100000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597288,"kb_used_data":291168,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93774552,"statfs":{"total":96636764160,"available":96025141248,"internally_reserved":0,"allocated":298156032,"data_stored":297988847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.29699999999999999},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":0.19400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":0.22600000000000001},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.252}]}]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696808,"kb_used_data":390680,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93675032,"statfs":{"total":96636764160,"available":95923232768,"internally_reserved":0,"allocated":400056320,"data_stored":399889135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.19},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.246}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.27700000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":398876672,"data_stored":398876672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297877504,"data_stored":297877504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399216640,"data_stored":399216640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394784768,"data_stored":394784768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:50.697 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:51.254 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:51.254 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:51.282 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-18T11:48:49.932390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":788546956,"num_objects":96264,"num_object_clones":0,"num_object_copies":385056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110823,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3501,"num_bytes_recovered":27901952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110850,"ondisk_log_size":110850,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680084,"kb_used_data":1447648,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":374807276,"statfs":{"total":386547056640,"available":383802650624,"internally_reserved":0,"allocated":1482391552,"data_stored":1481722812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,1,9,8,5,6,1,0,0,0,4,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20905984,"num_objects":-2552,"num_object_clones":0,"num_object_copies":-10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2552,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1425408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005918"},"pg_stats":[{"pgid":"1.f","version":"33'7055","reported_seq":7111,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.790489+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:45.790489+0000","last_peered":"2024-09-18T11:48:45.790489+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:45.790489+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50749440,"num_objects":6195,"num_object_clones":0,"num_object_copies":24780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4072}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6963","reported_seq":9663,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:47.088391+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:47.088391+0000","last_peered":"2024-09-18T11:48:47.088391+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:47.088391+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6963,"log_dups_size":0,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49911030,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":787,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1319,"num_bytes_recovered":10772480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5306}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6861","reported_seq":6948,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.636033+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:46.636033+0000","last_peered":"2024-09-18T11:48:46.636033+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:46.636033+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48716435,"num_objects":5948,"num_object_clones":0,"num_object_copies":23792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6860,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3895}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6914","reported_seq":6986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.949330+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:46.949330+0000","last_peered":"2024-09-18T11:48:46.949330+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:46.949330+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49247478,"num_objects":6012,"num_object_clones":0,"num_object_copies":24048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6897","reported_seq":6962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.775226+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:45.775226+0000","last_peered":"2024-09-18T11:48:45.775226+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:45.775226+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49132790,"num_objects":5998,"num_object_clones":0,"num_object_copies":23992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6895,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6862","reported_seq":6913,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.709392+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:45.709392+0000","last_peered":"2024-09-18T11:48:45.709392+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:45.709392+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6862,"log_dups_size":0,"ondisk_log_size":6862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48529408,"num_objects":5924,"num_object_clones":0,"num_object_copies":23696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6862,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6977","reported_seq":7050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.937364+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:46.937364+0000","last_peered":"2024-09-18T11:48:46.937364+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:46.937364+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49468662,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3908}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6886","reported_seq":6957,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.882398+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:46.882398+0000","last_peered":"2024-09-18T11:48:46.882398+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:46.882398+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49059062,"num_objects":5989,"num_object_clones":0,"num_object_copies":23956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6884,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7063","reported_seq":7135,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.655962+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:45.655962+0000","last_peered":"2024-09-18T11:48:45.655962+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:45.655962+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7063,"log_dups_size":0,"ondisk_log_size":7063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50072044,"num_objects":6113,"num_object_clones":0,"num_object_copies":24452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6751","reported_seq":6839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.733296+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:46.733296+0000","last_peered":"2024-09-18T11:48:46.733296+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:46.733296+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6751,"log_dups_size":0,"ondisk_log_size":6751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47863030,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6749,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3869}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6972","reported_seq":7039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.611284+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:45.611284+0000","last_peered":"2024-09-18T11:48:45.611284+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:45.611284+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6972,"log_dups_size":0,"ondisk_log_size":6972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49647473,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6969,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6938","reported_seq":6991,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.701407+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:45.701407+0000","last_peered":"2024-09-18T11:48:45.701407+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:45.701407+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49912443,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6937,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3991}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6939","reported_seq":7017,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.692386+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:46.692386+0000","last_peered":"2024-09-18T11:48:46.692386+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:46.692386+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6939,"log_dups_size":0,"ondisk_log_size":6939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49379963,"num_objects":6028,"num_object_clones":0,"num_object_copies":24112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6938,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3974}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7029","reported_seq":7110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.806206+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:45.806206+0000","last_peered":"2024-09-18T11:48:45.806206+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:45.806206+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7029,"log_dups_size":0,"ondisk_log_size":7029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49648886,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7027,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3918}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6931","reported_seq":11216,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:45.799681+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:45.799681+0000","last_peered":"2024-09-18T11:48:45.799681+0000","last_clean":"2024-09-18T11:48:45.799681+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:45.799681+0000","last_undegraded":"2024-09-18T11:48:45.799681+0000","last_fullsized":"2024-09-18T11:48:45.799681+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6931,"log_dups_size":0,"ondisk_log_size":6931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48919798,"num_objects":5972,"num_object_clones":0,"num_object_copies":23888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6929,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6812","reported_seq":6888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.788486+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:46.788486+0000","last_peered":"2024-09-18T11:48:46.788486+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:46.788486+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48289014,"num_objects":5895,"num_object_clones":0,"num_object_copies":23580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6810,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":788546956,"num_objects":96264,"num_object_clones":0,"num_object_copies":385056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110823,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3501,"num_bytes_recovered":27901952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1479630848,"data_stored":1479630848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110850,"ondisk_log_size":110850,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688176,"kb_used_data":386144,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93683664,"statfs":{"total":96636764160,"available":95932071936,"internally_reserved":0,"allocated":395411456,"data_stored":395244271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,0,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.8809999999999998},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.9060000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":10.887},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":2.706}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":0.20499999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":4.6760000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695048,"kb_used_data":385216,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93676792,"statfs":{"total":96636764160,"available":95925035008,"internally_reserved":0,"allocated":394461184,"data_stored":394293999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,4,0,2,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":1.236},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.286}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":3.1230000000000002},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":1.403}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":1.2609999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.81799999999999995}]}]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601088,"kb_used_data":290808,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93770752,"statfs":{"total":96636764160,"available":96021250048,"internally_reserved":0,"allocated":297787392,"data_stored":297620207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.4660000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":9.5510000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":4.4370000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":4.4089999999999998},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.222}]}]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695772,"kb_used_data":385480,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93676068,"statfs":{"total":96636764160,"available":95924293632,"internally_reserved":0,"allocated":394731520,"data_stored":394564335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,2,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.307},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.38900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.54800000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.538},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":11.359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393551872,"data_stored":393551872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297451520,"data_stored":297451520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":393842688,"data_stored":393842688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394784768,"data_stored":394784768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:51.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:51.886 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:51.888 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:51.918 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-18T11:48:49.932390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":788546956,"num_objects":96264,"num_object_clones":0,"num_object_copies":385056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110823,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3501,"num_bytes_recovered":27901952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110850,"ondisk_log_size":110850,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680084,"kb_used_data":1447648,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":374807276,"statfs":{"total":386547056640,"available":383802650624,"internally_reserved":0,"allocated":1482391552,"data_stored":1481722812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,1,9,8,5,6,1,0,0,0,4,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20905984,"num_objects":-2552,"num_object_clones":0,"num_object_copies":-10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2552,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1425408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005918"},"pg_stats":[{"pgid":"1.f","version":"33'7055","reported_seq":7111,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.790489+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:45.790489+0000","last_peered":"2024-09-18T11:48:45.790489+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:45.790489+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50749440,"num_objects":6195,"num_object_clones":0,"num_object_copies":24780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4072}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6963","reported_seq":9663,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:47.088391+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:47.088391+0000","last_peered":"2024-09-18T11:48:47.088391+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:47.088391+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6963,"log_dups_size":0,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49911030,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":787,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1319,"num_bytes_recovered":10772480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5306}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6861","reported_seq":6948,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.636033+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:46.636033+0000","last_peered":"2024-09-18T11:48:46.636033+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:46.636033+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48716435,"num_objects":5948,"num_object_clones":0,"num_object_copies":23792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6860,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3895}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6914","reported_seq":6986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.949330+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:46.949330+0000","last_peered":"2024-09-18T11:48:46.949330+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:46.949330+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49247478,"num_objects":6012,"num_object_clones":0,"num_object_copies":24048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6897","reported_seq":6962,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.775226+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:45.775226+0000","last_peered":"2024-09-18T11:48:45.775226+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:45.775226+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49132790,"num_objects":5998,"num_object_clones":0,"num_object_copies":23992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6895,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6862","reported_seq":6913,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.709392+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:45.709392+0000","last_peered":"2024-09-18T11:48:45.709392+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:45.709392+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6862,"log_dups_size":0,"ondisk_log_size":6862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48529408,"num_objects":5924,"num_object_clones":0,"num_object_copies":23696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6862,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6977","reported_seq":7050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.937364+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:46.937364+0000","last_peered":"2024-09-18T11:48:46.937364+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:46.937364+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49468662,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3908}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6886","reported_seq":6957,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.882398+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:46.882398+0000","last_peered":"2024-09-18T11:48:46.882398+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:46.882398+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49059062,"num_objects":5989,"num_object_clones":0,"num_object_copies":23956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6884,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7063","reported_seq":7135,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.655962+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:45.655962+0000","last_peered":"2024-09-18T11:48:45.655962+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:45.655962+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7063,"log_dups_size":0,"ondisk_log_size":7063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50072044,"num_objects":6113,"num_object_clones":0,"num_object_copies":24452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6751","reported_seq":6839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.733296+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:46.733296+0000","last_peered":"2024-09-18T11:48:46.733296+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:46.733296+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6751,"log_dups_size":0,"ondisk_log_size":6751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47863030,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6749,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3869}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6972","reported_seq":7039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.611284+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:45.611284+0000","last_peered":"2024-09-18T11:48:45.611284+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:45.611284+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6972,"log_dups_size":0,"ondisk_log_size":6972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49647473,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6969,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6938","reported_seq":6991,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.701407+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:45.701407+0000","last_peered":"2024-09-18T11:48:45.701407+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:45.701407+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49912443,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6937,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3991}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6939","reported_seq":7017,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.692386+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:46.692386+0000","last_peered":"2024-09-18T11:48:46.692386+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:46.692386+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6939,"log_dups_size":0,"ondisk_log_size":6939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49379963,"num_objects":6028,"num_object_clones":0,"num_object_copies":24112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6938,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3974}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7029","reported_seq":7110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:45.806206+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:45.806206+0000","last_peered":"2024-09-18T11:48:45.806206+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:45.806206+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7029,"log_dups_size":0,"ondisk_log_size":7029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49648886,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7027,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3918}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6931","reported_seq":11216,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:45.799681+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:45.799681+0000","last_peered":"2024-09-18T11:48:45.799681+0000","last_clean":"2024-09-18T11:48:45.799681+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:45.799681+0000","last_undegraded":"2024-09-18T11:48:45.799681+0000","last_fullsized":"2024-09-18T11:48:45.799681+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6931,"log_dups_size":0,"ondisk_log_size":6931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48919798,"num_objects":5972,"num_object_clones":0,"num_object_copies":23888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6929,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6812","reported_seq":6888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.788486+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:46.788486+0000","last_peered":"2024-09-18T11:48:46.788486+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:46.788486+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48289014,"num_objects":5895,"num_object_clones":0,"num_object_copies":23580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6810,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":788546956,"num_objects":96264,"num_object_clones":0,"num_object_copies":385056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110823,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3501,"num_bytes_recovered":27901952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1479630848,"data_stored":1479630848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110850,"ondisk_log_size":110850,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688176,"kb_used_data":386144,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93683664,"statfs":{"total":96636764160,"available":95932071936,"internally_reserved":0,"allocated":395411456,"data_stored":395244271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,0,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.8809999999999998},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":2.9060000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":10.887},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":2.706}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":0.20499999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":4.6760000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695048,"kb_used_data":385216,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93676792,"statfs":{"total":96636764160,"available":95925035008,"internally_reserved":0,"allocated":394461184,"data_stored":394293999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,4,0,2,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":1.236},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.286}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":3.1230000000000002},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":1.403}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":1.2609999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.81799999999999995}]}]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601088,"kb_used_data":290808,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93770752,"statfs":{"total":96636764160,"available":96021250048,"internally_reserved":0,"allocated":297787392,"data_stored":297620207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.4660000000000002},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":9.5510000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":4.4370000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":4.4089999999999998},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.222}]}]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695772,"kb_used_data":385480,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93676068,"statfs":{"total":96636764160,"available":95924293632,"internally_reserved":0,"allocated":394731520,"data_stored":394564335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,2,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.307},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.38900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.54800000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.538},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":11.359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393551872,"data_stored":393551872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297451520,"data_stored":297451520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":393842688,"data_stored":393842688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394784768,"data_stored":394784768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:51.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:51.920 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7055", 'reported_seq': 7111, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.790489+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:45.790489+0000', 'last_peered': '2024-09-18T11:48:45.790489+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:45.790489+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7055, 'log_dups_size': 0, 'ondisk_log_size': 7055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50749440, 'num_objects': 6195, 'num_object_clones': 0, 'num_object_copies': 24780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7055, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4072}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:51.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:51.920 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6963", 'reported_seq': 9663, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:47.088391+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:47.088391+0000', 'last_peered': '2024-09-18T11:48:47.088391+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:47.088391+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6963, 'log_dups_size': 0, 'ondisk_log_size': 6963, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49911030, 'num_objects': 6093, 'num_object_clones': 0, 'num_object_copies': 24372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 787, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6961, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1319, 'num_bytes_recovered': 10772480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:51.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:51.920 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6861", 'reported_seq': 6948, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:46.636033+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:46.636033+0000', 'last_peered': '2024-09-18T11:48:46.636033+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:46.636033+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6861, 'log_dups_size': 0, 'ondisk_log_size': 6861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48716435, 'num_objects': 5948, 'num_object_clones': 0, 'num_object_copies': 23792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5948, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6860, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3895}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:51.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:51.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6914", 'reported_seq': 6986, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:46.949330+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:46.949330+0000', 'last_peered': '2024-09-18T11:48:46.949330+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:46.949330+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6914, 'log_dups_size': 0, 'ondisk_log_size': 6914, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49247478, 'num_objects': 6012, 'num_object_clones': 0, 'num_object_copies': 24048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6912, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:51.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:51.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6897", 'reported_seq': 6962, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.775226+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:45.775226+0000', 'last_peered': '2024-09-18T11:48:45.775226+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:45.775226+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6897, 'log_dups_size': 0, 'ondisk_log_size': 6897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49132790, 'num_objects': 5998, 'num_object_clones': 0, 'num_object_copies': 23992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2080, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5998, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6895, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3920}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:51.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:51.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6862", 'reported_seq': 6913, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.709392+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:45.709392+0000', 'last_peered': '2024-09-18T11:48:45.709392+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:45.709392+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6862, 'log_dups_size': 0, 'ondisk_log_size': 6862, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48529408, 'num_objects': 5924, 'num_object_clones': 0, 'num_object_copies': 23696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2151, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6862, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3773}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:51.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:51.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6977", 'reported_seq': 7050, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:46.937364+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:46.937364+0000', 'last_peered': '2024-09-18T11:48:46.937364+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:46.937364+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6977, 'log_dups_size': 0, 'ondisk_log_size': 6977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49468662, 'num_objects': 6039, 'num_object_clones': 0, 'num_object_copies': 24156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6975, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3908}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:51.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:51.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6886", 'reported_seq': 6957, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:46.882398+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:46.882398+0000', 'last_peered': '2024-09-18T11:48:46.882398+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:46.882398+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6886, 'log_dups_size': 0, 'ondisk_log_size': 6886, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49059062, 'num_objects': 5989, 'num_object_clones': 0, 'num_object_copies': 23956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6884, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3958}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:51.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:51.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7063", 'reported_seq': 7135, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.655962+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:45.655962+0000', 'last_peered': '2024-09-18T11:48:45.655962+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:45.655962+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7063, 'log_dups_size': 0, 'ondisk_log_size': 7063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50072044, 'num_objects': 6113, 'num_object_clones': 0, 'num_object_copies': 24452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7060, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3984}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:51.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:51.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6751", 'reported_seq': 6839, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:46.733296+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:46.733296+0000', 'last_peered': '2024-09-18T11:48:46.733296+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:46.733296+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6751, 'log_dups_size': 0, 'ondisk_log_size': 6751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47863030, 'num_objects': 5843, 'num_object_clones': 0, 'num_object_copies': 23372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6749, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3869}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:51.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:51.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6972", 'reported_seq': 7039, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.611284+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:45.611284+0000', 'last_peered': '2024-09-18T11:48:45.611284+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:45.611284+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6972, 'log_dups_size': 0, 'ondisk_log_size': 6972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49647473, 'num_objects': 6061, 'num_object_clones': 0, 'num_object_copies': 24244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2062, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6969, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4002}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:51.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:51.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6938", 'reported_seq': 6991, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.701407+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:45.701407+0000', 'last_peered': '2024-09-18T11:48:45.701407+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:45.701407+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6938, 'log_dups_size': 0, 'ondisk_log_size': 6938, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49912443, 'num_objects': 6093, 'num_object_clones': 0, 'num_object_copies': 24372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6937, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3991}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:51.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:51.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6939", 'reported_seq': 7017, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:46.692386+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:46.692386+0000', 'last_peered': '2024-09-18T11:48:46.692386+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:46.692386+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6939, 'log_dups_size': 0, 'ondisk_log_size': 6939, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49379963, 'num_objects': 6028, 'num_object_clones': 0, 'num_object_copies': 24112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6938, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3974}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:51.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:51.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7029", 'reported_seq': 7110, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:45.806206+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:45.806206+0000', 'last_peered': '2024-09-18T11:48:45.806206+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:45.806206+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7029, 'log_dups_size': 0, 'ondisk_log_size': 7029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49648886, 'num_objects': 6061, 'num_object_clones': 0, 'num_object_copies': 24244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7027, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3918}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:51.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:51.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6812", 'reported_seq': 6888, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:46.788486+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:46.788486+0000', 'last_peered': '2024-09-18T11:48:46.788486+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:46.788486+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6812, 'log_dups_size': 0, 'ondisk_log_size': 6812, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48289014, 'num_objects': 5895, 'num_object_clones': 0, 'num_object_copies': 23580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5895, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6810, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3888}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:51.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:52.567 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:52.567 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29955/381348 objects degraded (7.855%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":330,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":95337,"data_bytes":780952972,"bytes_used":2715996160,"bytes_avail":383831060480,"bytes_total":386547056640,"degraded_objects":29955,"degraded_total":381348,"degraded_ratio":0.078550300512917337,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":118724,"recovering_keys_per_sec":0,"num_objects_recovered":190,"num_bytes_recovered":1425408,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":289},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:52.593 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:52.594 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:53.190 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:53.191 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:53.211 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-18T11:48:51.932990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":780952972,"num_objects":95337,"num_object_clones":0,"num_object_copies":381348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111750,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3501,"num_bytes_recovered":27901952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111777,"ondisk_log_size":111777,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2652340,"kb_used_data":1436528,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374835020,"statfs":{"total":386547056640,"available":383831060480,"internally_reserved":0,"allocated":1471004672,"data_stored":1470335932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,3,5,7,10,6,6,0,0,0,3,0,1,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28499968,"num_objects":-3479,"num_object_clones":0,"num_object_copies":-13916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1425408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006025"},"pg_stats":[{"pgid":"1.f","version":"33'7138","reported_seq":7195,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:50.820211+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:50.820211+0000","last_peered":"2024-09-18T11:48:50.820211+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:50.820211+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7138,"log_dups_size":0,"ondisk_log_size":7138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50069504,"num_objects":6112,"num_object_clones":0,"num_object_copies":24448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7138,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6963","reported_seq":9663,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:47.088391+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:47.088391+0000","last_peered":"2024-09-18T11:48:47.088391+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:47.088391+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6963,"log_dups_size":0,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49911030,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":787,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1319,"num_bytes_recovered":10772480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5306}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6952","reported_seq":7039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:51.789215+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:51.789215+0000","last_peered":"2024-09-18T11:48:51.789215+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:51.789215+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6952,"log_dups_size":0,"ondisk_log_size":6952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47970963,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6951,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6914","reported_seq":6986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.949330+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:46.949330+0000","last_peered":"2024-09-18T11:48:46.949330+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:46.949330+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49247478,"num_objects":6012,"num_object_clones":0,"num_object_copies":24048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6977","reported_seq":7042,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:50.635917+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:50.635917+0000","last_peered":"2024-09-18T11:48:50.635917+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:50.635917+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48477430,"num_objects":5918,"num_object_clones":0,"num_object_copies":23672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6935","reported_seq":6984,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:50.791327+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:50.791327+0000","last_peered":"2024-09-18T11:48:50.791327+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:50.791327+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6935,"log_dups_size":0,"ondisk_log_size":6935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47931392,"num_objects":5851,"num_object_clones":0,"num_object_copies":23404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6935,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6977","reported_seq":7050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.937364+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:46.937364+0000","last_peered":"2024-09-18T11:48:46.937364+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:46.937364+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49468662,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3908}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6886","reported_seq":6957,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.882398+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:46.882398+0000","last_peered":"2024-09-18T11:48:46.882398+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:46.882398+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49059062,"num_objects":5989,"num_object_clones":0,"num_object_copies":23956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6884,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7145","reported_seq":7216,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:50.857629+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:50.857629+0000","last_peered":"2024-09-18T11:48:50.857629+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:50.857629+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7145,"log_dups_size":0,"ondisk_log_size":7145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49400300,"num_objects":6031,"num_object_clones":0,"num_object_copies":24124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7142,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6816","reported_seq":6904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:51.737100+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:51.737100+0000","last_peered":"2024-09-18T11:48:51.737100+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:51.737100+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47330550,"num_objects":5778,"num_object_clones":0,"num_object_copies":23112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7058","reported_seq":7122,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:50.764186+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:50.764186+0000","last_peered":"2024-09-18T11:48:50.764186+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:50.764186+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48942961,"num_objects":5975,"num_object_clones":0,"num_object_copies":23900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7032","reported_seq":7085,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:50.765554+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:50.765554+0000","last_peered":"2024-09-18T11:48:50.765554+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:50.765554+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7032,"log_dups_size":0,"ondisk_log_size":7032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49142395,"num_objects":5999,"num_object_clones":0,"num_object_copies":23996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7031,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7020","reported_seq":7098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:51.706288+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:51.706288+0000","last_peered":"2024-09-18T11:48:51.706288+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:51.706288+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7020,"log_dups_size":0,"ondisk_log_size":7020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48716411,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7019,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7130","reported_seq":7210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:50.801397+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:50.801397+0000","last_peered":"2024-09-18T11:48:50.801397+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:50.801397+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7130,"log_dups_size":0,"ondisk_log_size":7130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48821494,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7128,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3817}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7022","reported_seq":11307,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:50.892408+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:50.892408+0000","last_peered":"2024-09-18T11:48:50.892408+0000","last_clean":"2024-09-18T11:48:50.892408+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:50.892408+0000","last_undegraded":"2024-09-18T11:48:50.892408+0000","last_fullsized":"2024-09-18T11:48:50.892408+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48174326,"num_objects":5881,"num_object_clones":0,"num_object_copies":23524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6812","reported_seq":6888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:46.788486+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:46.788486+0000","last_peered":"2024-09-18T11:48:46.788486+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:46.788486+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48289014,"num_objects":5895,"num_object_clones":0,"num_object_copies":23580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6810,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":780952972,"num_objects":95337,"num_object_clones":0,"num_object_copies":381348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111750,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3501,"num_bytes_recovered":27901952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1468178432,"data_stored":1468178432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111777,"ondisk_log_size":111777,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691124,"kb_used_data":380760,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93680716,"statfs":{"total":96636764160,"available":95929053184,"internally_reserved":0,"allocated":389898240,"data_stored":389731055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1,3,4,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.5810000000000004},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":5.6130000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":7.1550000000000002,"5min":5.6109999999999998,"15min":5.5250000000000004},"min":{"1min":0.73299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":3.831,"5min":4.1070000000000002,"15min":4.1689999999999996},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":21.132000000000001,"5min":21.132000000000001,"15min":21.132000000000001},"last":10.787000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":0.26400000000000001},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":4.7610000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677328,"kb_used_data":379972,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93694512,"statfs":{"total":96636764160,"available":95943180288,"internally_reserved":0,"allocated":389091328,"data_stored":388924143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,3,5,0,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":14.281000000000001},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":4.1349999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.456},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":3.7389999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":3.706},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":3.153}]}]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588116,"kb_used_data":290316,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93783724,"statfs":{"total":96636764160,"available":96034533376,"internally_reserved":0,"allocated":297283584,"data_stored":297116399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":7.9379999999999997},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":11.050000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":11.289},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":12.061}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":11.007},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.3609999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695772,"kb_used_data":385480,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93676068,"statfs":{"total":96636764160,"available":95924293632,"internally_reserved":0,"allocated":394731520,"data_stored":394564335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,2,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":0.307},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.38900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.54800000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.538},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":11.359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393551872,"data_stored":393551872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296972288,"data_stored":296972288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":388345856,"data_stored":388345856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389308416,"data_stored":389308416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:56.213 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:56.788 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:56.788 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:56.815 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-18T11:48:55.934574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":772793740,"num_objects":94341,"num_object_clones":0,"num_object_copies":377364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94341,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112746,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":28393472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112773,"ondisk_log_size":112773,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2616820,"kb_used_data":1425980,"kb_used_omap":14,"kb_used_meta":1190769,"kb_avail":374870540,"statfs":{"total":386547056640,"available":383867432960,"internally_reserved":0,"allocated":1460203520,"data_stored":1459534780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1219347483},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,3,6,8,14,8,3,0,0,0,0,2,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26419200,"num_objects":-3225,"num_object_clones":0,"num_object_copies":-12900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3225,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004831"},"pg_stats":[{"pgid":"1.f","version":"33'7218","reported_seq":7275,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.729054+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:55.729054+0000","last_peered":"2024-09-18T11:48:55.729054+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:55.729054+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7218,"log_dups_size":0,"ondisk_log_size":7218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49414144,"num_objects":6032,"num_object_clones":0,"num_object_copies":24128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7218,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7058","reported_seq":9878,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:52.119970+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:52.119970+0000","last_peered":"2024-09-18T11:48:52.119970+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:52.119970+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49132790,"num_objects":5998,"num_object_clones":0,"num_object_copies":23992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7056,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1379,"num_bytes_recovered":11264000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5271}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6952","reported_seq":7039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:51.789215+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:51.789215+0000","last_peered":"2024-09-18T11:48:51.789215+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:51.789215+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6952,"log_dups_size":0,"ondisk_log_size":6952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47970963,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6951,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6998","reported_seq":7070,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:52.094566+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:52.094566+0000","last_peered":"2024-09-18T11:48:52.094566+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:52.094566+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48559350,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7043","reported_seq":7108,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.718547+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:55.718547+0000","last_peered":"2024-09-18T11:48:55.718547+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:55.718547+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47936758,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7011","reported_seq":7065,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.862815+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:55.862815+0000","last_peered":"2024-09-18T11:48:55.862815+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:55.862815+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7011,"log_dups_size":0,"ondisk_log_size":7011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47308800,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7011,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7049","reported_seq":7122,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:52.036143+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:52.036143+0000","last_peered":"2024-09-18T11:48:52.036143+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:52.036143+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7049,"log_dups_size":0,"ondisk_log_size":7049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48878838,"num_objects":5967,"num_object_clones":0,"num_object_copies":23868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7047,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3836}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6963","reported_seq":7034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:52.035707+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:52.035707+0000","last_peered":"2024-09-18T11:48:52.035707+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:52.035707+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6963,"log_dups_size":0,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48428278,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7241","reported_seq":7313,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.827909+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:55.827909+0000","last_peered":"2024-09-18T11:48:55.827909+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:55.827909+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48613868,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6816","reported_seq":6904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:51.737100+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:51.737100+0000","last_peered":"2024-09-18T11:48:51.737100+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:51.737100+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47330550,"num_objects":5778,"num_object_clones":0,"num_object_copies":23112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7151","reported_seq":7220,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.868241+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:55.868241+0000","last_peered":"2024-09-18T11:48:55.868241+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:55.868241+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7151,"log_dups_size":0,"ondisk_log_size":7151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48181105,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7148,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7120","reported_seq":7173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.840417+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:55.840417+0000","last_peered":"2024-09-18T11:48:55.840417+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:55.840417+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7120,"log_dups_size":0,"ondisk_log_size":7120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48421499,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7119,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3809}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7020","reported_seq":7098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:51.706288+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:51.706288+0000","last_peered":"2024-09-18T11:48:51.706288+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:51.706288+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7020,"log_dups_size":0,"ondisk_log_size":7020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48716411,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7019,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7210","reported_seq":7291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.836536+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:55.836536+0000","last_peered":"2024-09-18T11:48:55.836536+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:55.836536+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48166134,"num_objects":5880,"num_object_clones":0,"num_object_copies":23520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7208,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3737}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7022","reported_seq":11307,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:50.892408+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:50.892408+0000","last_peered":"2024-09-18T11:48:50.892408+0000","last_clean":"2024-09-18T11:48:50.892408+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:50.892408+0000","last_undegraded":"2024-09-18T11:48:50.892408+0000","last_fullsized":"2024-09-18T11:48:50.892408+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48174326,"num_objects":5881,"num_object_clones":0,"num_object_copies":23524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6901","reported_seq":6977,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:52.036125+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:52.036125+0000","last_peered":"2024-09-18T11:48:52.036125+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:52.036125+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6901,"log_dups_size":0,"ondisk_log_size":6901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47559926,"num_objects":5806,"num_object_clones":0,"num_object_copies":23224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6899,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":772793740,"num_objects":94341,"num_object_clones":0,"num_object_copies":377364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94341,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112746,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":28393472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1457242112,"data_stored":1457242112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112773,"ondisk_log_size":112773,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673416,"kb_used_data":375552,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93698424,"statfs":{"total":96636764160,"available":95947186176,"internally_reserved":0,"allocated":384565248,"data_stored":384398063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,5,4,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.7640000000000002},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.7859999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":6.7119999999999997},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":2.8730000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":7.5350000000000001},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.2709999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677328,"kb_used_data":379972,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93694512,"statfs":{"total":96636764160,"available":95943180288,"internally_reserved":0,"allocated":389091328,"data_stored":388924143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,3,5,0,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":14.281000000000001},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":4.1349999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.456},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":3.7389999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":3.706},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":3.153}]}]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588116,"kb_used_data":290316,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93783724,"statfs":{"total":96636764160,"available":96034533376,"internally_reserved":0,"allocated":297283584,"data_stored":297116399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":7.9379999999999997},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":11.050000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":11.289},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":12.061}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":11.007},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.3609999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677960,"kb_used_data":380140,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93693880,"statfs":{"total":96636764160,"available":95942533120,"internally_reserved":0,"allocated":389263360,"data_stored":389096175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.2130000000000001},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":5.9530000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.2669999999999999},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":1.3300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":1.427}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":388009984,"data_stored":388009984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296972288,"data_stored":296972288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":388345856,"data_stored":388345856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383913984,"data_stored":383913984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:56.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:57.363 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:57.364 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:57.384 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-18T11:48:55.934574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":772793740,"num_objects":94341,"num_object_clones":0,"num_object_copies":377364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94341,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112746,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":28393472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112773,"ondisk_log_size":112773,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2616820,"kb_used_data":1425980,"kb_used_omap":14,"kb_used_meta":1190769,"kb_avail":374870540,"statfs":{"total":386547056640,"available":383867432960,"internally_reserved":0,"allocated":1460203520,"data_stored":1459534780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1219347483},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,3,6,8,14,8,3,0,0,0,0,2,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26419200,"num_objects":-3225,"num_object_clones":0,"num_object_copies":-12900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3225,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004831"},"pg_stats":[{"pgid":"1.f","version":"33'7218","reported_seq":7275,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.729054+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:55.729054+0000","last_peered":"2024-09-18T11:48:55.729054+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:55.729054+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7218,"log_dups_size":0,"ondisk_log_size":7218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49414144,"num_objects":6032,"num_object_clones":0,"num_object_copies":24128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7218,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7058","reported_seq":9878,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:52.119970+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:52.119970+0000","last_peered":"2024-09-18T11:48:52.119970+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:52.119970+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49132790,"num_objects":5998,"num_object_clones":0,"num_object_copies":23992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7056,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1379,"num_bytes_recovered":11264000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5271}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6952","reported_seq":7039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:51.789215+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:51.789215+0000","last_peered":"2024-09-18T11:48:51.789215+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:51.789215+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6952,"log_dups_size":0,"ondisk_log_size":6952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47970963,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6951,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6998","reported_seq":7070,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:52.094566+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:52.094566+0000","last_peered":"2024-09-18T11:48:52.094566+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:52.094566+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48559350,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7043","reported_seq":7108,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.718547+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:55.718547+0000","last_peered":"2024-09-18T11:48:55.718547+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:55.718547+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47936758,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7011","reported_seq":7065,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.862815+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:55.862815+0000","last_peered":"2024-09-18T11:48:55.862815+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:55.862815+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7011,"log_dups_size":0,"ondisk_log_size":7011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47308800,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7011,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7049","reported_seq":7122,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:52.036143+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:52.036143+0000","last_peered":"2024-09-18T11:48:52.036143+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:52.036143+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7049,"log_dups_size":0,"ondisk_log_size":7049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48878838,"num_objects":5967,"num_object_clones":0,"num_object_copies":23868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7047,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3836}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6963","reported_seq":7034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:52.035707+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:52.035707+0000","last_peered":"2024-09-18T11:48:52.035707+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:52.035707+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6963,"log_dups_size":0,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48428278,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7241","reported_seq":7313,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.827909+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:55.827909+0000","last_peered":"2024-09-18T11:48:55.827909+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:55.827909+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48613868,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6816","reported_seq":6904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:51.737100+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:51.737100+0000","last_peered":"2024-09-18T11:48:51.737100+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:51.737100+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47330550,"num_objects":5778,"num_object_clones":0,"num_object_copies":23112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7151","reported_seq":7220,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.868241+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:55.868241+0000","last_peered":"2024-09-18T11:48:55.868241+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:55.868241+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7151,"log_dups_size":0,"ondisk_log_size":7151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48181105,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7148,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7120","reported_seq":7173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.840417+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:55.840417+0000","last_peered":"2024-09-18T11:48:55.840417+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:55.840417+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7120,"log_dups_size":0,"ondisk_log_size":7120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48421499,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7119,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3809}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7020","reported_seq":7098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:51.706288+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:51.706288+0000","last_peered":"2024-09-18T11:48:51.706288+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:51.706288+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7020,"log_dups_size":0,"ondisk_log_size":7020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48716411,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7019,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7210","reported_seq":7291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.836536+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:55.836536+0000","last_peered":"2024-09-18T11:48:55.836536+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:55.836536+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48166134,"num_objects":5880,"num_object_clones":0,"num_object_copies":23520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7208,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3737}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7022","reported_seq":11307,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:50.892408+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:50.892408+0000","last_peered":"2024-09-18T11:48:50.892408+0000","last_clean":"2024-09-18T11:48:50.892408+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:50.892408+0000","last_undegraded":"2024-09-18T11:48:50.892408+0000","last_fullsized":"2024-09-18T11:48:50.892408+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48174326,"num_objects":5881,"num_object_clones":0,"num_object_copies":23524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6901","reported_seq":6977,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:52.036125+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:52.036125+0000","last_peered":"2024-09-18T11:48:52.036125+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:52.036125+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6901,"log_dups_size":0,"ondisk_log_size":6901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47559926,"num_objects":5806,"num_object_clones":0,"num_object_copies":23224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6899,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":772793740,"num_objects":94341,"num_object_clones":0,"num_object_copies":377364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94341,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112746,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":28393472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1457242112,"data_stored":1457242112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112773,"ondisk_log_size":112773,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673416,"kb_used_data":375552,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93698424,"statfs":{"total":96636764160,"available":95947186176,"internally_reserved":0,"allocated":384565248,"data_stored":384398063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,5,4,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.7640000000000002},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.7859999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":6.7119999999999997},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":2.8730000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":7.5350000000000001},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.2709999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677328,"kb_used_data":379972,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93694512,"statfs":{"total":96636764160,"available":95943180288,"internally_reserved":0,"allocated":389091328,"data_stored":388924143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,3,5,0,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":14.281000000000001},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":4.1349999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:47:49 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":6.1829999999999998,"15min":6.0140000000000002},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":20.443000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.456},{"interface":"front","average":{"1min":4.3529999999999998,"5min":3.7639999999999998,"15min":3.6920000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.603999999999999,"5min":14.31,"15min":14.31},"last":3.7389999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":3.706},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":3.153}]}]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588116,"kb_used_data":290316,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93783724,"statfs":{"total":96636764160,"available":96034533376,"internally_reserved":0,"allocated":297283584,"data_stored":297116399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.415,"5min":3.9489999999999998,"15min":3.5089999999999999},"min":{"1min":0.224,"5min":0.159,"15min":0.159},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":7.9379999999999997},{"interface":"front","average":{"1min":5.2190000000000003,"5min":4.7519999999999998,"15min":4.9859999999999998},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":16.056999999999999,"5min":18.221,"15min":18.221},"last":11.050000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":6.694,"5min":4.6539999999999999,"15min":4.1719999999999997},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.223},"max":{"1min":18.716999999999999,"5min":24.600000000000001,"15min":24.600000000000001},"last":11.289},{"interface":"front","average":{"1min":3.6579999999999999,"5min":4.1390000000000002,"15min":3.9249999999999998},"min":{"1min":0.25600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":15.255000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":12.061}]},{"osd":3,"last update":"Wed Sep 18 11:47:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5209999999999999,"5min":4.0460000000000003,"15min":4.1360000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.903,"5min":15.903,"15min":15.903},"last":11.007},{"interface":"front","average":{"1min":6.6040000000000001,"5min":4.9889999999999999,"15min":4.7389999999999999},"min":{"1min":0.33000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":25.440000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.3609999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677960,"kb_used_data":380140,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93693880,"statfs":{"total":96636764160,"available":95942533120,"internally_reserved":0,"allocated":389263360,"data_stored":389096175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:47:52 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":3.903,"15min":3.7549999999999999},"min":{"1min":0.216,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":14.872999999999999,"5min":14.898999999999999,"15min":14.898999999999999},"last":2.2130000000000001},{"interface":"front","average":{"1min":6.1449999999999996,"5min":4.0999999999999996,"15min":3.8740000000000001},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":19.077999999999999,"5min":19.077999999999999,"15min":19.077999999999999},"last":5.9530000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.2669999999999999},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":1.3300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":1.427}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":388009984,"data_stored":388009984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296972288,"data_stored":296972288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":388345856,"data_stored":388345856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383913984,"data_stored":383913984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:48:57.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:48:57.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7218", 'reported_seq': 7275, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:55.729054+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:48:55.729054+0000', 'last_peered': '2024-09-18T11:48:55.729054+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:48:55.729054+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7218, 'log_dups_size': 0, 'ondisk_log_size': 7218, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49414144, 'num_objects': 6032, 'num_object_clones': 0, 'num_object_copies': 24128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7218, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3909}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:57.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:48:57.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7058", 'reported_seq': 9878, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:52.119970+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:52.119970+0000', 'last_peered': '2024-09-18T11:48:52.119970+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:52.119970+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7058, 'log_dups_size': 0, 'ondisk_log_size': 7058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49132790, 'num_objects': 5998, 'num_object_clones': 0, 'num_object_copies': 23992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 727, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5998, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7056, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1379, 'num_bytes_recovered': 11264000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5271}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:57.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:48:57.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6952", 'reported_seq': 7039, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:51.789215+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:48:51.789215+0000', 'last_peered': '2024-09-18T11:48:51.789215+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:48:51.789215+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6952, 'log_dups_size': 0, 'ondisk_log_size': 6952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47970963, 'num_objects': 5857, 'num_object_clones': 0, 'num_object_copies': 23428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5857, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6951, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3804}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:57.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:48:57.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6998", 'reported_seq': 7070, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:52.094566+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:52.094566+0000', 'last_peered': '2024-09-18T11:48:52.094566+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:52.094566+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6998, 'log_dups_size': 0, 'ondisk_log_size': 6998, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48559350, 'num_objects': 5928, 'num_object_clones': 0, 'num_object_copies': 23712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6996, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3818}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:57.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:48:57.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7043", 'reported_seq': 7108, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:55.718547+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:48:55.718547+0000', 'last_peered': '2024-09-18T11:48:55.718547+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:48:55.718547+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7043, 'log_dups_size': 0, 'ondisk_log_size': 7043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47936758, 'num_objects': 5852, 'num_object_clones': 0, 'num_object_copies': 23408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2080, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7041, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3774}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:57.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:48:57.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7011", 'reported_seq': 7065, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:55.862815+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:48:55.862815+0000', 'last_peered': '2024-09-18T11:48:55.862815+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:48:55.862815+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7011, 'log_dups_size': 0, 'ondisk_log_size': 7011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47308800, 'num_objects': 5775, 'num_object_clones': 0, 'num_object_copies': 23100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7011, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3626}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7049", 'reported_seq': 7122, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:52.036143+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:52.036143+0000', 'last_peered': '2024-09-18T11:48:52.036143+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:52.036143+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7049, 'log_dups_size': 0, 'ondisk_log_size': 7049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48878838, 'num_objects': 5967, 'num_object_clones': 0, 'num_object_copies': 23868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7047, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3836}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6963", 'reported_seq': 7034, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:52.035707+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:52.035707+0000', 'last_peered': '2024-09-18T11:48:52.035707+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:52.035707+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6963, 'log_dups_size': 0, 'ondisk_log_size': 6963, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48428278, 'num_objects': 5912, 'num_object_clones': 0, 'num_object_copies': 23648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6961, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7241", 'reported_seq': 7313, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:55.827909+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:48:55.827909+0000', 'last_peered': '2024-09-18T11:48:55.827909+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:48:55.827909+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7241, 'log_dups_size': 0, 'ondisk_log_size': 7241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48613868, 'num_objects': 5935, 'num_object_clones': 0, 'num_object_copies': 23740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7238, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3806}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6816", 'reported_seq': 6904, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:51.737100+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:48:51.737100+0000', 'last_peered': '2024-09-18T11:48:51.737100+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:48:51.737100+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6816, 'log_dups_size': 0, 'ondisk_log_size': 6816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47330550, 'num_objects': 5778, 'num_object_clones': 0, 'num_object_copies': 23112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6814, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3804}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7151", 'reported_seq': 7220, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:55.868241+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:48:55.868241+0000', 'last_peered': '2024-09-18T11:48:55.868241+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:48:55.868241+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7151, 'log_dups_size': 0, 'ondisk_log_size': 7151, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48181105, 'num_objects': 5882, 'num_object_clones': 0, 'num_object_copies': 23528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7148, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3824}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:57.387 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:48:57.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7120", 'reported_seq': 7173, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:55.840417+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:48:55.840417+0000', 'last_peered': '2024-09-18T11:48:55.840417+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:48:55.840417+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7120, 'log_dups_size': 0, 'ondisk_log_size': 7120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48421499, 'num_objects': 5911, 'num_object_clones': 0, 'num_object_copies': 23644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7119, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3809}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:57.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:48:57.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7020", 'reported_seq': 7098, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:51.706288+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:48:51.706288+0000', 'last_peered': '2024-09-18T11:48:51.706288+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:48:51.706288+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7020, 'log_dups_size': 0, 'ondisk_log_size': 7020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48716411, 'num_objects': 5947, 'num_object_clones': 0, 'num_object_copies': 23788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7019, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:48:57.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:48:57.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7210", 'reported_seq': 7291, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:55.836536+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:48:55.836536+0000', 'last_peered': '2024-09-18T11:48:55.836536+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:48:55.836536+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7210, 'log_dups_size': 0, 'ondisk_log_size': 7210, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48166134, 'num_objects': 5880, 'num_object_clones': 0, 'num_object_copies': 23520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7208, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3737}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:48:57.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:48:57.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6901", 'reported_seq': 6977, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:52.036125+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:52.036125+0000', 'last_peered': '2024-09-18T11:48:52.036125+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:52.036125+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6901, 'log_dups_size': 0, 'ondisk_log_size': 6901, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47559926, 'num_objects': 5806, 'num_object_clones': 0, 'num_object_copies': 23224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6899, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3799}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:48:57.388 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:48:58.078 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:58.078 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29892/377364 objects degraded (7.921%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":336,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94341,"data_bytes":772793740,"bytes_used":2679623680,"bytes_avail":383867432960,"bytes_total":386547056640,"degraded_objects":29892,"degraded_total":377364,"degraded_ratio":0.079212643495404966,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":85298,"recovering_keys_per_sec":0,"num_objects_recovered":144,"num_bytes_recovered":1024000,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":268},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:48:58.096 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:48:58.096 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:48:58.652 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:48:58.653 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:48:58.679 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-18T11:48:57.935286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":766780812,"num_objects":93607,"num_object_clones":0,"num_object_copies":374428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93607,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113480,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3631,"num_bytes_recovered":28893184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113507,"ondisk_log_size":113507,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2621440,"kb_used_data":1413956,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":374865920,"statfs":{"total":386547056640,"available":383862702080,"internally_reserved":0,"allocated":1447890944,"data_stored":1447222204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,11,9,14,9,1,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27729920,"num_objects":-3385,"num_object_clones":0,"num_object_copies":-13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3385,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004510"},"pg_stats":[{"pgid":"1.f","version":"33'7218","reported_seq":7275,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.729054+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:48:55.729054+0000","last_peered":"2024-09-18T11:48:55.729054+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:48:55.729054+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7218,"log_dups_size":0,"ondisk_log_size":7218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49414144,"num_objects":6032,"num_object_clones":0,"num_object_copies":24128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7218,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7148","reported_seq":10088,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.178521+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:57.178521+0000","last_peered":"2024-09-18T11:48:57.178521+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:57.178521+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48395510,"num_objects":5908,"num_object_clones":0,"num_object_copies":23632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1440,"num_bytes_recovered":11763712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7024","reported_seq":7115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:56.692841+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:48:56.692841+0000","last_peered":"2024-09-18T11:48:56.692841+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:48:56.692841+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7024,"log_dups_size":0,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47381139,"num_objects":5785,"num_object_clones":0,"num_object_copies":23140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7023,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3734}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7079","reported_seq":7151,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.050355+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:57.050355+0000","last_peered":"2024-09-18T11:48:57.050355+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:57.050355+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47895798,"num_objects":5847,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7043","reported_seq":7108,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.718547+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:48:55.718547+0000","last_peered":"2024-09-18T11:48:55.718547+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:48:55.718547+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47936758,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7011","reported_seq":7065,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.862815+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:48:55.862815+0000","last_peered":"2024-09-18T11:48:55.862815+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:48:55.862815+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7011,"log_dups_size":0,"ondisk_log_size":7011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47308800,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7011,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7131","reported_seq":7208,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.201932+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:57.201932+0000","last_peered":"2024-09-18T11:48:57.201932+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:57.201932+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7131,"log_dups_size":0,"ondisk_log_size":7131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207094,"num_objects":5885,"num_object_clones":0,"num_object_copies":23540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3756}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7043","reported_seq":7115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.186346+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:57.186346+0000","last_peered":"2024-09-18T11:48:57.186346+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:57.186346+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772918,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7241","reported_seq":7313,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.827909+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:48:55.827909+0000","last_peered":"2024-09-18T11:48:55.827909+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:48:55.827909+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48613868,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6901","reported_seq":6989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:56.782450+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:48:56.782450+0000","last_peered":"2024-09-18T11:48:56.782450+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:48:56.782450+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6901,"log_dups_size":0,"ondisk_log_size":6901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46634230,"num_objects":5693,"num_object_clones":0,"num_object_copies":22772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6899,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3719}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7151","reported_seq":7220,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.868241+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:48:55.868241+0000","last_peered":"2024-09-18T11:48:55.868241+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:48:55.868241+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7151,"log_dups_size":0,"ondisk_log_size":7151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48181105,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7148,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7120","reported_seq":7173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.840417+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:48:55.840417+0000","last_peered":"2024-09-18T11:48:55.840417+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:48:55.840417+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7120,"log_dups_size":0,"ondisk_log_size":7120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48421499,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7119,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3809}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7094","reported_seq":7174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:56.713471+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:48:56.713471+0000","last_peered":"2024-09-18T11:48:56.713471+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:48:56.713471+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7094,"log_dups_size":0,"ondisk_log_size":7094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48110203,"num_objects":5873,"num_object_clones":0,"num_object_copies":23492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7093,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7210","reported_seq":7291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:55.836536+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:48:55.836536+0000","last_peered":"2024-09-18T11:48:55.836536+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:48:55.836536+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48166134,"num_objects":5880,"num_object_clones":0,"num_object_copies":23520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7208,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3737}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7106","reported_seq":11391,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:48:55.852633+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:48:55.852633+0000","last_peered":"2024-09-18T11:48:55.852633+0000","last_clean":"2024-09-18T11:48:55.852633+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:48:55.852633+0000","last_undegraded":"2024-09-18T11:48:55.852633+0000","last_fullsized":"2024-09-18T11:48:55.852633+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7106,"log_dups_size":0,"ondisk_log_size":7106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47486198,"num_objects":5797,"num_object_clones":0,"num_object_copies":23188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7104,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6987","reported_seq":7063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.176305+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:57.176305+0000","last_peered":"2024-09-18T11:48:57.176305+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:57.176305+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6987,"log_dups_size":0,"ondisk_log_size":6987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855414,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":766780812,"num_objects":93607,"num_object_clones":0,"num_object_copies":374428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93607,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113480,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3631,"num_bytes_recovered":28893184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1445937152,"data_stored":1445937152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113507,"ondisk_log_size":113507,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673416,"kb_used_data":375552,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93698424,"statfs":{"total":96636764160,"available":95947186176,"internally_reserved":0,"allocated":384565248,"data_stored":384398063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,5,4,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.7640000000000002},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":6.7859999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":6.7119999999999997},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":2.8730000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":7.5350000000000001},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.2709999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680324,"kb_used_data":374624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93691516,"statfs":{"total":96636764160,"available":95940112384,"internally_reserved":0,"allocated":383614976,"data_stored":383447791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,3,4,1,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":2.145},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":17.285}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.496},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":17.350000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":3.8029999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":2.0910000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591816,"kb_used_data":289856,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93780024,"statfs":{"total":96636764160,"available":96030744576,"internally_reserved":0,"allocated":296812544,"data_stored":296645359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.954},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":1.9990000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":2.6059999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.573}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":2.0339999999999998},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.6080000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675884,"kb_used_data":373924,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93695956,"statfs":{"total":96636764160,"available":95944658944,"internally_reserved":0,"allocated":382898176,"data_stored":382730991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":10.727},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":1.304}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":23.48},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":1.331}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":10.766},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":2.2200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":382599168,"data_stored":382599168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296476672,"data_stored":296476672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382947328,"data_stored":382947328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383913984,"data_stored":383913984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:01.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:02.287 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:02.288 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:02.316 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-18T11:49:01.936896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":759514508,"num_objects":92720,"num_object_clones":0,"num_object_copies":370880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29809,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114367,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3647,"num_bytes_recovered":28893184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114394,"ondisk_log_size":114394,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2623480,"kb_used_data":1403468,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374863880,"statfs":{"total":386547056640,"available":383860613120,"internally_reserved":0,"allocated":1437151232,"data_stored":1436482492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,14,9,11,5,0,0,3,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29032448,"num_objects":-3544,"num_object_clones":0,"num_object_copies":-14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3544,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004506"},"pg_stats":[{"pgid":"1.f","version":"33'7315","reported_seq":7374,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.883579+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:00.883579+0000","last_peered":"2024-09-18T11:49:00.883579+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:00.883579+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48619520,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7148","reported_seq":10088,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.178521+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:57.178521+0000","last_peered":"2024-09-18T11:48:57.178521+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:57.178521+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48395510,"num_objects":5908,"num_object_clones":0,"num_object_copies":23632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1440,"num_bytes_recovered":11763712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7101","reported_seq":7192,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.622445+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:01.622445+0000","last_peered":"2024-09-18T11:49:01.622445+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:01.622445+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7101,"log_dups_size":0,"ondisk_log_size":7101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46750355,"num_objects":5708,"num_object_clones":0,"num_object_copies":22832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7100,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7079","reported_seq":7151,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.050355+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:57.050355+0000","last_peered":"2024-09-18T11:48:57.050355+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:57.050355+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47895798,"num_objects":5847,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7125","reported_seq":7198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.781658+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:00.781658+0000","last_peered":"2024-09-18T11:49:00.781658+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:00.781658+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47265014,"num_objects":5770,"num_object_clones":0,"num_object_copies":23080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7123,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7101","reported_seq":7160,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.765641+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:00.765641+0000","last_peered":"2024-09-18T11:49:00.765641+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:00.765641+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7101,"log_dups_size":0,"ondisk_log_size":7101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46571520,"num_objects":5685,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3538}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7131","reported_seq":7208,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.201932+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:57.201932+0000","last_peered":"2024-09-18T11:48:57.201932+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:57.201932+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7131,"log_dups_size":0,"ondisk_log_size":7131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207094,"num_objects":5885,"num_object_clones":0,"num_object_copies":23540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3756}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7043","reported_seq":7115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.186346+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:57.186346+0000","last_peered":"2024-09-18T11:48:57.186346+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:57.186346+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772918,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7321","reported_seq":7397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.922751+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:00.922751+0000","last_peered":"2024-09-18T11:49:00.922751+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:00.922751+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7321,"log_dups_size":0,"ondisk_log_size":7321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47958508,"num_objects":5855,"num_object_clones":0,"num_object_copies":23420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7318,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6980","reported_seq":7068,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.717105+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:01.717105+0000","last_peered":"2024-09-18T11:49:01.717105+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:01.717105+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45987062,"num_objects":5614,"num_object_clones":0,"num_object_copies":22456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6978,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7224","reported_seq":7301,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.867271+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:00.867271+0000","last_peered":"2024-09-18T11:49:00.867271+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:00.867271+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7224,"log_dups_size":0,"ondisk_log_size":7224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47583089,"num_objects":5809,"num_object_clones":0,"num_object_copies":23236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7211","reported_seq":7266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.930251+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:00.930251+0000","last_peered":"2024-09-18T11:49:00.930251+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:00.930251+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47676027,"num_objects":5820,"num_object_clones":0,"num_object_copies":23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7210,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3719}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7165","reported_seq":7245,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.826706+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:01.826706+0000","last_peered":"2024-09-18T11:49:01.826706+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:01.826706+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7165,"log_dups_size":0,"ondisk_log_size":7165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47528571,"num_objects":5802,"num_object_clones":0,"num_object_copies":23208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7280","reported_seq":7365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.510074+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:00.510074+0000","last_peered":"2024-09-18T11:49:00.510074+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:00.510074+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7280,"log_dups_size":0,"ondisk_log_size":7280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47592694,"num_objects":5810,"num_object_clones":0,"num_object_copies":23240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7278,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3669}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7183","reported_seq":11468,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:00.993810+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:00.993810+0000","last_peered":"2024-09-18T11:49:00.993810+0000","last_clean":"2024-09-18T11:49:00.993810+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:00.993810+0000","last_undegraded":"2024-09-18T11:49:00.993810+0000","last_fullsized":"2024-09-18T11:49:00.993810+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855414,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6987","reported_seq":7063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.176305+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:57.176305+0000","last_peered":"2024-09-18T11:48:57.176305+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:57.176305+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6987,"log_dups_size":0,"ondisk_log_size":6987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855414,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":759514508,"num_objects":92720,"num_object_clones":0,"num_object_copies":370880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29809,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114367,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3647,"num_bytes_recovered":28893184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1435164672,"data_stored":1435164672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114394,"ondisk_log_size":114394,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672580,"kb_used_data":370516,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93699260,"statfs":{"total":96636764160,"available":95948042240,"internally_reserved":0,"allocated":379408384,"data_stored":379241199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,5,3,5,1,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":18.751999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":18.774000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":18.718},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":8.8970000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":3.4769999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.8450000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679380,"kb_used_data":369516,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93692460,"statfs":{"total":96636764160,"available":95941079040,"internally_reserved":0,"allocated":378384384,"data_stored":378217199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":5.7039999999999997},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.718},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.49299999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595636,"kb_used_data":289512,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93776204,"statfs":{"total":96636764160,"available":96026832896,"internally_reserved":0,"allocated":296460288,"data_stored":296293103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":9.0579999999999998},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":11.096}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.1020000000000003},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.0970000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":11.057},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.1369999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675884,"kb_used_data":373924,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93695956,"statfs":{"total":96636764160,"available":95944658944,"internally_reserved":0,"allocated":382898176,"data_stored":382730991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":10.727},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":1.304}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":23.48},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":1.331}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":10.766},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":2.2200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":382599168,"data_stored":382599168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296153088,"data_stored":296153088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377708544,"data_stored":377708544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":378703872,"data_stored":378703872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:02.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:02.853 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:02.854 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:02.874 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-18T11:49:01.936896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":759514508,"num_objects":92720,"num_object_clones":0,"num_object_copies":370880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29809,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114367,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3647,"num_bytes_recovered":28893184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114394,"ondisk_log_size":114394,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2623480,"kb_used_data":1403468,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":374863880,"statfs":{"total":386547056640,"available":383860613120,"internally_reserved":0,"allocated":1437151232,"data_stored":1436482492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,14,9,11,5,0,0,3,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29032448,"num_objects":-3544,"num_object_clones":0,"num_object_copies":-14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3544,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004506"},"pg_stats":[{"pgid":"1.f","version":"33'7315","reported_seq":7374,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.883579+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:00.883579+0000","last_peered":"2024-09-18T11:49:00.883579+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:00.883579+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48619520,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7148","reported_seq":10088,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.178521+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:48:57.178521+0000","last_peered":"2024-09-18T11:48:57.178521+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:48:57.178521+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48395510,"num_objects":5908,"num_object_clones":0,"num_object_copies":23632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1440,"num_bytes_recovered":11763712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7101","reported_seq":7192,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.622445+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:01.622445+0000","last_peered":"2024-09-18T11:49:01.622445+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:01.622445+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7101,"log_dups_size":0,"ondisk_log_size":7101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46750355,"num_objects":5708,"num_object_clones":0,"num_object_copies":22832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7100,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7079","reported_seq":7151,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.050355+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:48:57.050355+0000","last_peered":"2024-09-18T11:48:57.050355+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:48:57.050355+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47895798,"num_objects":5847,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7125","reported_seq":7198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.781658+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:00.781658+0000","last_peered":"2024-09-18T11:49:00.781658+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:00.781658+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47265014,"num_objects":5770,"num_object_clones":0,"num_object_copies":23080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7123,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7101","reported_seq":7160,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.765641+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:00.765641+0000","last_peered":"2024-09-18T11:49:00.765641+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:00.765641+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7101,"log_dups_size":0,"ondisk_log_size":7101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46571520,"num_objects":5685,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3538}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7131","reported_seq":7208,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.201932+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:48:57.201932+0000","last_peered":"2024-09-18T11:48:57.201932+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:48:57.201932+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7131,"log_dups_size":0,"ondisk_log_size":7131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207094,"num_objects":5885,"num_object_clones":0,"num_object_copies":23540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3756}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7043","reported_seq":7115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.186346+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:48:57.186346+0000","last_peered":"2024-09-18T11:48:57.186346+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:48:57.186346+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772918,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7321","reported_seq":7397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.922751+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:00.922751+0000","last_peered":"2024-09-18T11:49:00.922751+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:00.922751+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7321,"log_dups_size":0,"ondisk_log_size":7321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47958508,"num_objects":5855,"num_object_clones":0,"num_object_copies":23420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7318,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6980","reported_seq":7068,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.717105+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:01.717105+0000","last_peered":"2024-09-18T11:49:01.717105+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:01.717105+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45987062,"num_objects":5614,"num_object_clones":0,"num_object_copies":22456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6978,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7224","reported_seq":7301,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.867271+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:00.867271+0000","last_peered":"2024-09-18T11:49:00.867271+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:00.867271+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7224,"log_dups_size":0,"ondisk_log_size":7224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47583089,"num_objects":5809,"num_object_clones":0,"num_object_copies":23236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7211","reported_seq":7266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.930251+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:00.930251+0000","last_peered":"2024-09-18T11:49:00.930251+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:00.930251+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47676027,"num_objects":5820,"num_object_clones":0,"num_object_copies":23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7210,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3719}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7165","reported_seq":7245,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.826706+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:01.826706+0000","last_peered":"2024-09-18T11:49:01.826706+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:01.826706+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7165,"log_dups_size":0,"ondisk_log_size":7165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47528571,"num_objects":5802,"num_object_clones":0,"num_object_copies":23208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7280","reported_seq":7365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.510074+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:00.510074+0000","last_peered":"2024-09-18T11:49:00.510074+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:00.510074+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7280,"log_dups_size":0,"ondisk_log_size":7280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47592694,"num_objects":5810,"num_object_clones":0,"num_object_copies":23240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7278,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3669}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7183","reported_seq":11468,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:00.993810+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:00.993810+0000","last_peered":"2024-09-18T11:49:00.993810+0000","last_clean":"2024-09-18T11:49:00.993810+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:00.993810+0000","last_undegraded":"2024-09-18T11:49:00.993810+0000","last_fullsized":"2024-09-18T11:49:00.993810+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855414,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'6987","reported_seq":7063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:48:57.176305+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:48:57.176305+0000","last_peered":"2024-09-18T11:48:57.176305+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:48:57.176305+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6987,"log_dups_size":0,"ondisk_log_size":6987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855414,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":759514508,"num_objects":92720,"num_object_clones":0,"num_object_copies":370880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29809,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114367,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3647,"num_bytes_recovered":28893184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1435164672,"data_stored":1435164672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114394,"ondisk_log_size":114394,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672580,"kb_used_data":370516,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93699260,"statfs":{"total":96636764160,"available":95948042240,"internally_reserved":0,"allocated":379408384,"data_stored":379241199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,5,3,5,1,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":18.751999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":18.774000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":18.718},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":8.8970000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":3.4769999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.8450000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679380,"kb_used_data":369516,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93692460,"statfs":{"total":96636764160,"available":95941079040,"internally_reserved":0,"allocated":378384384,"data_stored":378217199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":5.7039999999999997},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.718},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.49299999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595636,"kb_used_data":289512,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93776204,"statfs":{"total":96636764160,"available":96026832896,"internally_reserved":0,"allocated":296460288,"data_stored":296293103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":9.0579999999999998},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":11.096}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.1020000000000003},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.0970000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":11.057},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.1369999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675884,"kb_used_data":373924,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93695956,"statfs":{"total":96636764160,"available":95944658944,"internally_reserved":0,"allocated":382898176,"data_stored":382730991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":10.727},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":1.304}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":23.48},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":1.331}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":10.766},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":2.2200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":382599168,"data_stored":382599168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296153088,"data_stored":296153088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377708544,"data_stored":377708544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":378703872,"data_stored":378703872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:02.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:02.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7315", 'reported_seq': 7374, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:00.883579+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:00.883579+0000', 'last_peered': '2024-09-18T11:49:00.883579+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:00.883579+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7315, 'log_dups_size': 0, 'ondisk_log_size': 7315, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48619520, 'num_objects': 5935, 'num_object_clones': 0, 'num_object_copies': 23740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2122, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7315, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3813}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:02.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:49:02.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7148", 'reported_seq': 10088, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:57.178521+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:48:57.178521+0000', 'last_peered': '2024-09-18T11:48:57.178521+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:48:57.178521+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7148, 'log_dups_size': 0, 'ondisk_log_size': 7148, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48395510, 'num_objects': 5908, 'num_object_clones': 0, 'num_object_copies': 23632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 666, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7146, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1440, 'num_bytes_recovered': 11763712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5242}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:02.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:02.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7101", 'reported_seq': 7192, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:01.622445+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:01.622445+0000', 'last_peered': '2024-09-18T11:49:01.622445+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:01.622445+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7101, 'log_dups_size': 0, 'ondisk_log_size': 7101, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46750355, 'num_objects': 5708, 'num_object_clones': 0, 'num_object_copies': 22832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2052, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5708, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7100, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3657}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:02.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:02.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7079", 'reported_seq': 7151, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:57.050355+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:48:57.050355+0000', 'last_peered': '2024-09-18T11:48:57.050355+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:48:57.050355+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7079, 'log_dups_size': 0, 'ondisk_log_size': 7079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47895798, 'num_objects': 5847, 'num_object_clones': 0, 'num_object_copies': 23388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7077, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3737}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:02.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:02.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7125", 'reported_seq': 7198, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:00.781658+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:00.781658+0000', 'last_peered': '2024-09-18T11:49:00.781658+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:00.781658+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7125, 'log_dups_size': 0, 'ondisk_log_size': 7125, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47265014, 'num_objects': 5770, 'num_object_clones': 0, 'num_object_copies': 23080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7123, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3696}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:02.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:02.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7101", 'reported_seq': 7160, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:00.765641+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:00.765641+0000', 'last_peered': '2024-09-18T11:49:00.765641+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:00.765641+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7101, 'log_dups_size': 0, 'ondisk_log_size': 7101, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46571520, 'num_objects': 5685, 'num_object_clones': 0, 'num_object_copies': 22740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7101, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3538}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:02.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:02.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7131", 'reported_seq': 7208, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:57.201932+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:48:57.201932+0000', 'last_peered': '2024-09-18T11:48:57.201932+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:48:57.201932+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7131, 'log_dups_size': 0, 'ondisk_log_size': 7131, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48207094, 'num_objects': 5885, 'num_object_clones': 0, 'num_object_copies': 23540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2131, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7129, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3756}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7043", 'reported_seq': 7115, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:57.186346+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:48:57.186346+0000', 'last_peered': '2024-09-18T11:48:57.186346+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:48:57.186346+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7043, 'log_dups_size': 0, 'ondisk_log_size': 7043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47772918, 'num_objects': 5832, 'num_object_clones': 0, 'num_object_copies': 23328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2032, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7041, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3802}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7321", 'reported_seq': 7397, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:00.922751+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:49:00.922751+0000', 'last_peered': '2024-09-18T11:49:00.922751+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:00.922751+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7321, 'log_dups_size': 0, 'ondisk_log_size': 7321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47958508, 'num_objects': 5855, 'num_object_clones': 0, 'num_object_copies': 23420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7318, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3728}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6980", 'reported_seq': 7068, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:01.717105+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:01.717105+0000', 'last_peered': '2024-09-18T11:49:01.717105+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:01.717105+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6980, 'log_dups_size': 0, 'ondisk_log_size': 6980, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45987062, 'num_objects': 5614, 'num_object_clones': 0, 'num_object_copies': 22456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6978, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3640}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7224", 'reported_seq': 7301, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:00.867271+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:00.867271+0000', 'last_peered': '2024-09-18T11:49:00.867271+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:00.867271+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7224, 'log_dups_size': 0, 'ondisk_log_size': 7224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47583089, 'num_objects': 5809, 'num_object_clones': 0, 'num_object_copies': 23236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2057, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7221, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3755}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7211", 'reported_seq': 7266, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:00.930251+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:00.930251+0000', 'last_peered': '2024-09-18T11:49:00.930251+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:00.930251+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7211, 'log_dups_size': 0, 'ondisk_log_size': 7211, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47676027, 'num_objects': 5820, 'num_object_clones': 0, 'num_object_copies': 23280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2102, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7210, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3719}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:02.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:02.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7165", 'reported_seq': 7245, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:01.826706+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:01.826706+0000', 'last_peered': '2024-09-18T11:49:01.826706+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:01.826706+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7165, 'log_dups_size': 0, 'ondisk_log_size': 7165, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47528571, 'num_objects': 5802, 'num_object_clones': 0, 'num_object_copies': 23208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7164, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:02.878 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:02.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7280", 'reported_seq': 7365, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:00.510074+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:00.510074+0000', 'last_peered': '2024-09-18T11:49:00.510074+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:00.510074+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7280, 'log_dups_size': 0, 'ondisk_log_size': 7280, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47592694, 'num_objects': 5810, 'num_object_clones': 0, 'num_object_copies': 23240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7278, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3669}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:02.878 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:02.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6987", 'reported_seq': 7063, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:48:57.176305+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:48:57.176305+0000', 'last_peered': '2024-09-18T11:48:57.176305+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:48:57.176305+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 6987, 'log_dups_size': 0, 'ondisk_log_size': 6987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46855414, 'num_objects': 5720, 'num_object_clones': 0, 'num_object_copies': 22880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6985, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3713}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:02.878 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:03.524 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:03.524 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29809/370880 objects degraded (8.037%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":341,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":92720,"data_bytes":759514508,"bytes_used":2686443520,"bytes_avail":383860613120,"bytes_total":386547056640,"degraded_objects":29809,"degraded_total":370880,"degraded_ratio":0.080373705780845558,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":82571,"recovering_keys_per_sec":0,"num_objects_recovered":146,"num_bytes_recovered":991232,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":295},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:03.545 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:03.546 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:04.083 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:04.084 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:04.106 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-18T11:49:03.937753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":756327820,"num_objects":92331,"num_object_clones":0,"num_object_copies":369324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114756,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3720,"num_bytes_recovered":29491200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114783,"ondisk_log_size":114783,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2622524,"kb_used_data":1398344,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374864836,"statfs":{"total":386547056640,"available":383861592064,"internally_reserved":0,"allocated":1431904256,"data_stored":1431235516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,8,11,10,4,0,0,3,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24625152,"num_objects":-3006,"num_object_clones":0,"num_object_copies":-12024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3006,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":1589248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004763"},"pg_stats":[{"pgid":"1.f","version":"33'7315","reported_seq":7374,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.883579+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:00.883579+0000","last_peered":"2024-09-18T11:49:00.883579+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:00.883579+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48619520,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7224","reported_seq":10312,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:02.174138+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:02.174138+0000","last_peered":"2024-09-18T11:49:02.174138+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:02.174138+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7224,"log_dups_size":0,"ondisk_log_size":7224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772918,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1513,"num_bytes_recovered":12361728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7101","reported_seq":7192,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.622445+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:01.622445+0000","last_peered":"2024-09-18T11:49:01.622445+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:01.622445+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7101,"log_dups_size":0,"ondisk_log_size":7101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46750355,"num_objects":5708,"num_object_clones":0,"num_object_copies":22832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7100,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7151","reported_seq":7222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:02.093197+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:02.093197+0000","last_peered":"2024-09-18T11:49:02.093197+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:02.093197+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7151,"log_dups_size":0,"ondisk_log_size":7151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47305974,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7149,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7125","reported_seq":7198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.781658+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:00.781658+0000","last_peered":"2024-09-18T11:49:00.781658+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:00.781658+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47265014,"num_objects":5770,"num_object_clones":0,"num_object_copies":23080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7123,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7101","reported_seq":7160,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.765641+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:00.765641+0000","last_peered":"2024-09-18T11:49:00.765641+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:00.765641+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7101,"log_dups_size":0,"ondisk_log_size":7101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46571520,"num_objects":5685,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3538}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7228","reported_seq":7305,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:02.095707+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:02.095707+0000","last_peered":"2024-09-18T11:49:02.095707+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:02.095707+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7228,"log_dups_size":0,"ondisk_log_size":7228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47412470,"num_objects":5788,"num_object_clones":0,"num_object_copies":23152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7226,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7111","reported_seq":7183,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:02.115312+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:02.115312+0000","last_peered":"2024-09-18T11:49:02.115312+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:02.115312+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7111,"log_dups_size":0,"ondisk_log_size":7111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215862,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7109,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3734}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7321","reported_seq":7397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.922751+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:00.922751+0000","last_peered":"2024-09-18T11:49:00.922751+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:00.922751+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7321,"log_dups_size":0,"ondisk_log_size":7321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47958508,"num_objects":5855,"num_object_clones":0,"num_object_copies":23420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7318,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6980","reported_seq":7068,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.717105+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:01.717105+0000","last_peered":"2024-09-18T11:49:01.717105+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:01.717105+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45987062,"num_objects":5614,"num_object_clones":0,"num_object_copies":22456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6978,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7224","reported_seq":7301,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.867271+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:00.867271+0000","last_peered":"2024-09-18T11:49:00.867271+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:00.867271+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7224,"log_dups_size":0,"ondisk_log_size":7224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47583089,"num_objects":5809,"num_object_clones":0,"num_object_copies":23236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7211","reported_seq":7266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.930251+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:00.930251+0000","last_peered":"2024-09-18T11:49:00.930251+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:00.930251+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47676027,"num_objects":5820,"num_object_clones":0,"num_object_copies":23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7210,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3719}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7165","reported_seq":7245,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.826706+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:01.826706+0000","last_peered":"2024-09-18T11:49:01.826706+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:01.826706+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7165,"log_dups_size":0,"ondisk_log_size":7165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47528571,"num_objects":5802,"num_object_clones":0,"num_object_copies":23208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7280","reported_seq":7365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.510074+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:00.510074+0000","last_peered":"2024-09-18T11:49:00.510074+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:00.510074+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7280,"log_dups_size":0,"ondisk_log_size":7280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47592694,"num_objects":5810,"num_object_clones":0,"num_object_copies":23240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7278,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3669}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7183","reported_seq":11468,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:00.993810+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:00.993810+0000","last_peered":"2024-09-18T11:49:00.993810+0000","last_clean":"2024-09-18T11:49:00.993810+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:00.993810+0000","last_undegraded":"2024-09-18T11:49:00.993810+0000","last_fullsized":"2024-09-18T11:49:00.993810+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855414,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7063","reported_seq":7138,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:02.093591+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:02.093591+0000","last_peered":"2024-09-18T11:49:02.093591+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:02.093591+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7063,"log_dups_size":0,"ondisk_log_size":7063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232822,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7061,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":756327820,"num_objects":92331,"num_object_clones":0,"num_object_copies":369324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114756,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3720,"num_bytes_recovered":29491200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1429917696,"data_stored":1429917696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114783,"ondisk_log_size":114783,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672580,"kb_used_data":370516,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93699260,"statfs":{"total":96636764160,"available":95948042240,"internally_reserved":0,"allocated":379408384,"data_stored":379241199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,5,3,5,1,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":18.751999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":18.774000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":18.718},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":8.8970000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":3.4769999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.8450000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679380,"kb_used_data":369516,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93692460,"statfs":{"total":96636764160,"available":95941079040,"internally_reserved":0,"allocated":378384384,"data_stored":378217199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":5.7039999999999997},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.718},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.49299999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595636,"kb_used_data":289512,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93776204,"statfs":{"total":96636764160,"available":96026832896,"internally_reserved":0,"allocated":296460288,"data_stored":296293103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":9.0579999999999998},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":11.096}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.1020000000000003},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.0970000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":11.057},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.1369999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674928,"kb_used_data":368800,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93696912,"statfs":{"total":96636764160,"available":95945637888,"internally_reserved":0,"allocated":377651200,"data_stored":377484015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":2.9630000000000001},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.4569999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.722},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.5199999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":5.6440000000000001},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":4.492}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":377352192,"data_stored":377352192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296153088,"data_stored":296153088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377708544,"data_stored":377708544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":378703872,"data_stored":378703872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:07.108 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:07.705 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:07.705 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:07.731 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-18T11:49:05.940062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":756327820,"num_objects":92331,"num_object_clones":0,"num_object_copies":369324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114756,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3720,"num_bytes_recovered":29491200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114783,"ondisk_log_size":114783,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2622524,"kb_used_data":1398344,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374864836,"statfs":{"total":386547056640,"available":383861592064,"internally_reserved":0,"allocated":1431904256,"data_stored":1431235516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,8,11,10,4,0,0,3,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21209088,"num_objects":-2589,"num_object_clones":0,"num_object_copies":-10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2589,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006420"},"pg_stats":[{"pgid":"1.f","version":"33'7315","reported_seq":7374,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.883579+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:00.883579+0000","last_peered":"2024-09-18T11:49:00.883579+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:00.883579+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48619520,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7224","reported_seq":10312,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:02.174138+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:02.174138+0000","last_peered":"2024-09-18T11:49:02.174138+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:02.174138+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7224,"log_dups_size":0,"ondisk_log_size":7224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772918,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1513,"num_bytes_recovered":12361728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7101","reported_seq":7192,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.622445+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:01.622445+0000","last_peered":"2024-09-18T11:49:01.622445+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:01.622445+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7101,"log_dups_size":0,"ondisk_log_size":7101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46750355,"num_objects":5708,"num_object_clones":0,"num_object_copies":22832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7100,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7151","reported_seq":7222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:02.093197+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:02.093197+0000","last_peered":"2024-09-18T11:49:02.093197+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:02.093197+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7151,"log_dups_size":0,"ondisk_log_size":7151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47305974,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7149,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7125","reported_seq":7198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.781658+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:00.781658+0000","last_peered":"2024-09-18T11:49:00.781658+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:00.781658+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47265014,"num_objects":5770,"num_object_clones":0,"num_object_copies":23080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7123,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7101","reported_seq":7160,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.765641+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:00.765641+0000","last_peered":"2024-09-18T11:49:00.765641+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:00.765641+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7101,"log_dups_size":0,"ondisk_log_size":7101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46571520,"num_objects":5685,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3538}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7228","reported_seq":7305,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:02.095707+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:02.095707+0000","last_peered":"2024-09-18T11:49:02.095707+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:02.095707+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7228,"log_dups_size":0,"ondisk_log_size":7228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47412470,"num_objects":5788,"num_object_clones":0,"num_object_copies":23152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7226,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7111","reported_seq":7183,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:02.115312+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:02.115312+0000","last_peered":"2024-09-18T11:49:02.115312+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:02.115312+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7111,"log_dups_size":0,"ondisk_log_size":7111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215862,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7109,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3734}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7321","reported_seq":7397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.922751+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:00.922751+0000","last_peered":"2024-09-18T11:49:00.922751+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:00.922751+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7321,"log_dups_size":0,"ondisk_log_size":7321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47958508,"num_objects":5855,"num_object_clones":0,"num_object_copies":23420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7318,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6980","reported_seq":7068,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.717105+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:01.717105+0000","last_peered":"2024-09-18T11:49:01.717105+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:01.717105+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45987062,"num_objects":5614,"num_object_clones":0,"num_object_copies":22456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6978,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7224","reported_seq":7301,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.867271+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:00.867271+0000","last_peered":"2024-09-18T11:49:00.867271+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:00.867271+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7224,"log_dups_size":0,"ondisk_log_size":7224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47583089,"num_objects":5809,"num_object_clones":0,"num_object_copies":23236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7211","reported_seq":7266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.930251+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:00.930251+0000","last_peered":"2024-09-18T11:49:00.930251+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:00.930251+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47676027,"num_objects":5820,"num_object_clones":0,"num_object_copies":23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7210,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3719}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7165","reported_seq":7245,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:01.826706+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:01.826706+0000","last_peered":"2024-09-18T11:49:01.826706+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:01.826706+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7165,"log_dups_size":0,"ondisk_log_size":7165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47528571,"num_objects":5802,"num_object_clones":0,"num_object_copies":23208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7280","reported_seq":7365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:00.510074+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:00.510074+0000","last_peered":"2024-09-18T11:49:00.510074+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:00.510074+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7280,"log_dups_size":0,"ondisk_log_size":7280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47592694,"num_objects":5810,"num_object_clones":0,"num_object_copies":23240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7278,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3669}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7183","reported_seq":11468,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:00.993810+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:00.993810+0000","last_peered":"2024-09-18T11:49:00.993810+0000","last_clean":"2024-09-18T11:49:00.993810+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:00.993810+0000","last_undegraded":"2024-09-18T11:49:00.993810+0000","last_fullsized":"2024-09-18T11:49:00.993810+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855414,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7063","reported_seq":7138,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:02.093591+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:02.093591+0000","last_peered":"2024-09-18T11:49:02.093591+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:02.093591+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7063,"log_dups_size":0,"ondisk_log_size":7063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232822,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7061,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":756327820,"num_objects":92331,"num_object_clones":0,"num_object_copies":369324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114756,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3720,"num_bytes_recovered":29491200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1429917696,"data_stored":1429917696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114783,"ondisk_log_size":114783,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672580,"kb_used_data":370516,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93699260,"statfs":{"total":96636764160,"available":95948042240,"internally_reserved":0,"allocated":379408384,"data_stored":379241199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,5,3,5,1,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":18.751999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":18.774000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":18.718},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":8.8970000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":3.4769999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.8450000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679380,"kb_used_data":369516,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93692460,"statfs":{"total":96636764160,"available":95941079040,"internally_reserved":0,"allocated":378384384,"data_stored":378217199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:47:58 2024","interfaces":[{"interface":"back","average":{"1min":6.117,"5min":3.6669999999999998,"15min":2.2400000000000002},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.923999999999999,"5min":19.923999999999999,"15min":19.923999999999999},"last":5.7039999999999997},{"interface":"front","average":{"1min":2.734,"5min":2.5600000000000001,"15min":1.7949999999999999},"min":{"1min":0.29899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.0519999999999996,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.718},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.49299999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595636,"kb_used_data":289512,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93776204,"statfs":{"total":96636764160,"available":96026832896,"internally_reserved":0,"allocated":296460288,"data_stored":296293103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":9.0579999999999998},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":11.096}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":4.1020000000000003},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.0970000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":11.057},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.1369999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674928,"kb_used_data":368800,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93696912,"statfs":{"total":96636764160,"available":95945637888,"internally_reserved":0,"allocated":377651200,"data_stored":377484015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":2.9630000000000001},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.4569999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:04 2024","interfaces":[{"interface":"back","average":{"1min":7.2910000000000004,"5min":3.6720000000000002,"15min":1.7849999999999999},"min":{"1min":0.26300000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":32.795000000000002,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.722},{"interface":"front","average":{"1min":4.8330000000000002,"5min":3.2229999999999999,"15min":1.865},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.789,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.5199999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":5.6440000000000001},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":4.492}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":377352192,"data_stored":377352192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296153088,"data_stored":296153088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377708544,"data_stored":377708544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":378703872,"data_stored":378703872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:07.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:08.314 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:08.314 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:08.333 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-18T11:49:07.940630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":745809292,"num_objects":91047,"num_object_clones":0,"num_object_copies":364188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116040,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3810,"num_bytes_recovered":30064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116067,"ondisk_log_size":116067,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2607172,"kb_used_data":1383052,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374880188,"statfs":{"total":386547056640,"available":383877312512,"internally_reserved":0,"allocated":1416245248,"data_stored":1415576508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,9,8,9,13,3,7,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26984448,"num_objects":-3294,"num_object_clones":0,"num_object_copies":-13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3294,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006055"},"pg_stats":[{"pgid":"1.f","version":"33'7396","reported_seq":7456,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.912685+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:05.912685+0000","last_peered":"2024-09-18T11:49:05.912685+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:05.912685+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7396,"log_dups_size":0,"ondisk_log_size":7396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47955968,"num_objects":5854,"num_object_clones":0,"num_object_copies":23416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7306","reported_seq":10534,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.238909+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:07.238909+0000","last_peered":"2024-09-18T11:49:07.238909+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:07.238909+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47101174,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1583,"num_bytes_recovered":12935168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7176","reported_seq":7271,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:06.762797+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:06.762797+0000","last_peered":"2024-09-18T11:49:06.762797+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:06.762797+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7176,"log_dups_size":0,"ondisk_log_size":7176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46135955,"num_objects":5633,"num_object_clones":0,"num_object_copies":22532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7175,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3584}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7238","reported_seq":7310,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.178820+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:07.178820+0000","last_peered":"2024-09-18T11:49:07.178820+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:07.178820+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46593270,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7236,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7211","reported_seq":7290,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.938395+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:05.938395+0000","last_peered":"2024-09-18T11:49:05.938395+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:05.938395+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46560502,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3613}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7188","reported_seq":7251,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.786230+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:05.786230+0000","last_peered":"2024-09-18T11:49:05.786230+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:05.786230+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45858816,"num_objects":5598,"num_object_clones":0,"num_object_copies":22392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7188,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7327","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.139693+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:07.139693+0000","last_peered":"2024-09-18T11:49:07.139693+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:07.139693+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7327,"log_dups_size":0,"ondisk_log_size":7327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46601462,"num_objects":5689,"num_object_clones":0,"num_object_copies":22756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7325,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3560}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7191","reported_seq":7264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:06.939146+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:06.939146+0000","last_peered":"2024-09-18T11:49:06.939146+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:06.939146+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7191,"log_dups_size":0,"ondisk_log_size":7191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46560502,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7412","reported_seq":7499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.964772+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:05.964772+0000","last_peered":"2024-09-18T11:49:05.964772+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:05.964772+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47213036,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7409,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7064","reported_seq":7154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:06.767436+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:06.767436+0000","last_peered":"2024-09-18T11:49:06.767436+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:06.767436+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7064,"log_dups_size":0,"ondisk_log_size":7064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45298934,"num_objects":5530,"num_object_clones":0,"num_object_copies":22120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7062,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3557}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7283","reported_seq":7362,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.712415+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:05.712415+0000","last_peered":"2024-09-18T11:49:05.712415+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:05.712415+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7283,"log_dups_size":0,"ondisk_log_size":7283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47099761,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7280,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7289","reported_seq":7348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.761666+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:05.761666+0000","last_peered":"2024-09-18T11:49:05.761666+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:05.761666+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7289,"log_dups_size":0,"ondisk_log_size":7289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47037051,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7288,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3643}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7243","reported_seq":7325,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:06.860246+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:06.860246+0000","last_peered":"2024-09-18T11:49:06.860246+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:06.860246+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7243,"log_dups_size":0,"ondisk_log_size":7243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46889595,"num_objects":5724,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7242,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3672}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7361","reported_seq":7447,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.848671+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:05.848671+0000","last_peered":"2024-09-18T11:49:05.848671+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:05.848671+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46929142,"num_objects":5729,"num_object_clones":0,"num_object_copies":22916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3589}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7239","reported_seq":11524,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:05.901105+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:05.901105+0000","last_peered":"2024-09-18T11:49:05.901105+0000","last_clean":"2024-09-18T11:49:05.901105+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:05.901105+0000","last_undegraded":"2024-09-18T11:49:05.901105+0000","last_fullsized":"2024-09-18T11:49:05.901105+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7239,"log_dups_size":0,"ondisk_log_size":7239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46396662,"num_objects":5664,"num_object_clones":0,"num_object_copies":22656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7237,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7143","reported_seq":7217,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.251288+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:07.251288+0000","last_peered":"2024-09-18T11:49:07.251288+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:07.251288+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7143,"log_dups_size":0,"ondisk_log_size":7143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45577462,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7141,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":745809292,"num_objects":91047,"num_object_clones":0,"num_object_copies":364188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116040,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3810,"num_bytes_recovered":30064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413992448,"data_stored":1413992448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116067,"ondisk_log_size":116067,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671728,"kb_used_data":365520,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93700112,"statfs":{"total":96636764160,"available":95948914688,"internally_reserved":0,"allocated":374292480,"data_stored":374125295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":11.228999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":11.250999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":11.198},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":1.1830000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":1.768},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":11.151999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661860,"kb_used_data":364508,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93709980,"statfs":{"total":96636764160,"available":95959019520,"internally_reserved":0,"allocated":373256192,"data_stored":373089007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,5,1,4,1,5,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.155},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.67400000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":13.048},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":1.1850000000000001},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.65200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599568,"kb_used_data":289288,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93772272,"statfs":{"total":96636764160,"available":96022806528,"internally_reserved":0,"allocated":296230912,"data_stored":296063727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.6110000000000002},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":2.6720000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.0190000000000001},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.5779999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.76400000000000001},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.9790000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674016,"kb_used_data":363736,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93697824,"statfs":{"total":96636764160,"available":95946571776,"internally_reserved":0,"allocated":372465664,"data_stored":372298479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,3,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":13.26},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":29.437000000000001},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.96099999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.054},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":12.9}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":372088832,"data_stored":372088832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295915520,"data_stored":295915520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":372498432,"data_stored":372498432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373489664,"data_stored":373489664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:08.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:08.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7396", 'reported_seq': 7456, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:05.912685+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:05.912685+0000', 'last_peered': '2024-09-18T11:49:05.912685+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:05.912685+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7396, 'log_dups_size': 0, 'ondisk_log_size': 7396, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47955968, 'num_objects': 5854, 'num_object_clones': 0, 'num_object_copies': 23416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7396, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3733}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:08.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:49:08.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7306", 'reported_seq': 10534, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:07.238909+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:49:07.238909+0000', 'last_peered': '2024-09-18T11:49:07.238909+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:49:07.238909+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7306, 'log_dups_size': 0, 'ondisk_log_size': 7306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47101174, 'num_objects': 5750, 'num_object_clones': 0, 'num_object_copies': 23000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 523, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7304, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1583, 'num_bytes_recovered': 12935168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5227}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:08.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:08.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7176", 'reported_seq': 7271, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:06.762797+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:06.762797+0000', 'last_peered': '2024-09-18T11:49:06.762797+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:06.762797+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7176, 'log_dups_size': 0, 'ondisk_log_size': 7176, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46135955, 'num_objects': 5633, 'num_object_clones': 0, 'num_object_copies': 22532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5633, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7175, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3584}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:08.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:08.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7238", 'reported_seq': 7310, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:07.178820+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:07.178820+0000', 'last_peered': '2024-09-18T11:49:07.178820+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:07.178820+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7238, 'log_dups_size': 0, 'ondisk_log_size': 7238, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46593270, 'num_objects': 5688, 'num_object_clones': 0, 'num_object_copies': 22752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7236, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3578}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:08.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:08.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7211", 'reported_seq': 7290, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:05.938395+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:05.938395+0000', 'last_peered': '2024-09-18T11:49:05.938395+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:05.938395+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7211, 'log_dups_size': 0, 'ondisk_log_size': 7211, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46560502, 'num_objects': 5684, 'num_object_clones': 0, 'num_object_copies': 22736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7209, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3613}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:08.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:08.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7188", 'reported_seq': 7251, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:05.786230+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:05.786230+0000', 'last_peered': '2024-09-18T11:49:05.786230+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:05.786230+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7188, 'log_dups_size': 0, 'ondisk_log_size': 7188, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45858816, 'num_objects': 5598, 'num_object_clones': 0, 'num_object_copies': 22392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7188, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3453}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:08.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:08.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7327", 'reported_seq': 7404, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:07.139693+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:07.139693+0000', 'last_peered': '2024-09-18T11:49:07.139693+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:07.139693+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7327, 'log_dups_size': 0, 'ondisk_log_size': 7327, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46601462, 'num_objects': 5689, 'num_object_clones': 0, 'num_object_copies': 22756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2131, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7325, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3560}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:08.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:08.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7191", 'reported_seq': 7264, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:06.939146+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:06.939146+0000', 'last_peered': '2024-09-18T11:49:06.939146+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:06.939146+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7191, 'log_dups_size': 0, 'ondisk_log_size': 7191, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46560502, 'num_objects': 5684, 'num_object_clones': 0, 'num_object_copies': 22736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2032, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7189, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3654}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7412", 'reported_seq': 7499, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:05.964772+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:49:05.964772+0000', 'last_peered': '2024-09-18T11:49:05.964772+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:05.964772+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7412, 'log_dups_size': 0, 'ondisk_log_size': 7412, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47213036, 'num_objects': 5764, 'num_object_clones': 0, 'num_object_copies': 23056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2124, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7409, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3643}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7064", 'reported_seq': 7154, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:06.767436+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:06.767436+0000', 'last_peered': '2024-09-18T11:49:06.767436+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:06.767436+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7064, 'log_dups_size': 0, 'ondisk_log_size': 7064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45298934, 'num_objects': 5530, 'num_object_clones': 0, 'num_object_copies': 22120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1975, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7062, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3557}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7283", 'reported_seq': 7362, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:05.712415+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:05.712415+0000', 'last_peered': '2024-09-18T11:49:05.712415+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:05.712415+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7283, 'log_dups_size': 0, 'ondisk_log_size': 7283, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47099761, 'num_objects': 5750, 'num_object_clones': 0, 'num_object_copies': 23000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7280, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3697}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7289", 'reported_seq': 7348, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:05.761666+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:05.761666+0000', 'last_peered': '2024-09-18T11:49:05.761666+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:05.761666+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7289, 'log_dups_size': 0, 'ondisk_log_size': 7289, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47037051, 'num_objects': 5742, 'num_object_clones': 0, 'num_object_copies': 22968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2100, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7288, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3643}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7243", 'reported_seq': 7325, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:06.860246+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:06.860246+0000', 'last_peered': '2024-09-18T11:49:06.860246+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:06.860246+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7243, 'log_dups_size': 0, 'ondisk_log_size': 7243, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46889595, 'num_objects': 5724, 'num_object_clones': 0, 'num_object_copies': 22896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2053, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7242, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3672}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:08.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:08.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7361", 'reported_seq': 7447, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:05.848671+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:05.848671+0000', 'last_peered': '2024-09-18T11:49:05.848671+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:05.848671+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7361, 'log_dups_size': 0, 'ondisk_log_size': 7361, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46929142, 'num_objects': 5729, 'num_object_clones': 0, 'num_object_copies': 22916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7359, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3589}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:08.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:08.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7143", 'reported_seq': 7217, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:07.251288+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:07.251288+0000', 'last_peered': '2024-09-18T11:49:07.251288+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:07.251288+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7143, 'log_dups_size': 0, 'ondisk_log_size': 7143, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45577462, 'num_objects': 5564, 'num_object_clones': 0, 'num_object_copies': 22256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7141, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:08.337 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:09.050 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:09.051 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29646/364188 objects degraded (8.140%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":347,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91047,"data_bytes":745809292,"bytes_used":2669744128,"bytes_avail":383877312512,"bytes_total":386547056640,"degraded_objects":29646,"degraded_total":364188,"degraded_ratio":0.081403011631355243,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":139193,"recovering_keys_per_sec":0,"num_objects_recovered":246,"num_bytes_recovered":1671168,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":274},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:09.076 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:09.077 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:09.627 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:09.627 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:09.645 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-18T11:49:07.940630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":745809292,"num_objects":91047,"num_object_clones":0,"num_object_copies":364188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116040,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3810,"num_bytes_recovered":30064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116067,"ondisk_log_size":116067,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2607172,"kb_used_data":1383052,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374880188,"statfs":{"total":386547056640,"available":383877312512,"internally_reserved":0,"allocated":1416245248,"data_stored":1415576508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,9,8,9,13,3,7,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26984448,"num_objects":-3294,"num_object_clones":0,"num_object_copies":-13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3294,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006055"},"pg_stats":[{"pgid":"1.f","version":"33'7396","reported_seq":7456,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.912685+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:05.912685+0000","last_peered":"2024-09-18T11:49:05.912685+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:05.912685+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7396,"log_dups_size":0,"ondisk_log_size":7396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47955968,"num_objects":5854,"num_object_clones":0,"num_object_copies":23416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7306","reported_seq":10534,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.238909+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:07.238909+0000","last_peered":"2024-09-18T11:49:07.238909+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:07.238909+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47101174,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1583,"num_bytes_recovered":12935168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7176","reported_seq":7271,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:06.762797+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:06.762797+0000","last_peered":"2024-09-18T11:49:06.762797+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:06.762797+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7176,"log_dups_size":0,"ondisk_log_size":7176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46135955,"num_objects":5633,"num_object_clones":0,"num_object_copies":22532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7175,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3584}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7238","reported_seq":7310,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.178820+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:07.178820+0000","last_peered":"2024-09-18T11:49:07.178820+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:07.178820+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46593270,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7236,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7211","reported_seq":7290,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.938395+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:05.938395+0000","last_peered":"2024-09-18T11:49:05.938395+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:05.938395+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46560502,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3613}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7188","reported_seq":7251,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.786230+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:05.786230+0000","last_peered":"2024-09-18T11:49:05.786230+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:05.786230+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45858816,"num_objects":5598,"num_object_clones":0,"num_object_copies":22392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7188,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7327","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.139693+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:07.139693+0000","last_peered":"2024-09-18T11:49:07.139693+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:07.139693+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7327,"log_dups_size":0,"ondisk_log_size":7327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46601462,"num_objects":5689,"num_object_clones":0,"num_object_copies":22756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7325,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3560}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7191","reported_seq":7264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:06.939146+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:06.939146+0000","last_peered":"2024-09-18T11:49:06.939146+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:06.939146+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7191,"log_dups_size":0,"ondisk_log_size":7191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46560502,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7412","reported_seq":7499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.964772+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:05.964772+0000","last_peered":"2024-09-18T11:49:05.964772+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:05.964772+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47213036,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7409,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7064","reported_seq":7154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:06.767436+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:06.767436+0000","last_peered":"2024-09-18T11:49:06.767436+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:06.767436+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7064,"log_dups_size":0,"ondisk_log_size":7064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45298934,"num_objects":5530,"num_object_clones":0,"num_object_copies":22120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7062,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3557}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7283","reported_seq":7362,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.712415+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:05.712415+0000","last_peered":"2024-09-18T11:49:05.712415+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:05.712415+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7283,"log_dups_size":0,"ondisk_log_size":7283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47099761,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7280,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7289","reported_seq":7348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.761666+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:05.761666+0000","last_peered":"2024-09-18T11:49:05.761666+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:05.761666+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7289,"log_dups_size":0,"ondisk_log_size":7289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47037051,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7288,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3643}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7243","reported_seq":7325,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:06.860246+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:06.860246+0000","last_peered":"2024-09-18T11:49:06.860246+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:06.860246+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7243,"log_dups_size":0,"ondisk_log_size":7243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46889595,"num_objects":5724,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7242,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3672}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7361","reported_seq":7447,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:05.848671+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:05.848671+0000","last_peered":"2024-09-18T11:49:05.848671+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:05.848671+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46929142,"num_objects":5729,"num_object_clones":0,"num_object_copies":22916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3589}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7239","reported_seq":11524,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:05.901105+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:05.901105+0000","last_peered":"2024-09-18T11:49:05.901105+0000","last_clean":"2024-09-18T11:49:05.901105+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:05.901105+0000","last_undegraded":"2024-09-18T11:49:05.901105+0000","last_fullsized":"2024-09-18T11:49:05.901105+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7239,"log_dups_size":0,"ondisk_log_size":7239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46396662,"num_objects":5664,"num_object_clones":0,"num_object_copies":22656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7237,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7143","reported_seq":7217,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.251288+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:07.251288+0000","last_peered":"2024-09-18T11:49:07.251288+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:07.251288+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7143,"log_dups_size":0,"ondisk_log_size":7143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45577462,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7141,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":745809292,"num_objects":91047,"num_object_clones":0,"num_object_copies":364188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116040,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3810,"num_bytes_recovered":30064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413992448,"data_stored":1413992448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116067,"ondisk_log_size":116067,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671728,"kb_used_data":365520,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93700112,"statfs":{"total":96636764160,"available":95948914688,"internally_reserved":0,"allocated":374292480,"data_stored":374125295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":11.228999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":11.250999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":11.198},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":1.1830000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":1.768},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":11.151999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661860,"kb_used_data":364508,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93709980,"statfs":{"total":96636764160,"available":95959019520,"internally_reserved":0,"allocated":373256192,"data_stored":373089007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,5,1,4,1,5,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.155},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.67400000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":13.048},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":1.1850000000000001},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.65200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599568,"kb_used_data":289288,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93772272,"statfs":{"total":96636764160,"available":96022806528,"internally_reserved":0,"allocated":296230912,"data_stored":296063727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.6110000000000002},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":2.6720000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.0190000000000001},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.5779999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.76400000000000001},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.9790000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674016,"kb_used_data":363736,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93697824,"statfs":{"total":96636764160,"available":95946571776,"internally_reserved":0,"allocated":372465664,"data_stored":372298479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,3,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":13.26},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":29.437000000000001},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.96099999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.054},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":12.9}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":372088832,"data_stored":372088832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295915520,"data_stored":295915520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":372498432,"data_stored":372498432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373489664,"data_stored":373489664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:12.647 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:13.296 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:13.297 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:13.315 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-18T11:49:11.941945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":738796940,"num_objects":90191,"num_object_clones":0,"num_object_copies":360764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116896,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3811,"num_bytes_recovered":30064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116923,"ondisk_log_size":116923,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2596388,"kb_used_data":1372284,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374890972,"statfs":{"total":386547056640,"available":383888355328,"internally_reserved":0,"allocated":1405218816,"data_stored":1404550076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,8,11,14,10,5,3,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27983872,"num_objects":-3416,"num_object_clones":0,"num_object_copies":-13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":1171456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005832"},"pg_stats":[{"pgid":"1.f","version":"33'7479","reported_seq":7540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.870535+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:10.870535+0000","last_peered":"2024-09-18T11:49:10.870535+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:10.870535+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47276032,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7479,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7306","reported_seq":10534,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.238909+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:07.238909+0000","last_peered":"2024-09-18T11:49:07.238909+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:07.238909+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47101174,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1583,"num_bytes_recovered":12935168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7244","reported_seq":7339,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.712022+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:11.712022+0000","last_peered":"2024-09-18T11:49:11.712022+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:11.712022+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7244,"log_dups_size":0,"ondisk_log_size":7244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45578899,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7243,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7238","reported_seq":7310,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.178820+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:07.178820+0000","last_peered":"2024-09-18T11:49:07.178820+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:07.178820+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46593270,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7236,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7292","reported_seq":7370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.011521+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:11.011521+0000","last_peered":"2024-09-18T11:49:11.011521+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:11.011521+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7292,"log_dups_size":0,"ondisk_log_size":7292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45896950,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7290,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7272","reported_seq":7335,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.928890+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:10.928890+0000","last_peered":"2024-09-18T11:49:10.928890+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:10.928890+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45170688,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7272,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7327","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.139693+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:07.139693+0000","last_peered":"2024-09-18T11:49:07.139693+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:07.139693+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7327,"log_dups_size":0,"ondisk_log_size":7327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46601462,"num_objects":5689,"num_object_clones":0,"num_object_copies":22756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7325,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3560}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7191","reported_seq":7264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:06.939146+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:06.939146+0000","last_peered":"2024-09-18T11:49:06.939146+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:06.939146+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7191,"log_dups_size":0,"ondisk_log_size":7191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46560502,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7501","reported_seq":7588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.932392+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:10.932392+0000","last_peered":"2024-09-18T11:49:10.932392+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:10.932392+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46483948,"num_objects":5675,"num_object_clones":0,"num_object_copies":22700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3554}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7129","reported_seq":7219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.581385+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:11.581385+0000","last_peered":"2024-09-18T11:49:11.581385+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:11.581385+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44766454,"num_objects":5465,"num_object_clones":0,"num_object_copies":21860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7127,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3492}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7351","reported_seq":7429,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.990238+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:10.990238+0000","last_peered":"2024-09-18T11:49:10.990238+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:10.990238+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46542705,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7348,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3629}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7360","reported_seq":7419,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.884335+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:10.884335+0000","last_peered":"2024-09-18T11:49:10.884335+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:10.884335+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46455419,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3572}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7320","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.420970+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:11.420970+0000","last_peered":"2024-09-18T11:49:11.420970+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:11.420970+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7320,"log_dups_size":0,"ondisk_log_size":7320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46258811,"num_objects":5647,"num_object_clones":0,"num_object_copies":22588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7319,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7446","reported_seq":7533,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.922875+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:10.922875+0000","last_peered":"2024-09-18T11:49:10.922875+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:10.922875+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7446,"log_dups_size":0,"ondisk_log_size":7446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232822,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7444,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3504}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7324","reported_seq":11608,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:10.932672+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:10.932672+0000","last_peered":"2024-09-18T11:49:10.932672+0000","last_clean":"2024-09-18T11:49:10.932672+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:10.932672+0000","last_undegraded":"2024-09-18T11:49:10.932672+0000","last_fullsized":"2024-09-18T11:49:10.932672+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7324,"log_dups_size":0,"ondisk_log_size":7324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45700342,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7322,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7143","reported_seq":7217,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.251288+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:07.251288+0000","last_peered":"2024-09-18T11:49:07.251288+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:07.251288+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7143,"log_dups_size":0,"ondisk_log_size":7143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45577462,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7141,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":738796940,"num_objects":90191,"num_object_clones":0,"num_object_copies":360764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116896,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3811,"num_bytes_recovered":30064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1402884096,"data_stored":1402884096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116923,"ondisk_log_size":116923,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670676,"kb_used_data":360332,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93701164,"statfs":{"total":96636764160,"available":95949991936,"internally_reserved":0,"allocated":368979968,"data_stored":368812783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,3,5,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.6320000000000001},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.6629999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":13.082000000000001},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":5.9029999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":4.7149999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":4.9939999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660892,"kb_used_data":359372,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93710948,"statfs":{"total":96636764160,"available":95960010752,"internally_reserved":0,"allocated":367996928,"data_stored":367829743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,4,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":4.056},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.909}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":21.073},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":14.699999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":1.877},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":12.151}]}]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590804,"kb_used_data":288844,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93781036,"statfs":{"total":96636764160,"available":96031780864,"internally_reserved":0,"allocated":295776256,"data_stored":295609071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.6110000000000002},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":2.6720000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.0190000000000001},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.5779999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.76400000000000001},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.9790000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674016,"kb_used_data":363736,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93697824,"statfs":{"total":96636764160,"available":95946571776,"internally_reserved":0,"allocated":372465664,"data_stored":372298479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,3,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":13.26},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":29.437000000000001},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.96099999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.054},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":12.9}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":372088832,"data_stored":372088832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295428096,"data_stored":295428096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367185920,"data_stored":367185920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":368181248,"data_stored":368181248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:13.317 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:13.885 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:13.886 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:13.914 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-18T11:49:11.941945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":738796940,"num_objects":90191,"num_object_clones":0,"num_object_copies":360764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116896,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3811,"num_bytes_recovered":30064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116923,"ondisk_log_size":116923,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2596388,"kb_used_data":1372284,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":374890972,"statfs":{"total":386547056640,"available":383888355328,"internally_reserved":0,"allocated":1405218816,"data_stored":1404550076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,8,11,14,10,5,3,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27983872,"num_objects":-3416,"num_object_clones":0,"num_object_copies":-13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":1171456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005832"},"pg_stats":[{"pgid":"1.f","version":"33'7479","reported_seq":7540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.870535+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:10.870535+0000","last_peered":"2024-09-18T11:49:10.870535+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:10.870535+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47276032,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7479,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7306","reported_seq":10534,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.238909+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:07.238909+0000","last_peered":"2024-09-18T11:49:07.238909+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:07.238909+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47101174,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1583,"num_bytes_recovered":12935168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7244","reported_seq":7339,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.712022+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:11.712022+0000","last_peered":"2024-09-18T11:49:11.712022+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:11.712022+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7244,"log_dups_size":0,"ondisk_log_size":7244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45578899,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7243,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7238","reported_seq":7310,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.178820+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:07.178820+0000","last_peered":"2024-09-18T11:49:07.178820+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:07.178820+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46593270,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7236,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7292","reported_seq":7370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.011521+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:11.011521+0000","last_peered":"2024-09-18T11:49:11.011521+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:11.011521+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7292,"log_dups_size":0,"ondisk_log_size":7292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45896950,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7290,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7272","reported_seq":7335,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.928890+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:10.928890+0000","last_peered":"2024-09-18T11:49:10.928890+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:10.928890+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45170688,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7272,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7327","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.139693+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:07.139693+0000","last_peered":"2024-09-18T11:49:07.139693+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:07.139693+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7327,"log_dups_size":0,"ondisk_log_size":7327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46601462,"num_objects":5689,"num_object_clones":0,"num_object_copies":22756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7325,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3560}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7191","reported_seq":7264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:06.939146+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:06.939146+0000","last_peered":"2024-09-18T11:49:06.939146+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:06.939146+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7191,"log_dups_size":0,"ondisk_log_size":7191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46560502,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7501","reported_seq":7588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.932392+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:10.932392+0000","last_peered":"2024-09-18T11:49:10.932392+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:10.932392+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46483948,"num_objects":5675,"num_object_clones":0,"num_object_copies":22700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3554}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7129","reported_seq":7219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.581385+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:11.581385+0000","last_peered":"2024-09-18T11:49:11.581385+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:11.581385+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44766454,"num_objects":5465,"num_object_clones":0,"num_object_copies":21860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7127,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3492}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7351","reported_seq":7429,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.990238+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:10.990238+0000","last_peered":"2024-09-18T11:49:10.990238+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:10.990238+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46542705,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7348,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3629}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7360","reported_seq":7419,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.884335+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:10.884335+0000","last_peered":"2024-09-18T11:49:10.884335+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:10.884335+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46455419,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3572}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7320","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.420970+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:11.420970+0000","last_peered":"2024-09-18T11:49:11.420970+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:11.420970+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7320,"log_dups_size":0,"ondisk_log_size":7320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46258811,"num_objects":5647,"num_object_clones":0,"num_object_copies":22588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7319,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7446","reported_seq":7533,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.922875+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:10.922875+0000","last_peered":"2024-09-18T11:49:10.922875+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:10.922875+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7446,"log_dups_size":0,"ondisk_log_size":7446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232822,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7444,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3504}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7324","reported_seq":11608,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:10.932672+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:10.932672+0000","last_peered":"2024-09-18T11:49:10.932672+0000","last_clean":"2024-09-18T11:49:10.932672+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:10.932672+0000","last_undegraded":"2024-09-18T11:49:10.932672+0000","last_fullsized":"2024-09-18T11:49:10.932672+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7324,"log_dups_size":0,"ondisk_log_size":7324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45700342,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7322,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7143","reported_seq":7217,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:07.251288+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:07.251288+0000","last_peered":"2024-09-18T11:49:07.251288+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:07.251288+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7143,"log_dups_size":0,"ondisk_log_size":7143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45577462,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7141,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":738796940,"num_objects":90191,"num_object_clones":0,"num_object_copies":360764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116896,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3811,"num_bytes_recovered":30064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1402884096,"data_stored":1402884096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116923,"ondisk_log_size":116923,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670676,"kb_used_data":360332,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93701164,"statfs":{"total":96636764160,"available":95949991936,"internally_reserved":0,"allocated":368979968,"data_stored":368812783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,3,5,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.6320000000000001},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.6629999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":13.082000000000001},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":5.9029999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":4.7149999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":4.9939999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660892,"kb_used_data":359372,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93710948,"statfs":{"total":96636764160,"available":95960010752,"internally_reserved":0,"allocated":367996928,"data_stored":367829743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,4,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":4.056},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.909}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":21.073},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":14.699999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":1.877},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":12.151}]}]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590804,"kb_used_data":288844,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93781036,"statfs":{"total":96636764160,"available":96031780864,"internally_reserved":0,"allocated":295776256,"data_stored":295609071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.6110000000000002},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":2.6720000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.0190000000000001},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.5779999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.76400000000000001},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.9790000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674016,"kb_used_data":363736,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93697824,"statfs":{"total":96636764160,"available":95946571776,"internally_reserved":0,"allocated":372465664,"data_stored":372298479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,3,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":13.26},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":29.437000000000001},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.96099999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.054},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":12.9}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":372088832,"data_stored":372088832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295428096,"data_stored":295428096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367185920,"data_stored":367185920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":368181248,"data_stored":368181248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:13.916 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:13.916 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7479", 'reported_seq': 7540, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:10.870535+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:10.870535+0000', 'last_peered': '2024-09-18T11:49:10.870535+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:10.870535+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7479, 'log_dups_size': 0, 'ondisk_log_size': 7479, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47276032, 'num_objects': 5771, 'num_object_clones': 0, 'num_object_copies': 23084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7479, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3650}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:13.916 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:49:13.916 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7306", 'reported_seq': 10534, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:07.238909+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:49:07.238909+0000', 'last_peered': '2024-09-18T11:49:07.238909+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:49:07.238909+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7306, 'log_dups_size': 0, 'ondisk_log_size': 7306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47101174, 'num_objects': 5750, 'num_object_clones': 0, 'num_object_copies': 23000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 523, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7304, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1583, 'num_bytes_recovered': 12935168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5227}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:13.916 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:13.916 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7244", 'reported_seq': 7339, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:11.712022+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:11.712022+0000', 'last_peered': '2024-09-18T11:49:11.712022+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:11.712022+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7244, 'log_dups_size': 0, 'ondisk_log_size': 7244, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45578899, 'num_objects': 5565, 'num_object_clones': 0, 'num_object_copies': 22260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5565, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7243, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3516}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:13.916 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:13.916 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7238", 'reported_seq': 7310, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:07.178820+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:07.178820+0000', 'last_peered': '2024-09-18T11:49:07.178820+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:07.178820+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7238, 'log_dups_size': 0, 'ondisk_log_size': 7238, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46593270, 'num_objects': 5688, 'num_object_clones': 0, 'num_object_copies': 22752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7236, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3578}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:13.917 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:13.917 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7292", 'reported_seq': 7370, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:11.011521+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:11.011521+0000', 'last_peered': '2024-09-18T11:49:11.011521+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:11.011521+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7292, 'log_dups_size': 0, 'ondisk_log_size': 7292, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45896950, 'num_objects': 5603, 'num_object_clones': 0, 'num_object_copies': 22412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7290, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3532}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:13.917 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:13.917 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7272", 'reported_seq': 7335, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:10.928890+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:10.928890+0000', 'last_peered': '2024-09-18T11:49:10.928890+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:10.928890+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7272, 'log_dups_size': 0, 'ondisk_log_size': 7272, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45170688, 'num_objects': 5514, 'num_object_clones': 0, 'num_object_copies': 22056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7272, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3369}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:13.917 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:13.917 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7327", 'reported_seq': 7404, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:07.139693+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:07.139693+0000', 'last_peered': '2024-09-18T11:49:07.139693+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:07.139693+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7327, 'log_dups_size': 0, 'ondisk_log_size': 7327, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46601462, 'num_objects': 5689, 'num_object_clones': 0, 'num_object_copies': 22756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2131, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7325, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3560}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:13.917 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:13.917 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7191", 'reported_seq': 7264, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:06.939146+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:06.939146+0000', 'last_peered': '2024-09-18T11:49:06.939146+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:06.939146+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7191, 'log_dups_size': 0, 'ondisk_log_size': 7191, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46560502, 'num_objects': 5684, 'num_object_clones': 0, 'num_object_copies': 22736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2032, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7189, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3654}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:13.917 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7501", 'reported_seq': 7588, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:10.932392+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:49:10.932392+0000', 'last_peered': '2024-09-18T11:49:10.932392+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:10.932392+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7501, 'log_dups_size': 0, 'ondisk_log_size': 7501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46483948, 'num_objects': 5675, 'num_object_clones': 0, 'num_object_copies': 22700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2124, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7498, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3554}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7129", 'reported_seq': 7219, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:11.581385+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:11.581385+0000', 'last_peered': '2024-09-18T11:49:11.581385+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:11.581385+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7129, 'log_dups_size': 0, 'ondisk_log_size': 7129, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44766454, 'num_objects': 5465, 'num_object_clones': 0, 'num_object_copies': 21860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1975, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7127, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3492}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7351", 'reported_seq': 7429, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:10.990238+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:10.990238+0000', 'last_peered': '2024-09-18T11:49:10.990238+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:10.990238+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7351, 'log_dups_size': 0, 'ondisk_log_size': 7351, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46542705, 'num_objects': 5682, 'num_object_clones': 0, 'num_object_copies': 22728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7348, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3629}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7360", 'reported_seq': 7419, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:10.884335+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:10.884335+0000', 'last_peered': '2024-09-18T11:49:10.884335+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:10.884335+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7360, 'log_dups_size': 0, 'ondisk_log_size': 7360, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46455419, 'num_objects': 5671, 'num_object_clones': 0, 'num_object_copies': 22684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2100, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7359, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7320", 'reported_seq': 7404, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:11.420970+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:11.420970+0000', 'last_peered': '2024-09-18T11:49:11.420970+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:11.420970+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7320, 'log_dups_size': 0, 'ondisk_log_size': 7320, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46258811, 'num_objects': 5647, 'num_object_clones': 0, 'num_object_copies': 22588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2052, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7319, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3596}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:13.918 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7446", 'reported_seq': 7533, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:10.922875+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:10.922875+0000', 'last_peered': '2024-09-18T11:49:10.922875+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:10.922875+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7446, 'log_dups_size': 0, 'ondisk_log_size': 7446, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46232822, 'num_objects': 5644, 'num_object_clones': 0, 'num_object_copies': 22576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7444, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3504}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:13.919 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:13.919 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7143", 'reported_seq': 7217, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:07.251288+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:07.251288+0000', 'last_peered': '2024-09-18T11:49:07.251288+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:07.251288+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7143, 'log_dups_size': 0, 'ondisk_log_size': 7143, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45577462, 'num_objects': 5564, 'num_object_clones': 0, 'num_object_copies': 22256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7141, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:13.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:14.604 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:14.604 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29584/359124 objects degraded (8.238%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":352,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89781,"data_bytes":735438220,"bytes_used":2640629760,"bytes_avail":383906426880,"bytes_total":386547056640,"degraded_objects":29584,"degraded_total":359124,"degraded_ratio":0.082378231474365404,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":138512,"recovering_keys_per_sec":0,"num_objects_recovered":225,"num_bytes_recovered":1662976,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":244},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:14.658 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:14.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:15.247 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:15.252 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:15.285 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-18T11:49:13.942827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":735438220,"num_objects":89781,"num_object_clones":0,"num_object_copies":359124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117306,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3872,"num_bytes_recovered":30556160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117333,"ondisk_log_size":117333,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2578740,"kb_used_data":1367116,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":374908620,"statfs":{"total":386547056640,"available":383906426880,"internally_reserved":0,"allocated":1399926784,"data_stored":1399258044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,12,15,11,7,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24076288,"num_objects":-2939,"num_object_clones":0,"num_object_copies":-11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2939,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005931"},"pg_stats":[{"pgid":"1.f","version":"33'7479","reported_seq":7540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.870535+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:10.870535+0000","last_peered":"2024-09-18T11:49:10.870535+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:10.870535+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47276032,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7479,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7392","reported_seq":10739,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:12.246832+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:12.246832+0000","last_peered":"2024-09-18T11:49:12.246832+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:12.246832+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7392,"log_dups_size":0,"ondisk_log_size":7392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46396662,"num_objects":5664,"num_object_clones":0,"num_object_copies":22656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7390,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1643,"num_bytes_recovered":13426688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7244","reported_seq":7339,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.712022+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:11.712022+0000","last_peered":"2024-09-18T11:49:11.712022+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:11.712022+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7244,"log_dups_size":0,"ondisk_log_size":7244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45578899,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7243,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7314","reported_seq":7386,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:12.191043+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:12.191043+0000","last_peered":"2024-09-18T11:49:12.191043+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:12.191043+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45970678,"num_objects":5612,"num_object_clones":0,"num_object_copies":22448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7312,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7292","reported_seq":7370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.011521+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:11.011521+0000","last_peered":"2024-09-18T11:49:11.011521+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:11.011521+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7292,"log_dups_size":0,"ondisk_log_size":7292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45896950,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7290,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7272","reported_seq":7335,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.928890+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:10.928890+0000","last_peered":"2024-09-18T11:49:10.928890+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:10.928890+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45170688,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7272,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7414","reported_seq":7491,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:12.140097+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:12.140097+0000","last_peered":"2024-09-18T11:49:12.140097+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:12.140097+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7414,"log_dups_size":0,"ondisk_log_size":7414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45888758,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3473}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7276","reported_seq":7351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:12.239685+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:12.239685+0000","last_peered":"2024-09-18T11:49:12.239685+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:12.239685+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7276,"log_dups_size":0,"ondisk_log_size":7276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45864182,"num_objects":5599,"num_object_clones":0,"num_object_copies":22396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7274,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7501","reported_seq":7588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.932392+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:10.932392+0000","last_peered":"2024-09-18T11:49:10.932392+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:10.932392+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46483948,"num_objects":5675,"num_object_clones":0,"num_object_copies":22700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3554}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7129","reported_seq":7219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.581385+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:11.581385+0000","last_peered":"2024-09-18T11:49:11.581385+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:11.581385+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44766454,"num_objects":5465,"num_object_clones":0,"num_object_copies":21860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7127,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3492}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7351","reported_seq":7429,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.990238+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:10.990238+0000","last_peered":"2024-09-18T11:49:10.990238+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:10.990238+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46542705,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7348,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3629}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7360","reported_seq":7419,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.884335+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:10.884335+0000","last_peered":"2024-09-18T11:49:10.884335+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:10.884335+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46455419,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3572}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7320","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:11.420970+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:11.420970+0000","last_peered":"2024-09-18T11:49:11.420970+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:11.420970+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7320,"log_dups_size":0,"ondisk_log_size":7320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46258811,"num_objects":5647,"num_object_clones":0,"num_object_copies":22588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7319,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7446","reported_seq":7533,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:10.922875+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:10.922875+0000","last_peered":"2024-09-18T11:49:10.922875+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:10.922875+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7446,"log_dups_size":0,"ondisk_log_size":7446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232822,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7444,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3504}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7324","reported_seq":11608,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:10.932672+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:10.932672+0000","last_peered":"2024-09-18T11:49:10.932672+0000","last_clean":"2024-09-18T11:49:10.932672+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:10.932672+0000","last_undegraded":"2024-09-18T11:49:10.932672+0000","last_fullsized":"2024-09-18T11:49:10.932672+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7324,"log_dups_size":0,"ondisk_log_size":7324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45700342,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7322,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7219","reported_seq":7295,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:12.096277+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:12.096277+0000","last_peered":"2024-09-18T11:49:12.096277+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:12.096277+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7219,"log_dups_size":0,"ondisk_log_size":7219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44954870,"num_objects":5488,"num_object_clones":0,"num_object_copies":21952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7217,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3481}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":735438220,"num_objects":89781,"num_object_clones":0,"num_object_copies":359124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117306,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3872,"num_bytes_recovered":30556160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1397600256,"data_stored":1397600256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117333,"ondisk_log_size":117333,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670676,"kb_used_data":360332,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93701164,"statfs":{"total":96636764160,"available":95949991936,"internally_reserved":0,"allocated":368979968,"data_stored":368812783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,3,5,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.6320000000000001},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.6629999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":13.082000000000001},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":5.9029999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":4.7149999999999999},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":4.9939999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660892,"kb_used_data":359372,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93710948,"statfs":{"total":96636764160,"available":95960010752,"internally_reserved":0,"allocated":367996928,"data_stored":367829743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,4,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":4.056},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.909}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":21.073},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":14.699999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":1.877},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":12.151}]}]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590804,"kb_used_data":288844,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93781036,"statfs":{"total":96636764160,"available":96031780864,"internally_reserved":0,"allocated":295776256,"data_stored":295609071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.6110000000000002},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":2.6720000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.0190000000000001},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.5779999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.76400000000000001},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.9790000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":656368,"kb_used_data":358568,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93715472,"statfs":{"total":96636764160,"available":95964643328,"internally_reserved":0,"allocated":367173632,"data_stored":367006447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":0.70899999999999996},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.69299999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.9429999999999996},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.25}]},{"osd":3,"last update":"Wed Sep 18 11:48:16 2024","interfaces":[{"interface":"back","average":{"1min":4.2779999999999996,"5min":3.194,"15min":2.6379999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":18.832000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":6.7510000000000003,"5min":3.3730000000000002,"15min":2.1379999999999999},"min":{"1min":0.27400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.038,"5min":19.038,"15min":19.038},"last":11.351000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":366804992,"data_stored":366804992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295428096,"data_stored":295428096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367185920,"data_stored":367185920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":368181248,"data_stored":368181248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:18.287 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:18.797 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:18.817 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-18T11:49:17.944111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":725034380,"num_objects":88511,"num_object_clones":0,"num_object_copies":354044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88511,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118576,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3969,"num_bytes_recovered":31162368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118603,"ondisk_log_size":118603,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2567760,"kb_used_data":1351944,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374919600,"statfs":{"total":386547056640,"available":383917670400,"internally_reserved":0,"allocated":1384390656,"data_stored":1383721916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,10,11,9,17,10,0,0,0,0,9,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-31293440,"num_objects":-3820,"num_object_clones":0,"num_object_copies":-15280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004049"},"pg_stats":[{"pgid":"1.f","version":"33'7562","reported_seq":7629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.976687+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:15.976687+0000","last_peered":"2024-09-18T11:49:15.976687+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:15.976687+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46596096,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7562,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7484","reported_seq":10980,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.269968+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:17.269968+0000","last_peered":"2024-09-18T11:49:17.269968+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:17.269968+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7484,"log_dups_size":0,"ondisk_log_size":7484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45642998,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":14032896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7317","reported_seq":7418,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.876908+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:16.876908+0000","last_peered":"2024-09-18T11:49:16.876908+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:16.876908+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7317,"log_dups_size":0,"ondisk_log_size":7317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44980883,"num_objects":5492,"num_object_clones":0,"num_object_copies":21968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7316,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3446}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7404","reported_seq":7479,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.232599+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:17.232599+0000","last_peered":"2024-09-18T11:49:17.232599+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:17.232599+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7404,"log_dups_size":0,"ondisk_log_size":7404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45233398,"num_objects":5522,"num_object_clones":0,"num_object_copies":22088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7402,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7371","reported_seq":7452,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.864476+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:15.864476+0000","last_peered":"2024-09-18T11:49:15.864476+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:15.864476+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7371,"log_dups_size":0,"ondisk_log_size":7371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249782,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7361","reported_seq":7424,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.001669+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:16.001669+0000","last_peered":"2024-09-18T11:49:16.001669+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:16.001669+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44441600,"num_objects":5425,"num_object_clones":0,"num_object_copies":21700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7497","reported_seq":7578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.108510+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:17.108510+0000","last_peered":"2024-09-18T11:49:17.108510+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:17.108510+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208822,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7495,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3392}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7363","reported_seq":7446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.120155+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:17.120155+0000","last_peered":"2024-09-18T11:49:17.120155+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:17.120155+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7363,"log_dups_size":0,"ondisk_log_size":7363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45151478,"num_objects":5512,"num_object_clones":0,"num_object_copies":22048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7573","reported_seq":7663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.925265+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:15.925265+0000","last_peered":"2024-09-18T11:49:15.925265+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:15.925265+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45894124,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7570,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7203","reported_seq":7295,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.785520+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:16.785520+0000","last_peered":"2024-09-18T11:49:16.785520+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:16.785520+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44160246,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7201,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7430","reported_seq":7519,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.036000+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:16.036000+0000","last_peered":"2024-09-18T11:49:16.036000+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:16.036000+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45895537,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7444","reported_seq":7505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.928708+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:15.928708+0000","last_peered":"2024-09-18T11:49:15.928708+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:15.928708+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7444,"log_dups_size":0,"ondisk_log_size":7444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45767291,"num_objects":5587,"num_object_clones":0,"num_object_copies":22348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7443,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7393","reported_seq":7477,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.768526+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:16.768526+0000","last_peered":"2024-09-18T11:49:16.768526+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:16.768526+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45660795,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7529","reported_seq":7615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.954512+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:15.954512+0000","last_peered":"2024-09-18T11:49:15.954512+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:15.954512+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45552886,"num_objects":5561,"num_object_clones":0,"num_object_copies":22244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7385","reported_seq":11670,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:15.678198+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:15.678198+0000","last_peered":"2024-09-18T11:49:15.678198+0000","last_clean":"2024-09-18T11:49:15.678198+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:15.678198+0000","last_undegraded":"2024-09-18T11:49:15.678198+0000","last_fullsized":"2024-09-18T11:49:15.678198+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45200630,"num_objects":5518,"num_object_clones":0,"num_object_copies":22072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7287","reported_seq":7363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.070914+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:17.070914+0000","last_peered":"2024-09-18T11:49:17.070914+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:17.070914+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7287,"log_dups_size":0,"ondisk_log_size":7287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44397814,"num_objects":5420,"num_object_clones":0,"num_object_copies":21680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7285,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3413}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":725034380,"num_objects":88511,"num_object_clones":0,"num_object_copies":354044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88511,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118576,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3969,"num_bytes_recovered":31162368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1382035456,"data_stored":1382035456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118603,"ondisk_log_size":118603,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653140,"kb_used_data":355256,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93718700,"statfs":{"total":96636764160,"available":95967948800,"internally_reserved":0,"allocated":363782144,"data_stored":363614959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,3,3,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.63600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":2.7250000000000001},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.643}]}]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660112,"kb_used_data":354428,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93711728,"statfs":{"total":96636764160,"available":95960809472,"internally_reserved":0,"allocated":362934272,"data_stored":362767087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,6,2,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":3.8660000000000001},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.8959999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.833},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":1.4630000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":14.394},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":6.2439999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594628,"kb_used_data":288508,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93777212,"statfs":{"total":96636764160,"available":96027865088,"internally_reserved":0,"allocated":295432192,"data_stored":295265007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.747}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.39000000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":4.3419999999999996},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.0940000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659880,"kb_used_data":353752,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93711960,"statfs":{"total":96636764160,"available":95961047040,"internally_reserved":0,"allocated":362242048,"data_stored":362074863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,2,3,1,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":2.0790000000000002},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.6360000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.0470000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.5919999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":6.6660000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361750528,"data_stored":361750528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295104512,"data_stored":295104512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362127360,"data_stored":362127360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363053056,"data_stored":363053056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:18.818 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:19.360 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:19.361 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:19.382 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-18T11:49:17.944111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":725034380,"num_objects":88511,"num_object_clones":0,"num_object_copies":354044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88511,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118576,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3969,"num_bytes_recovered":31162368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118603,"ondisk_log_size":118603,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2567760,"kb_used_data":1351944,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374919600,"statfs":{"total":386547056640,"available":383917670400,"internally_reserved":0,"allocated":1384390656,"data_stored":1383721916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,10,11,9,17,10,0,0,0,0,9,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-31293440,"num_objects":-3820,"num_object_clones":0,"num_object_copies":-15280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004049"},"pg_stats":[{"pgid":"1.f","version":"33'7562","reported_seq":7629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.976687+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:15.976687+0000","last_peered":"2024-09-18T11:49:15.976687+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:15.976687+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46596096,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7562,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7484","reported_seq":10980,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.269968+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:17.269968+0000","last_peered":"2024-09-18T11:49:17.269968+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:17.269968+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7484,"log_dups_size":0,"ondisk_log_size":7484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45642998,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":14032896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7317","reported_seq":7418,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.876908+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:16.876908+0000","last_peered":"2024-09-18T11:49:16.876908+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:16.876908+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7317,"log_dups_size":0,"ondisk_log_size":7317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44980883,"num_objects":5492,"num_object_clones":0,"num_object_copies":21968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7316,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3446}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7404","reported_seq":7479,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.232599+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:17.232599+0000","last_peered":"2024-09-18T11:49:17.232599+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:17.232599+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7404,"log_dups_size":0,"ondisk_log_size":7404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45233398,"num_objects":5522,"num_object_clones":0,"num_object_copies":22088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7402,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7371","reported_seq":7452,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.864476+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:15.864476+0000","last_peered":"2024-09-18T11:49:15.864476+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:15.864476+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7371,"log_dups_size":0,"ondisk_log_size":7371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249782,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7361","reported_seq":7424,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.001669+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:16.001669+0000","last_peered":"2024-09-18T11:49:16.001669+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:16.001669+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44441600,"num_objects":5425,"num_object_clones":0,"num_object_copies":21700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7497","reported_seq":7578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.108510+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:17.108510+0000","last_peered":"2024-09-18T11:49:17.108510+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:17.108510+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208822,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7495,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3392}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7363","reported_seq":7446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.120155+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:17.120155+0000","last_peered":"2024-09-18T11:49:17.120155+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:17.120155+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7363,"log_dups_size":0,"ondisk_log_size":7363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45151478,"num_objects":5512,"num_object_clones":0,"num_object_copies":22048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7573","reported_seq":7663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.925265+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:15.925265+0000","last_peered":"2024-09-18T11:49:15.925265+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:15.925265+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45894124,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7570,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7203","reported_seq":7295,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.785520+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:16.785520+0000","last_peered":"2024-09-18T11:49:16.785520+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:16.785520+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44160246,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7201,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7430","reported_seq":7519,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.036000+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:16.036000+0000","last_peered":"2024-09-18T11:49:16.036000+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:16.036000+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45895537,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7444","reported_seq":7505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.928708+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:15.928708+0000","last_peered":"2024-09-18T11:49:15.928708+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:15.928708+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7444,"log_dups_size":0,"ondisk_log_size":7444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45767291,"num_objects":5587,"num_object_clones":0,"num_object_copies":22348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7443,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7393","reported_seq":7477,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.768526+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:16.768526+0000","last_peered":"2024-09-18T11:49:16.768526+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:16.768526+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45660795,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7529","reported_seq":7615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.954512+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:15.954512+0000","last_peered":"2024-09-18T11:49:15.954512+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:15.954512+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45552886,"num_objects":5561,"num_object_clones":0,"num_object_copies":22244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7385","reported_seq":11670,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:15.678198+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:15.678198+0000","last_peered":"2024-09-18T11:49:15.678198+0000","last_clean":"2024-09-18T11:49:15.678198+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:15.678198+0000","last_undegraded":"2024-09-18T11:49:15.678198+0000","last_fullsized":"2024-09-18T11:49:15.678198+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45200630,"num_objects":5518,"num_object_clones":0,"num_object_copies":22072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7287","reported_seq":7363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.070914+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:17.070914+0000","last_peered":"2024-09-18T11:49:17.070914+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:17.070914+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7287,"log_dups_size":0,"ondisk_log_size":7287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44397814,"num_objects":5420,"num_object_clones":0,"num_object_copies":21680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7285,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3413}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":725034380,"num_objects":88511,"num_object_clones":0,"num_object_copies":354044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88511,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118576,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3969,"num_bytes_recovered":31162368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1382035456,"data_stored":1382035456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118603,"ondisk_log_size":118603,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653140,"kb_used_data":355256,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93718700,"statfs":{"total":96636764160,"available":95967948800,"internally_reserved":0,"allocated":363782144,"data_stored":363614959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,3,3,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.63600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":2.7250000000000001},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.643}]}]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660112,"kb_used_data":354428,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93711728,"statfs":{"total":96636764160,"available":95960809472,"internally_reserved":0,"allocated":362934272,"data_stored":362767087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,6,2,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":3.8660000000000001},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.8959999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.833},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":1.4630000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":14.394},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":6.2439999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594628,"kb_used_data":288508,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93777212,"statfs":{"total":96636764160,"available":96027865088,"internally_reserved":0,"allocated":295432192,"data_stored":295265007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.747}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.39000000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":4.3419999999999996},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.0940000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659880,"kb_used_data":353752,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93711960,"statfs":{"total":96636764160,"available":95961047040,"internally_reserved":0,"allocated":362242048,"data_stored":362074863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,2,3,1,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":2.0790000000000002},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.6360000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.0470000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.5919999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":6.6660000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361750528,"data_stored":361750528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295104512,"data_stored":295104512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362127360,"data_stored":362127360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363053056,"data_stored":363053056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:19.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:19.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7562", 'reported_seq': 7629, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:15.976687+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:15.976687+0000', 'last_peered': '2024-09-18T11:49:15.976687+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:15.976687+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7562, 'log_dups_size': 0, 'ondisk_log_size': 7562, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46596096, 'num_objects': 5688, 'num_object_clones': 0, 'num_object_copies': 22752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7562, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3570}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:19.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:49:19.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7484", 'reported_seq': 10980, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:17.269968+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:49:17.269968+0000', 'last_peered': '2024-09-18T11:49:17.269968+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:49:17.269968+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7484, 'log_dups_size': 0, 'ondisk_log_size': 7484, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45642998, 'num_objects': 5572, 'num_object_clones': 0, 'num_object_copies': 22288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 389, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7482, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1717, 'num_bytes_recovered': 14032896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5183}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:19.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:19.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7317", 'reported_seq': 7418, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:16.876908+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:16.876908+0000', 'last_peered': '2024-09-18T11:49:16.876908+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:16.876908+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7317, 'log_dups_size': 0, 'ondisk_log_size': 7317, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44980883, 'num_objects': 5492, 'num_object_clones': 0, 'num_object_copies': 21968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5492, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7316, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3446}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:19.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:19.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7404", 'reported_seq': 7479, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:17.232599+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:17.232599+0000', 'last_peered': '2024-09-18T11:49:17.232599+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:17.232599+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7404, 'log_dups_size': 0, 'ondisk_log_size': 7404, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45233398, 'num_objects': 5522, 'num_object_clones': 0, 'num_object_copies': 22088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7402, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3414}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:19.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:19.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7371", 'reported_seq': 7452, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:15.864476+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:15.864476+0000', 'last_peered': '2024-09-18T11:49:15.864476+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:15.864476+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7371, 'log_dups_size': 0, 'ondisk_log_size': 7371, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45249782, 'num_objects': 5524, 'num_object_clones': 0, 'num_object_copies': 22096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7369, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3454}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:19.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:19.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7361", 'reported_seq': 7424, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:16.001669+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:16.001669+0000', 'last_peered': '2024-09-18T11:49:16.001669+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:16.001669+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7361, 'log_dups_size': 0, 'ondisk_log_size': 7361, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44441600, 'num_objects': 5425, 'num_object_clones': 0, 'num_object_copies': 21700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7361, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3280}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:19.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:19.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7497", 'reported_seq': 7578, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:17.108510+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:17.108510+0000', 'last_peered': '2024-09-18T11:49:17.108510+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:17.108510+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7497, 'log_dups_size': 0, 'ondisk_log_size': 7497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45208822, 'num_objects': 5519, 'num_object_clones': 0, 'num_object_copies': 22076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7495, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3392}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:19.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:19.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7363", 'reported_seq': 7446, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:17.120155+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:17.120155+0000', 'last_peered': '2024-09-18T11:49:17.120155+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:17.120155+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7363, 'log_dups_size': 0, 'ondisk_log_size': 7363, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45151478, 'num_objects': 5512, 'num_object_clones': 0, 'num_object_copies': 22048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2027, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7361, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3487}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:19.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:19.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7573", 'reported_seq': 7663, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:15.925265+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:49:15.925265+0000', 'last_peered': '2024-09-18T11:49:15.925265+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:15.925265+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7573, 'log_dups_size': 0, 'ondisk_log_size': 7573, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45894124, 'num_objects': 5603, 'num_object_clones': 0, 'num_object_copies': 22412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7570, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3483}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:19.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:19.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7203", 'reported_seq': 7295, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:16.785520+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:16.785520+0000', 'last_peered': '2024-09-18T11:49:16.785520+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:16.785520+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7203, 'log_dups_size': 0, 'ondisk_log_size': 7203, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44160246, 'num_objects': 5391, 'num_object_clones': 0, 'num_object_copies': 21564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1974, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7201, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3419}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:19.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:19.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7430", 'reported_seq': 7519, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:16.036000+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:16.036000+0000', 'last_peered': '2024-09-18T11:49:16.036000+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:16.036000+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7430, 'log_dups_size': 0, 'ondisk_log_size': 7430, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45895537, 'num_objects': 5603, 'num_object_clones': 0, 'num_object_copies': 22412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7427, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3555}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:19.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:19.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7444", 'reported_seq': 7505, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:15.928708+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:15.928708+0000', 'last_peered': '2024-09-18T11:49:15.928708+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:15.928708+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7444, 'log_dups_size': 0, 'ondisk_log_size': 7444, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45767291, 'num_objects': 5587, 'num_object_clones': 0, 'num_object_copies': 22348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2099, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7443, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:19.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:19.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7393", 'reported_seq': 7477, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:16.768526+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:16.768526+0000', 'last_peered': '2024-09-18T11:49:16.768526+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:16.768526+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7393, 'log_dups_size': 0, 'ondisk_log_size': 7393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45660795, 'num_objects': 5574, 'num_object_clones': 0, 'num_object_copies': 22296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2052, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7392, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3523}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:19.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:19.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7529", 'reported_seq': 7615, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:15.954512+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:15.954512+0000', 'last_peered': '2024-09-18T11:49:15.954512+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:15.954512+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7529, 'log_dups_size': 0, 'ondisk_log_size': 7529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45552886, 'num_objects': 5561, 'num_object_clones': 0, 'num_object_copies': 22244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7527, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:19.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:19.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7287", 'reported_seq': 7363, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:17.070914+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:17.070914+0000', 'last_peered': '2024-09-18T11:49:17.070914+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:17.070914+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7287, 'log_dups_size': 0, 'ondisk_log_size': 7287, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44397814, 'num_objects': 5420, 'num_object_clones': 0, 'num_object_copies': 21680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7285, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3413}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:19.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:20.040 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:20.040 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29487/354044 objects degraded (8.329%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":358,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88511,"data_bytes":725034380,"bytes_used":2629386240,"bytes_avail":383917670400,"bytes_total":386547056640,"degraded_objects":29487,"degraded_total":354044,"degraded_ratio":0.083286258205194835,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":139217,"recovering_keys_per_sec":0,"num_objects_recovered":249,"num_bytes_recovered":1671168,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":318},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:20.062 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:20.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:20.602 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:20.603 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:20.627 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-18T11:49:19.944969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":725034380,"num_objects":88511,"num_object_clones":0,"num_object_copies":354044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88511,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118576,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3969,"num_bytes_recovered":31162368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118603,"ondisk_log_size":118603,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2567760,"kb_used_data":1351944,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374919600,"statfs":{"total":386547056640,"available":383917670400,"internally_reserved":0,"allocated":1384390656,"data_stored":1383721916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,10,11,9,17,10,0,0,0,0,9,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20774912,"num_objects":-2536,"num_object_clones":0,"num_object_copies":-10144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2536,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004338"},"pg_stats":[{"pgid":"1.f","version":"33'7562","reported_seq":7629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.976687+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:15.976687+0000","last_peered":"2024-09-18T11:49:15.976687+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:15.976687+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46596096,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7562,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7484","reported_seq":10980,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.269968+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:17.269968+0000","last_peered":"2024-09-18T11:49:17.269968+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:17.269968+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7484,"log_dups_size":0,"ondisk_log_size":7484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45642998,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":14032896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7317","reported_seq":7418,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.876908+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:16.876908+0000","last_peered":"2024-09-18T11:49:16.876908+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:16.876908+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7317,"log_dups_size":0,"ondisk_log_size":7317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44980883,"num_objects":5492,"num_object_clones":0,"num_object_copies":21968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7316,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3446}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7404","reported_seq":7479,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.232599+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:17.232599+0000","last_peered":"2024-09-18T11:49:17.232599+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:17.232599+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7404,"log_dups_size":0,"ondisk_log_size":7404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45233398,"num_objects":5522,"num_object_clones":0,"num_object_copies":22088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7402,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7371","reported_seq":7452,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.864476+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:15.864476+0000","last_peered":"2024-09-18T11:49:15.864476+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:15.864476+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7371,"log_dups_size":0,"ondisk_log_size":7371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249782,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7361","reported_seq":7424,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.001669+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:16.001669+0000","last_peered":"2024-09-18T11:49:16.001669+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:16.001669+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44441600,"num_objects":5425,"num_object_clones":0,"num_object_copies":21700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7497","reported_seq":7578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.108510+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:17.108510+0000","last_peered":"2024-09-18T11:49:17.108510+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:17.108510+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208822,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7495,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3392}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7363","reported_seq":7446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.120155+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:17.120155+0000","last_peered":"2024-09-18T11:49:17.120155+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:17.120155+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7363,"log_dups_size":0,"ondisk_log_size":7363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45151478,"num_objects":5512,"num_object_clones":0,"num_object_copies":22048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7573","reported_seq":7663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.925265+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:15.925265+0000","last_peered":"2024-09-18T11:49:15.925265+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:15.925265+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45894124,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7570,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7203","reported_seq":7295,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.785520+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:16.785520+0000","last_peered":"2024-09-18T11:49:16.785520+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:16.785520+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44160246,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7201,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7430","reported_seq":7519,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.036000+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:16.036000+0000","last_peered":"2024-09-18T11:49:16.036000+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:16.036000+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45895537,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7444","reported_seq":7505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.928708+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:15.928708+0000","last_peered":"2024-09-18T11:49:15.928708+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:15.928708+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7444,"log_dups_size":0,"ondisk_log_size":7444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45767291,"num_objects":5587,"num_object_clones":0,"num_object_copies":22348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7443,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7393","reported_seq":7477,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:16.768526+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:16.768526+0000","last_peered":"2024-09-18T11:49:16.768526+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:16.768526+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45660795,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7529","reported_seq":7615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:15.954512+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:15.954512+0000","last_peered":"2024-09-18T11:49:15.954512+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:15.954512+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45552886,"num_objects":5561,"num_object_clones":0,"num_object_copies":22244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7385","reported_seq":11670,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:15.678198+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:15.678198+0000","last_peered":"2024-09-18T11:49:15.678198+0000","last_clean":"2024-09-18T11:49:15.678198+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:15.678198+0000","last_undegraded":"2024-09-18T11:49:15.678198+0000","last_fullsized":"2024-09-18T11:49:15.678198+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45200630,"num_objects":5518,"num_object_clones":0,"num_object_copies":22072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7287","reported_seq":7363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:17.070914+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:17.070914+0000","last_peered":"2024-09-18T11:49:17.070914+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:17.070914+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7287,"log_dups_size":0,"ondisk_log_size":7287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44397814,"num_objects":5420,"num_object_clones":0,"num_object_copies":21680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7285,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3413}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":725034380,"num_objects":88511,"num_object_clones":0,"num_object_copies":354044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88511,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118576,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3969,"num_bytes_recovered":31162368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1382035456,"data_stored":1382035456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118603,"ondisk_log_size":118603,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653140,"kb_used_data":355256,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93718700,"statfs":{"total":96636764160,"available":95967948800,"internally_reserved":0,"allocated":363782144,"data_stored":363614959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,3,3,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.6459999999999999,"5min":3.8700000000000001,"15min":3.1579999999999999},"min":{"1min":0.30599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":21.539999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":4.673,"5min":4.2199999999999998,"15min":3.1899999999999999},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":21.562000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.63600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:48:14 2024","interfaces":[{"interface":"back","average":{"1min":6.1660000000000004,"5min":3.1240000000000001,"15min":1.7},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":29.087,"5min":29.087,"15min":29.087},"last":2.7250000000000001},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.6970000000000001,"15min":2.9590000000000001},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":21.213000000000001,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.643}]}]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660112,"kb_used_data":354428,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93711728,"statfs":{"total":96636764160,"available":95960809472,"internally_reserved":0,"allocated":362934272,"data_stored":362767087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,6,2,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":3.8660000000000001},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.8959999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.833},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":1.4630000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:15 2024","interfaces":[{"interface":"back","average":{"1min":3.145,"5min":3.2799999999999998,"15min":2.6899999999999999},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.8209999999999997,"5min":14.34,"15min":14.34},"last":14.394},{"interface":"front","average":{"1min":2.274,"5min":2.6440000000000001,"15min":2.0139999999999998},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0970000000000004,"5min":16.673999999999999,"15min":16.673999999999999},"last":6.2439999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594628,"kb_used_data":288508,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93777212,"statfs":{"total":96636764160,"available":96027865088,"internally_reserved":0,"allocated":295432192,"data_stored":295265007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.747}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.39000000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":4.3419999999999996},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.0940000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659880,"kb_used_data":353752,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93711960,"statfs":{"total":96636764160,"available":95961047040,"internally_reserved":0,"allocated":362242048,"data_stored":362074863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,2,3,1,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":2.0790000000000002},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.6360000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.0470000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.5919999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":6.6660000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361750528,"data_stored":361750528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":295104512,"data_stored":295104512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362127360,"data_stored":362127360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363053056,"data_stored":363053056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:23.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:24.189 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:24.189 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:24.210 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-18T11:49:23.946629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":715195788,"num_objects":87310,"num_object_clones":0,"num_object_copies":349240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119777,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4065,"num_bytes_recovered":31891456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119804,"ondisk_log_size":119804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2569424,"kb_used_data":1337004,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":374917936,"statfs":{"total":386547056640,"available":383915966464,"internally_reserved":0,"allocated":1369092096,"data_stored":1368423356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,6,8,4,7,0,0,0,0,0,2,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23601152,"num_objects":-2881,"num_object_clones":0,"num_object_copies":-11524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2881,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":1826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004684"},"pg_stats":[{"pgid":"1.f","version":"33'7637","reported_seq":7704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.998554+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:20.998554+0000","last_peered":"2024-09-18T11:49:20.998554+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:20.998554+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7637,"log_dups_size":0,"ondisk_log_size":7637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45981696,"num_objects":5613,"num_object_clones":0,"num_object_copies":22452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7637,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7552","reported_seq":11226,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.270294+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:22.270294+0000","last_peered":"2024-09-18T11:49:22.270294+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:22.270294+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7552,"log_dups_size":0,"ondisk_log_size":7552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45085942,"num_objects":5504,"num_object_clones":0,"num_object_copies":22016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1806,"num_bytes_recovered":14761984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7385","reported_seq":7487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.893046+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:21.893046+0000","last_peered":"2024-09-18T11:49:21.893046+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:21.893046+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44423827,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7384,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3379}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7483","reported_seq":7559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.119421+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:22.119421+0000","last_peered":"2024-09-18T11:49:22.119421+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:22.119421+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44586230,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7481,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7453","reported_seq":7536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.056939+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:21.056939+0000","last_peered":"2024-09-18T11:49:21.056939+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:21.056939+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44578038,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7451,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7449","reported_seq":7512,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.944799+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:20.944799+0000","last_peered":"2024-09-18T11:49:20.944799+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:20.944799+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43720704,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7449,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7563","reported_seq":7644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.240940+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:22.240940+0000","last_peered":"2024-09-18T11:49:22.240940+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:22.240940+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44668150,"num_objects":5453,"num_object_clones":0,"num_object_copies":21812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7561,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3326}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7426","reported_seq":7509,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.117136+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:22.117136+0000","last_peered":"2024-09-18T11:49:22.117136+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:22.117136+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44635382,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7644","reported_seq":7734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.947237+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:20.947237+0000","last_peered":"2024-09-18T11:49:20.947237+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:20.947237+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7644,"log_dups_size":0,"ondisk_log_size":7644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45312492,"num_objects":5532,"num_object_clones":0,"num_object_copies":22128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7641,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7285","reported_seq":7381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.763881+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:21.763881+0000","last_peered":"2024-09-18T11:49:21.763881+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:21.763881+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7285,"log_dups_size":0,"ondisk_log_size":7285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43488502,"num_objects":5309,"num_object_clones":0,"num_object_copies":21236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7283,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7513","reported_seq":7606,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.983598+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:20.983598+0000","last_peered":"2024-09-18T11:49:20.983598+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:20.983598+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7513,"log_dups_size":0,"ondisk_log_size":7513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45215601,"num_objects":5520,"num_object_clones":0,"num_object_copies":22080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7521","reported_seq":7584,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.040975+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:21.040975+0000","last_peered":"2024-09-18T11:49:21.040975+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:21.040975+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7521,"log_dups_size":0,"ondisk_log_size":7521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45136507,"num_objects":5510,"num_object_clones":0,"num_object_copies":22040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3413}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7465","reported_seq":7549,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.666038+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:21.666038+0000","last_peered":"2024-09-18T11:49:21.666038+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:21.666038+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45070971,"num_objects":5502,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7464,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7606","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.866693+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:20.866693+0000","last_peered":"2024-09-18T11:49:20.866693+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:20.866693+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7606,"log_dups_size":0,"ondisk_log_size":7606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44922102,"num_objects":5484,"num_object_clones":0,"num_object_copies":21936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3344}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7460","reported_seq":11744,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:21.040424+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:21.040424+0000","last_peered":"2024-09-18T11:49:21.040424+0000","last_clean":"2024-09-18T11:49:21.040424+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:21.040424+0000","last_undegraded":"2024-09-18T11:49:21.040424+0000","last_fullsized":"2024-09-18T11:49:21.040424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7460,"log_dups_size":0,"ondisk_log_size":7460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44586230,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7458,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7362","reported_seq":7438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.180159+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:22.180159+0000","last_peered":"2024-09-18T11:49:22.180159+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:22.180159+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7362,"log_dups_size":0,"ondisk_log_size":7362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43783414,"num_objects":5345,"num_object_clones":0,"num_object_copies":21380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7360,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":715195788,"num_objects":87310,"num_object_clones":0,"num_object_copies":349240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119777,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4065,"num_bytes_recovered":31891456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1367060480,"data_stored":1367060480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119804,"ondisk_log_size":119804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652372,"kb_used_data":350344,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93719468,"statfs":{"total":96636764160,"available":95968735232,"internally_reserved":0,"allocated":358752256,"data_stored":358585071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":9.4770000000000003},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":9.5210000000000008}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":9.5510000000000002},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":3.75},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.4140000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659332,"kb_used_data":349500,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93712508,"statfs":{"total":96636764160,"available":95961608192,"internally_reserved":0,"allocated":357888000,"data_stored":357720815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.343},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.3740000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.6180000000000001},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":19.789999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":5.2939999999999996},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.9550000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598532,"kb_used_data":288252,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93773308,"statfs":{"total":96636764160,"available":96023867392,"internally_reserved":0,"allocated":295170048,"data_stored":295002863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":14.894},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":1.135},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":14.962}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.798}]}]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659188,"kb_used_data":348908,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93712652,"statfs":{"total":96636764160,"available":95961755648,"internally_reserved":0,"allocated":357281792,"data_stored":357114607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,1,1,2,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":0.38},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.435}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.4710000000000001},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.45900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":7.0229999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356900864,"data_stored":356900864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294805504,"data_stored":294805504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357236736,"data_stored":357236736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":358117376,"data_stored":358117376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:24.212 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:24.770 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:24.771 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:24.793 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-18T11:49:23.946629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":715195788,"num_objects":87310,"num_object_clones":0,"num_object_copies":349240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119777,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4065,"num_bytes_recovered":31891456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119804,"ondisk_log_size":119804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2569424,"kb_used_data":1337004,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":374917936,"statfs":{"total":386547056640,"available":383915966464,"internally_reserved":0,"allocated":1369092096,"data_stored":1368423356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,6,8,4,7,0,0,0,0,0,2,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23601152,"num_objects":-2881,"num_object_clones":0,"num_object_copies":-11524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2881,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":1826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004684"},"pg_stats":[{"pgid":"1.f","version":"33'7637","reported_seq":7704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.998554+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:20.998554+0000","last_peered":"2024-09-18T11:49:20.998554+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:20.998554+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7637,"log_dups_size":0,"ondisk_log_size":7637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45981696,"num_objects":5613,"num_object_clones":0,"num_object_copies":22452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7637,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7552","reported_seq":11226,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.270294+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:22.270294+0000","last_peered":"2024-09-18T11:49:22.270294+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:22.270294+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7552,"log_dups_size":0,"ondisk_log_size":7552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45085942,"num_objects":5504,"num_object_clones":0,"num_object_copies":22016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1806,"num_bytes_recovered":14761984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7385","reported_seq":7487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.893046+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:21.893046+0000","last_peered":"2024-09-18T11:49:21.893046+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:21.893046+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44423827,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7384,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3379}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7483","reported_seq":7559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.119421+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:22.119421+0000","last_peered":"2024-09-18T11:49:22.119421+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:22.119421+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44586230,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7481,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7453","reported_seq":7536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.056939+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:21.056939+0000","last_peered":"2024-09-18T11:49:21.056939+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:21.056939+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44578038,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7451,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7449","reported_seq":7512,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.944799+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:20.944799+0000","last_peered":"2024-09-18T11:49:20.944799+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:20.944799+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43720704,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7449,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7563","reported_seq":7644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.240940+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:22.240940+0000","last_peered":"2024-09-18T11:49:22.240940+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:22.240940+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44668150,"num_objects":5453,"num_object_clones":0,"num_object_copies":21812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7561,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3326}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7426","reported_seq":7509,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.117136+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:22.117136+0000","last_peered":"2024-09-18T11:49:22.117136+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:22.117136+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44635382,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7644","reported_seq":7734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.947237+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:20.947237+0000","last_peered":"2024-09-18T11:49:20.947237+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:20.947237+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7644,"log_dups_size":0,"ondisk_log_size":7644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45312492,"num_objects":5532,"num_object_clones":0,"num_object_copies":22128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7641,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7285","reported_seq":7381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.763881+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:21.763881+0000","last_peered":"2024-09-18T11:49:21.763881+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:21.763881+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7285,"log_dups_size":0,"ondisk_log_size":7285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43488502,"num_objects":5309,"num_object_clones":0,"num_object_copies":21236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7283,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7513","reported_seq":7606,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.983598+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:20.983598+0000","last_peered":"2024-09-18T11:49:20.983598+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:20.983598+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7513,"log_dups_size":0,"ondisk_log_size":7513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45215601,"num_objects":5520,"num_object_clones":0,"num_object_copies":22080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7521","reported_seq":7584,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.040975+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:21.040975+0000","last_peered":"2024-09-18T11:49:21.040975+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:21.040975+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7521,"log_dups_size":0,"ondisk_log_size":7521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45136507,"num_objects":5510,"num_object_clones":0,"num_object_copies":22040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3413}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7465","reported_seq":7549,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.666038+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:21.666038+0000","last_peered":"2024-09-18T11:49:21.666038+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:21.666038+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45070971,"num_objects":5502,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7464,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7606","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.866693+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:20.866693+0000","last_peered":"2024-09-18T11:49:20.866693+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:20.866693+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7606,"log_dups_size":0,"ondisk_log_size":7606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44922102,"num_objects":5484,"num_object_clones":0,"num_object_copies":21936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3344}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7460","reported_seq":11744,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:21.040424+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:21.040424+0000","last_peered":"2024-09-18T11:49:21.040424+0000","last_clean":"2024-09-18T11:49:21.040424+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:21.040424+0000","last_undegraded":"2024-09-18T11:49:21.040424+0000","last_fullsized":"2024-09-18T11:49:21.040424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7460,"log_dups_size":0,"ondisk_log_size":7460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44586230,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7458,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7362","reported_seq":7438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.180159+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:22.180159+0000","last_peered":"2024-09-18T11:49:22.180159+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:22.180159+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7362,"log_dups_size":0,"ondisk_log_size":7362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43783414,"num_objects":5345,"num_object_clones":0,"num_object_copies":21380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7360,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":715195788,"num_objects":87310,"num_object_clones":0,"num_object_copies":349240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119777,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4065,"num_bytes_recovered":31891456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1367060480,"data_stored":1367060480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119804,"ondisk_log_size":119804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652372,"kb_used_data":350344,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93719468,"statfs":{"total":96636764160,"available":95968735232,"internally_reserved":0,"allocated":358752256,"data_stored":358585071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":9.4770000000000003},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":9.5210000000000008}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":9.5510000000000002},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":3.75},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.4140000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659332,"kb_used_data":349500,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93712508,"statfs":{"total":96636764160,"available":95961608192,"internally_reserved":0,"allocated":357888000,"data_stored":357720815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.343},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.3740000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.6180000000000001},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":19.789999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":5.2939999999999996},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.9550000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598532,"kb_used_data":288252,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93773308,"statfs":{"total":96636764160,"available":96023867392,"internally_reserved":0,"allocated":295170048,"data_stored":295002863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":14.894},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":1.135},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":14.962}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.798}]}]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659188,"kb_used_data":348908,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93712652,"statfs":{"total":96636764160,"available":95961755648,"internally_reserved":0,"allocated":357281792,"data_stored":357114607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,1,1,2,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":0.38},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.435}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.4710000000000001},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.45900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":7.0229999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356900864,"data_stored":356900864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294805504,"data_stored":294805504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357236736,"data_stored":357236736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":358117376,"data_stored":358117376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:24.795 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:24.795 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7637", 'reported_seq': 7704, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:20.998554+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:20.998554+0000', 'last_peered': '2024-09-18T11:49:20.998554+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:20.998554+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7637, 'log_dups_size': 0, 'ondisk_log_size': 7637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45981696, 'num_objects': 5613, 'num_object_clones': 0, 'num_object_copies': 22452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7637, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3495}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:24.795 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:49:24.795 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7552", 'reported_seq': 11226, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:22.270294+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:49:22.270294+0000', 'last_peered': '2024-09-18T11:49:22.270294+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:49:22.270294+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7552, 'log_dups_size': 0, 'ondisk_log_size': 7552, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45085942, 'num_objects': 5504, 'num_object_clones': 0, 'num_object_copies': 22016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 300, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7550, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1806, 'num_bytes_recovered': 14761984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5204}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:24.795 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:24.795 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7385", 'reported_seq': 7487, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:21.893046+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:21.893046+0000', 'last_peered': '2024-09-18T11:49:21.893046+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:21.893046+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7385, 'log_dups_size': 0, 'ondisk_log_size': 7385, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44423827, 'num_objects': 5424, 'num_object_clones': 0, 'num_object_copies': 21696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5424, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7384, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3379}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:24.795 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:24.796 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7483", 'reported_seq': 7559, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:22.119421+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:22.119421+0000', 'last_peered': '2024-09-18T11:49:22.119421+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:22.119421+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7483, 'log_dups_size': 0, 'ondisk_log_size': 7483, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44586230, 'num_objects': 5443, 'num_object_clones': 0, 'num_object_copies': 21772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7481, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:24.796 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:24.796 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7453", 'reported_seq': 7536, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:21.056939+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:21.056939+0000', 'last_peered': '2024-09-18T11:49:21.056939+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:21.056939+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7453, 'log_dups_size': 0, 'ondisk_log_size': 7453, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44578038, 'num_objects': 5442, 'num_object_clones': 0, 'num_object_copies': 21768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2071, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7451, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3373}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:24.796 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:24.796 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7449", 'reported_seq': 7512, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:20.944799+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:20.944799+0000', 'last_peered': '2024-09-18T11:49:20.944799+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:20.944799+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7449, 'log_dups_size': 0, 'ondisk_log_size': 7449, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43720704, 'num_objects': 5337, 'num_object_clones': 0, 'num_object_copies': 21348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7449, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3192}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:24.796 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:24.796 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7563", 'reported_seq': 7644, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:22.240940+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:22.240940+0000', 'last_peered': '2024-09-18T11:49:22.240940+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:22.240940+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7563, 'log_dups_size': 0, 'ondisk_log_size': 7563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44668150, 'num_objects': 5453, 'num_object_clones': 0, 'num_object_copies': 21812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7561, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3326}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:24.796 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:24.796 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7426", 'reported_seq': 7509, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:22.117136+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:22.117136+0000', 'last_peered': '2024-09-18T11:49:22.117136+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:22.117136+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7426, 'log_dups_size': 0, 'ondisk_log_size': 7426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44635382, 'num_objects': 5449, 'num_object_clones': 0, 'num_object_copies': 21796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2027, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7424, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3424}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:24.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:24.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7644", 'reported_seq': 7734, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:20.947237+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:49:20.947237+0000', 'last_peered': '2024-09-18T11:49:20.947237+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:20.947237+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7644, 'log_dups_size': 0, 'ondisk_log_size': 7644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45312492, 'num_objects': 5532, 'num_object_clones': 0, 'num_object_copies': 22128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7641, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3412}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:24.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:24.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7285", 'reported_seq': 7381, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:21.763881+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:21.763881+0000', 'last_peered': '2024-09-18T11:49:21.763881+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:21.763881+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7285, 'log_dups_size': 0, 'ondisk_log_size': 7285, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43488502, 'num_objects': 5309, 'num_object_clones': 0, 'num_object_copies': 21236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1972, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7283, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3339}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:24.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:24.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7513", 'reported_seq': 7606, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:20.983598+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:20.983598+0000', 'last_peered': '2024-09-18T11:49:20.983598+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:20.983598+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7513, 'log_dups_size': 0, 'ondisk_log_size': 7513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45215601, 'num_objects': 5520, 'num_object_clones': 0, 'num_object_copies': 22080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7510, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:24.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:24.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7521", 'reported_seq': 7584, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:21.040975+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:21.040975+0000', 'last_peered': '2024-09-18T11:49:21.040975+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:21.040975+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7521, 'log_dups_size': 0, 'ondisk_log_size': 7521, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45136507, 'num_objects': 5510, 'num_object_clones': 0, 'num_object_copies': 22040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2098, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7520, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3413}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:24.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:24.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7465", 'reported_seq': 7549, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:21.666038+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:21.666038+0000', 'last_peered': '2024-09-18T11:49:21.666038+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:21.666038+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7465, 'log_dups_size': 0, 'ondisk_log_size': 7465, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45070971, 'num_objects': 5502, 'num_object_clones': 0, 'num_object_copies': 22008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2052, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7464, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3451}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:24.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:24.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7606", 'reported_seq': 7693, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:20.866693+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:20.866693+0000', 'last_peered': '2024-09-18T11:49:20.866693+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:20.866693+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7606, 'log_dups_size': 0, 'ondisk_log_size': 7606, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44922102, 'num_objects': 5484, 'num_object_clones': 0, 'num_object_copies': 21936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7604, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3344}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:24.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:24.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7362", 'reported_seq': 7438, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:22.180159+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:22.180159+0000', 'last_peered': '2024-09-18T11:49:22.180159+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:22.180159+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7362, 'log_dups_size': 0, 'ondisk_log_size': 7362, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43783414, 'num_objects': 5345, 'num_object_clones': 0, 'num_object_copies': 21380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7360, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3338}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:24.798 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:25.456 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:25.456 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29391/349240 objects degraded (8.416%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":363,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":87310,"data_bytes":715195788,"bytes_used":2631090176,"bytes_avail":383915966464,"bytes_total":386547056640,"degraded_objects":29391,"degraded_total":349240,"degraded_ratio":0.084157026686519296,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":152175,"recovering_keys_per_sec":0,"num_objects_recovered":254,"num_bytes_recovered":1826816,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":239},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:25.479 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:25.479 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:26.039 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:26.074 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-18T11:49:25.947415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":715195788,"num_objects":87310,"num_object_clones":0,"num_object_copies":349240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119777,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4065,"num_bytes_recovered":31891456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119804,"ondisk_log_size":119804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2569424,"kb_used_data":1337004,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":374917936,"statfs":{"total":386547056640,"available":383915966464,"internally_reserved":0,"allocated":1369092096,"data_stored":1368423356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,6,8,4,7,0,0,0,0,0,2,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20242432,"num_objects":-2471,"num_object_clones":0,"num_object_copies":-9884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2471,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1335296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004587"},"pg_stats":[{"pgid":"1.f","version":"33'7637","reported_seq":7704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.998554+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:20.998554+0000","last_peered":"2024-09-18T11:49:20.998554+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:20.998554+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7637,"log_dups_size":0,"ondisk_log_size":7637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45981696,"num_objects":5613,"num_object_clones":0,"num_object_copies":22452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7637,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7552","reported_seq":11226,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.270294+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:22.270294+0000","last_peered":"2024-09-18T11:49:22.270294+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:22.270294+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7552,"log_dups_size":0,"ondisk_log_size":7552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45085942,"num_objects":5504,"num_object_clones":0,"num_object_copies":22016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1806,"num_bytes_recovered":14761984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7385","reported_seq":7487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.893046+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:21.893046+0000","last_peered":"2024-09-18T11:49:21.893046+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:21.893046+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44423827,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7384,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3379}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7483","reported_seq":7559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.119421+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:22.119421+0000","last_peered":"2024-09-18T11:49:22.119421+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:22.119421+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44586230,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7481,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7453","reported_seq":7536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.056939+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:21.056939+0000","last_peered":"2024-09-18T11:49:21.056939+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:21.056939+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44578038,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7451,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7449","reported_seq":7512,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.944799+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:20.944799+0000","last_peered":"2024-09-18T11:49:20.944799+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:20.944799+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43720704,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7449,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7563","reported_seq":7644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.240940+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:22.240940+0000","last_peered":"2024-09-18T11:49:22.240940+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:22.240940+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44668150,"num_objects":5453,"num_object_clones":0,"num_object_copies":21812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7561,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3326}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7426","reported_seq":7509,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.117136+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:22.117136+0000","last_peered":"2024-09-18T11:49:22.117136+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:22.117136+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44635382,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7644","reported_seq":7734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.947237+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:20.947237+0000","last_peered":"2024-09-18T11:49:20.947237+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:20.947237+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7644,"log_dups_size":0,"ondisk_log_size":7644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45312492,"num_objects":5532,"num_object_clones":0,"num_object_copies":22128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7641,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7285","reported_seq":7381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.763881+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:21.763881+0000","last_peered":"2024-09-18T11:49:21.763881+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:21.763881+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7285,"log_dups_size":0,"ondisk_log_size":7285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43488502,"num_objects":5309,"num_object_clones":0,"num_object_copies":21236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7283,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7513","reported_seq":7606,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.983598+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:20.983598+0000","last_peered":"2024-09-18T11:49:20.983598+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:20.983598+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7513,"log_dups_size":0,"ondisk_log_size":7513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45215601,"num_objects":5520,"num_object_clones":0,"num_object_copies":22080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7521","reported_seq":7584,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.040975+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:21.040975+0000","last_peered":"2024-09-18T11:49:21.040975+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:21.040975+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7521,"log_dups_size":0,"ondisk_log_size":7521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45136507,"num_objects":5510,"num_object_clones":0,"num_object_copies":22040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3413}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7465","reported_seq":7549,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:21.666038+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:21.666038+0000","last_peered":"2024-09-18T11:49:21.666038+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:21.666038+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45070971,"num_objects":5502,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7464,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7606","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:20.866693+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:20.866693+0000","last_peered":"2024-09-18T11:49:20.866693+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:20.866693+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7606,"log_dups_size":0,"ondisk_log_size":7606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44922102,"num_objects":5484,"num_object_clones":0,"num_object_copies":21936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3344}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7460","reported_seq":11744,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:21.040424+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:21.040424+0000","last_peered":"2024-09-18T11:49:21.040424+0000","last_clean":"2024-09-18T11:49:21.040424+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:21.040424+0000","last_undegraded":"2024-09-18T11:49:21.040424+0000","last_fullsized":"2024-09-18T11:49:21.040424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7460,"log_dups_size":0,"ondisk_log_size":7460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44586230,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7458,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7362","reported_seq":7438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:22.180159+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:22.180159+0000","last_peered":"2024-09-18T11:49:22.180159+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:22.180159+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7362,"log_dups_size":0,"ondisk_log_size":7362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43783414,"num_objects":5345,"num_object_clones":0,"num_object_copies":21380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7360,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":715195788,"num_objects":87310,"num_object_clones":0,"num_object_copies":349240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119777,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4065,"num_bytes_recovered":31891456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1367060480,"data_stored":1367060480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119804,"ondisk_log_size":119804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652372,"kb_used_data":350344,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93719468,"statfs":{"total":96636764160,"available":95968735232,"internally_reserved":0,"allocated":358752256,"data_stored":358585071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":9.4770000000000003},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":9.5210000000000008}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":9.5510000000000002},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":3.75},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.4140000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659332,"kb_used_data":349500,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93712508,"statfs":{"total":96636764160,"available":95961608192,"internally_reserved":0,"allocated":357888000,"data_stored":357720815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.343},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.3740000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.6180000000000001},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":19.789999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":5.2939999999999996},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.9550000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598532,"kb_used_data":288252,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93773308,"statfs":{"total":96636764160,"available":96023867392,"internally_reserved":0,"allocated":295170048,"data_stored":295002863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":14.894},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":1.135},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":14.962}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.798}]}]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659188,"kb_used_data":348908,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93712652,"statfs":{"total":96636764160,"available":95961755648,"internally_reserved":0,"allocated":357281792,"data_stored":357114607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,1,1,2,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":0.38},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.435}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.4710000000000001},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.45900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":7.0229999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356900864,"data_stored":356900864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294805504,"data_stored":294805504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357236736,"data_stored":357236736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":358117376,"data_stored":358117376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:29.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:29.599 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:29.600 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:29.620 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-18T11:49:27.948196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":705897868,"num_objects":86175,"num_object_clones":0,"num_object_copies":344700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120912,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":32636928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120939,"ondisk_log_size":120939,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2522028,"kb_used_data":1322884,"kb_used_omap":14,"kb_used_meta":1199089,"kb_avail":374965332,"statfs":{"total":386547056640,"available":383964499968,"internally_reserved":0,"allocated":1354633216,"data_stored":1353964476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227867163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,6,5,12,5,0,0,0,0,0,2,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29540352,"num_objects":-3606,"num_object_clones":0,"num_object_copies":-14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":289,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004650"},"pg_stats":[{"pgid":"1.f","version":"33'7710","reported_seq":7777,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.033088+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:26.033088+0000","last_peered":"2024-09-18T11:49:26.033088+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:26.033088+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45383680,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7710,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7629","reported_seq":11485,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.315695+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:27.315695+0000","last_peered":"2024-09-18T11:49:27.315695+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:27.315695+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7629,"log_dups_size":0,"ondisk_log_size":7629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44455158,"num_objects":5427,"num_object_clones":0,"num_object_copies":21708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1897,"num_bytes_recovered":15507456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7454","reported_seq":7557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.909088+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:26.909088+0000","last_peered":"2024-09-18T11:49:26.909088+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:26.909088+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43858579,"num_objects":5355,"num_object_clones":0,"num_object_copies":21420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5355,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7453,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7560","reported_seq":7636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.974217+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:26.974217+0000","last_peered":"2024-09-18T11:49:26.974217+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:26.974217+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43955446,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7522","reported_seq":7605,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:25.969126+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:25.969126+0000","last_peered":"2024-09-18T11:49:25.969126+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:25.969126+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7522,"log_dups_size":0,"ondisk_log_size":7522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44012790,"num_objects":5373,"num_object_clones":0,"num_object_copies":21492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7520","reported_seq":7583,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:25.769491+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:25.769491+0000","last_peered":"2024-09-18T11:49:25.769491+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:25.769491+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7520,"log_dups_size":0,"ondisk_log_size":7520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43139072,"num_objects":5266,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7640","reported_seq":7724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.102997+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:27.102997+0000","last_peered":"2024-09-18T11:49:27.102997+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:27.102997+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7640,"log_dups_size":0,"ondisk_log_size":7640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44037366,"num_objects":5376,"num_object_clones":0,"num_object_copies":21504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7638,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3251}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7497","reported_seq":7586,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.242436+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:27.242436+0000","last_peered":"2024-09-18T11:49:27.242436+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:27.242436+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44053750,"num_objects":5378,"num_object_clones":0,"num_object_copies":21512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7495,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7719","reported_seq":7809,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.021223+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:26.021223+0000","last_peered":"2024-09-18T11:49:26.021223+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:26.021223+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44698092,"num_objects":5457,"num_object_clones":0,"num_object_copies":21828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7716,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7347","reported_seq":7443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.899524+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:26.899524+0000","last_peered":"2024-09-18T11:49:26.899524+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:26.899524+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42980598,"num_objects":5247,"num_object_clones":0,"num_object_copies":20988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7345,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7571","reported_seq":7664,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.001638+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:26.001638+0000","last_peered":"2024-09-18T11:49:26.001638+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:26.001638+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7571,"log_dups_size":0,"ondisk_log_size":7571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44740465,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7568,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7598","reported_seq":7661,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.030924+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:26.030924+0000","last_peered":"2024-09-18T11:49:26.030924+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:26.030924+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7598,"log_dups_size":0,"ondisk_log_size":7598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44505723,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7597,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3336}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7530","reported_seq":7614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.699934+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:26.699934+0000","last_peered":"2024-09-18T11:49:26.699934+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:26.699934+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7530,"log_dups_size":0,"ondisk_log_size":7530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44538491,"num_objects":5437,"num_object_clones":0,"num_object_copies":21748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7529,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7673","reported_seq":7760,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.000798+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:26.000798+0000","last_peered":"2024-09-18T11:49:26.000798+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:26.000798+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44373238,"num_objects":5417,"num_object_clones":0,"num_object_copies":21668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7529","reported_seq":11814,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:26.034315+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:26.034315+0000","last_peered":"2024-09-18T11:49:26.034315+0000","last_clean":"2024-09-18T11:49:26.034315+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:26.034315+0000","last_undegraded":"2024-09-18T11:49:26.034315+0000","last_fullsized":"2024-09-18T11:49:26.034315+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44020982,"num_objects":5374,"num_object_clones":0,"num_object_copies":21496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7440","reported_seq":7515,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.205246+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:27.205246+0000","last_peered":"2024-09-18T11:49:27.205246+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:27.205246+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7440,"log_dups_size":0,"ondisk_log_size":7440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43144438,"num_objects":5267,"num_object_clones":0,"num_object_copies":21068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7438,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":705897868,"num_objects":86175,"num_object_clones":0,"num_object_copies":344700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120912,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":32636928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1352765440,"data_stored":1352765440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120939,"ondisk_log_size":120939,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651852,"kb_used_data":345664,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93719988,"statfs":{"total":96636764160,"available":95969267712,"internally_reserved":0,"allocated":353959936,"data_stored":353792751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,1,5,2,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":19.792000000000002},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":19.847999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":6.7329999999999997},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":10.821999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":12.084},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.94499999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642228,"kb_used_data":344876,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93729612,"statfs":{"total":96636764160,"available":95979122688,"internally_reserved":0,"allocated":353153024,"data_stored":352985839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.68600000000000005}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":17.045999999999999},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":16.968}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":17.001000000000001},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":7.4939999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585852,"kb_used_data":288052,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93785988,"statfs":{"total":96636764160,"available":96036851712,"internally_reserved":0,"allocated":294965248,"data_stored":294798063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.9780000000000002},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.77700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":1.8919999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.0609999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":1.8540000000000001},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.9380000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642096,"kb_used_data":344292,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93729744,"statfs":{"total":96636764160,"available":95979257856,"internally_reserved":0,"allocated":352555008,"data_stored":352387823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":26.189},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":2.5470000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":2.5750000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":26.149999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":10.975}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352174080,"data_stored":352174080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294617088,"data_stored":294617088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352518144,"data_stored":352518144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":353456128,"data_stored":353456128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:29.621 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:30.138 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:30.138 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:30.159 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-18T11:49:29.949018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":705897868,"num_objects":86175,"num_object_clones":0,"num_object_copies":344700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120912,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":32636928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120939,"ondisk_log_size":120939,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2522028,"kb_used_data":1322884,"kb_used_omap":14,"kb_used_meta":1199089,"kb_avail":374965332,"statfs":{"total":386547056640,"available":383964499968,"internally_reserved":0,"allocated":1354633216,"data_stored":1353964476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227867163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,6,5,12,5,0,0,0,0,0,2,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19136512,"num_objects":-2336,"num_object_clones":0,"num_object_copies":-9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2336,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004907"},"pg_stats":[{"pgid":"1.f","version":"33'7710","reported_seq":7777,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.033088+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:26.033088+0000","last_peered":"2024-09-18T11:49:26.033088+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:26.033088+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45383680,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7710,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7629","reported_seq":11485,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.315695+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:27.315695+0000","last_peered":"2024-09-18T11:49:27.315695+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:27.315695+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7629,"log_dups_size":0,"ondisk_log_size":7629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44455158,"num_objects":5427,"num_object_clones":0,"num_object_copies":21708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1897,"num_bytes_recovered":15507456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7454","reported_seq":7557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.909088+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:26.909088+0000","last_peered":"2024-09-18T11:49:26.909088+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:26.909088+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43858579,"num_objects":5355,"num_object_clones":0,"num_object_copies":21420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5355,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7453,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7560","reported_seq":7636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.974217+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:26.974217+0000","last_peered":"2024-09-18T11:49:26.974217+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:26.974217+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43955446,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7522","reported_seq":7605,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:25.969126+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:25.969126+0000","last_peered":"2024-09-18T11:49:25.969126+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:25.969126+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7522,"log_dups_size":0,"ondisk_log_size":7522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44012790,"num_objects":5373,"num_object_clones":0,"num_object_copies":21492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7520","reported_seq":7583,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:25.769491+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:25.769491+0000","last_peered":"2024-09-18T11:49:25.769491+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:25.769491+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7520,"log_dups_size":0,"ondisk_log_size":7520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43139072,"num_objects":5266,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7640","reported_seq":7724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.102997+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:27.102997+0000","last_peered":"2024-09-18T11:49:27.102997+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:27.102997+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7640,"log_dups_size":0,"ondisk_log_size":7640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44037366,"num_objects":5376,"num_object_clones":0,"num_object_copies":21504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7638,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3251}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7497","reported_seq":7586,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.242436+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:27.242436+0000","last_peered":"2024-09-18T11:49:27.242436+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:27.242436+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44053750,"num_objects":5378,"num_object_clones":0,"num_object_copies":21512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7495,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7719","reported_seq":7809,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.021223+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:26.021223+0000","last_peered":"2024-09-18T11:49:26.021223+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:26.021223+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44698092,"num_objects":5457,"num_object_clones":0,"num_object_copies":21828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7716,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7347","reported_seq":7443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.899524+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:26.899524+0000","last_peered":"2024-09-18T11:49:26.899524+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:26.899524+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42980598,"num_objects":5247,"num_object_clones":0,"num_object_copies":20988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7345,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7571","reported_seq":7664,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.001638+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:26.001638+0000","last_peered":"2024-09-18T11:49:26.001638+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:26.001638+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7571,"log_dups_size":0,"ondisk_log_size":7571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44740465,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7568,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7598","reported_seq":7661,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.030924+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:26.030924+0000","last_peered":"2024-09-18T11:49:26.030924+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:26.030924+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7598,"log_dups_size":0,"ondisk_log_size":7598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44505723,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7597,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3336}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7530","reported_seq":7614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.699934+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:26.699934+0000","last_peered":"2024-09-18T11:49:26.699934+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:26.699934+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7530,"log_dups_size":0,"ondisk_log_size":7530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44538491,"num_objects":5437,"num_object_clones":0,"num_object_copies":21748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7529,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7673","reported_seq":7760,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.000798+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:26.000798+0000","last_peered":"2024-09-18T11:49:26.000798+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:26.000798+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44373238,"num_objects":5417,"num_object_clones":0,"num_object_copies":21668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7529","reported_seq":11814,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:26.034315+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:26.034315+0000","last_peered":"2024-09-18T11:49:26.034315+0000","last_clean":"2024-09-18T11:49:26.034315+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:26.034315+0000","last_undegraded":"2024-09-18T11:49:26.034315+0000","last_fullsized":"2024-09-18T11:49:26.034315+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44020982,"num_objects":5374,"num_object_clones":0,"num_object_copies":21496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7440","reported_seq":7515,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.205246+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:27.205246+0000","last_peered":"2024-09-18T11:49:27.205246+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:27.205246+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7440,"log_dups_size":0,"ondisk_log_size":7440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43144438,"num_objects":5267,"num_object_clones":0,"num_object_copies":21068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7438,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":705897868,"num_objects":86175,"num_object_clones":0,"num_object_copies":344700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120912,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":32636928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1352765440,"data_stored":1352765440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120939,"ondisk_log_size":120939,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651852,"kb_used_data":345664,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93719988,"statfs":{"total":96636764160,"available":95969267712,"internally_reserved":0,"allocated":353959936,"data_stored":353792751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,1,5,2,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":19.792000000000002},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":19.847999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":6.7329999999999997},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":10.821999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":12.084},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.94499999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642228,"kb_used_data":344876,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93729612,"statfs":{"total":96636764160,"available":95979122688,"internally_reserved":0,"allocated":353153024,"data_stored":352985839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.68600000000000005}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":17.045999999999999},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":16.968}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":17.001000000000001},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":7.4939999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585852,"kb_used_data":288052,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93785988,"statfs":{"total":96636764160,"available":96036851712,"internally_reserved":0,"allocated":294965248,"data_stored":294798063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.9780000000000002},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.77700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":1.8919999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.0609999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":1.8540000000000001},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.9380000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642096,"kb_used_data":344292,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93729744,"statfs":{"total":96636764160,"available":95979257856,"internally_reserved":0,"allocated":352555008,"data_stored":352387823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":26.189},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":2.5470000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":2.5750000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":26.149999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":10.975}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352174080,"data_stored":352174080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294617088,"data_stored":294617088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352518144,"data_stored":352518144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":353456128,"data_stored":353456128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:30.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:30.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7710", 'reported_seq': 7777, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:26.033088+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:26.033088+0000', 'last_peered': '2024-09-18T11:49:26.033088+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:26.033088+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7710, 'log_dups_size': 0, 'ondisk_log_size': 7710, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45383680, 'num_objects': 5540, 'num_object_clones': 0, 'num_object_copies': 22160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7710, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3422}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:30.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:49:30.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7629", 'reported_seq': 11485, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:27.315695+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:49:27.315695+0000', 'last_peered': '2024-09-18T11:49:27.315695+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:49:27.315695+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7629, 'log_dups_size': 0, 'ondisk_log_size': 7629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44455158, 'num_objects': 5427, 'num_object_clones': 0, 'num_object_copies': 21708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 209, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7627, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1897, 'num_bytes_recovered': 15507456, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7454", 'reported_seq': 7557, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:26.909088+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:26.909088+0000', 'last_peered': '2024-09-18T11:49:26.909088+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:26.909088+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7454, 'log_dups_size': 0, 'ondisk_log_size': 7454, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43858579, 'num_objects': 5355, 'num_object_clones': 0, 'num_object_copies': 21420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5355, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7453, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3310}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7560", 'reported_seq': 7636, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:26.974217+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:26.974217+0000', 'last_peered': '2024-09-18T11:49:26.974217+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:26.974217+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7560, 'log_dups_size': 0, 'ondisk_log_size': 7560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43955446, 'num_objects': 5366, 'num_object_clones': 0, 'num_object_copies': 21464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7558, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3258}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7522", 'reported_seq': 7605, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:25.969126+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:25.969126+0000', 'last_peered': '2024-09-18T11:49:25.969126+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:25.969126+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7522, 'log_dups_size': 0, 'ondisk_log_size': 7522, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44012790, 'num_objects': 5373, 'num_object_clones': 0, 'num_object_copies': 21492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2071, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7520, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3304}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7520", 'reported_seq': 7583, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:25.769491+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:25.769491+0000', 'last_peered': '2024-09-18T11:49:25.769491+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:25.769491+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7520, 'log_dups_size': 0, 'ondisk_log_size': 7520, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43139072, 'num_objects': 5266, 'num_object_clones': 0, 'num_object_copies': 21064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5266, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7520, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3121}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7640", 'reported_seq': 7724, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:27.102997+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:27.102997+0000', 'last_peered': '2024-09-18T11:49:27.102997+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:27.102997+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7640, 'log_dups_size': 0, 'ondisk_log_size': 7640, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44037366, 'num_objects': 5376, 'num_object_clones': 0, 'num_object_copies': 21504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2127, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5376, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7638, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3251}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:30.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:30.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7497", 'reported_seq': 7586, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:27.242436+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:27.242436+0000', 'last_peered': '2024-09-18T11:49:27.242436+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:27.242436+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7497, 'log_dups_size': 0, 'ondisk_log_size': 7497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44053750, 'num_objects': 5378, 'num_object_clones': 0, 'num_object_copies': 21512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2024, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7495, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3356}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:30.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:30.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7719", 'reported_seq': 7809, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:26.021223+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:49:26.021223+0000', 'last_peered': '2024-09-18T11:49:26.021223+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:26.021223+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7719, 'log_dups_size': 0, 'ondisk_log_size': 7719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44698092, 'num_objects': 5457, 'num_object_clones': 0, 'num_object_copies': 21828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7716, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3337}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:30.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:30.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7347", 'reported_seq': 7443, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:26.899524+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:26.899524+0000', 'last_peered': '2024-09-18T11:49:26.899524+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:26.899524+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7347, 'log_dups_size': 0, 'ondisk_log_size': 7347, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42980598, 'num_objects': 5247, 'num_object_clones': 0, 'num_object_copies': 20988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1972, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5247, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7345, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3277}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:30.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:30.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7571", 'reported_seq': 7664, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:26.001638+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:26.001638+0000', 'last_peered': '2024-09-18T11:49:26.001638+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:26.001638+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7571, 'log_dups_size': 0, 'ondisk_log_size': 7571, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44740465, 'num_objects': 5462, 'num_object_clones': 0, 'num_object_copies': 21848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7568, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3416}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:30.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:30.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7598", 'reported_seq': 7661, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:26.030924+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:26.030924+0000', 'last_peered': '2024-09-18T11:49:26.030924+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:26.030924+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7598, 'log_dups_size': 0, 'ondisk_log_size': 7598, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44505723, 'num_objects': 5433, 'num_object_clones': 0, 'num_object_copies': 21732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2098, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7597, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3336}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:30.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:30.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7530", 'reported_seq': 7614, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:26.699934+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:26.699934+0000', 'last_peered': '2024-09-18T11:49:26.699934+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:26.699934+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7530, 'log_dups_size': 0, 'ondisk_log_size': 7530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44538491, 'num_objects': 5437, 'num_object_clones': 0, 'num_object_copies': 21748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2052, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7529, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3386}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:30.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:30.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7673", 'reported_seq': 7760, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:26.000798+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:26.000798+0000', 'last_peered': '2024-09-18T11:49:26.000798+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:26.000798+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7673, 'log_dups_size': 0, 'ondisk_log_size': 7673, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44373238, 'num_objects': 5417, 'num_object_clones': 0, 'num_object_copies': 21668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7671, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:30.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:30.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7440", 'reported_seq': 7515, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:27.205246+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:27.205246+0000', 'last_peered': '2024-09-18T11:49:27.205246+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:27.205246+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7440, 'log_dups_size': 0, 'ondisk_log_size': 7440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43144438, 'num_objects': 5267, 'num_object_clones': 0, 'num_object_copies': 21068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7438, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3260}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:30.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:30.830 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:30.830 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29295/344700 objects degraded (8.499%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":368,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":86175,"data_bytes":705897868,"bytes_used":2582556672,"bytes_avail":383964499968,"bytes_total":386547056640,"degraded_objects":29295,"degraded_total":344700,"degraded_ratio":0.08498694516971278,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":122829,"recovering_keys_per_sec":0,"num_objects_recovered":192,"num_bytes_recovered":1474560,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":194},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:30.852 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:30.853 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:31.411 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:31.411 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:31.444 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-18T11:49:29.949018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":705897868,"num_objects":86175,"num_object_clones":0,"num_object_copies":344700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120912,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":32636928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120939,"ondisk_log_size":120939,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2522028,"kb_used_data":1322884,"kb_used_omap":14,"kb_used_meta":1199089,"kb_avail":374965332,"statfs":{"total":386547056640,"available":383964499968,"internally_reserved":0,"allocated":1354633216,"data_stored":1353964476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227867163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,6,5,12,5,0,0,0,0,0,2,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19136512,"num_objects":-2336,"num_object_clones":0,"num_object_copies":-9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2336,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004907"},"pg_stats":[{"pgid":"1.f","version":"33'7710","reported_seq":7777,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.033088+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:26.033088+0000","last_peered":"2024-09-18T11:49:26.033088+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:26.033088+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45383680,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7710,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7629","reported_seq":11485,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.315695+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:27.315695+0000","last_peered":"2024-09-18T11:49:27.315695+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:27.315695+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7629,"log_dups_size":0,"ondisk_log_size":7629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44455158,"num_objects":5427,"num_object_clones":0,"num_object_copies":21708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1897,"num_bytes_recovered":15507456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7454","reported_seq":7557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.909088+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:26.909088+0000","last_peered":"2024-09-18T11:49:26.909088+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:26.909088+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43858579,"num_objects":5355,"num_object_clones":0,"num_object_copies":21420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5355,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7453,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7560","reported_seq":7636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.974217+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:26.974217+0000","last_peered":"2024-09-18T11:49:26.974217+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:26.974217+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43955446,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7522","reported_seq":7605,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:25.969126+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:25.969126+0000","last_peered":"2024-09-18T11:49:25.969126+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:25.969126+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7522,"log_dups_size":0,"ondisk_log_size":7522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44012790,"num_objects":5373,"num_object_clones":0,"num_object_copies":21492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7520","reported_seq":7583,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:25.769491+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:25.769491+0000","last_peered":"2024-09-18T11:49:25.769491+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:25.769491+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7520,"log_dups_size":0,"ondisk_log_size":7520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43139072,"num_objects":5266,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7640","reported_seq":7724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.102997+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:27.102997+0000","last_peered":"2024-09-18T11:49:27.102997+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:27.102997+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7640,"log_dups_size":0,"ondisk_log_size":7640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44037366,"num_objects":5376,"num_object_clones":0,"num_object_copies":21504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7638,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3251}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7497","reported_seq":7586,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.242436+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:27.242436+0000","last_peered":"2024-09-18T11:49:27.242436+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:27.242436+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44053750,"num_objects":5378,"num_object_clones":0,"num_object_copies":21512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7495,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7719","reported_seq":7809,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.021223+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:26.021223+0000","last_peered":"2024-09-18T11:49:26.021223+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:26.021223+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44698092,"num_objects":5457,"num_object_clones":0,"num_object_copies":21828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7716,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7347","reported_seq":7443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.899524+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:26.899524+0000","last_peered":"2024-09-18T11:49:26.899524+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:26.899524+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42980598,"num_objects":5247,"num_object_clones":0,"num_object_copies":20988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7345,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7571","reported_seq":7664,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.001638+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:26.001638+0000","last_peered":"2024-09-18T11:49:26.001638+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:26.001638+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7571,"log_dups_size":0,"ondisk_log_size":7571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44740465,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7568,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7598","reported_seq":7661,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.030924+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:26.030924+0000","last_peered":"2024-09-18T11:49:26.030924+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:26.030924+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7598,"log_dups_size":0,"ondisk_log_size":7598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44505723,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7597,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3336}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7530","reported_seq":7614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.699934+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:26.699934+0000","last_peered":"2024-09-18T11:49:26.699934+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:26.699934+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7530,"log_dups_size":0,"ondisk_log_size":7530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44538491,"num_objects":5437,"num_object_clones":0,"num_object_copies":21748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7529,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7673","reported_seq":7760,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:26.000798+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:26.000798+0000","last_peered":"2024-09-18T11:49:26.000798+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:26.000798+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44373238,"num_objects":5417,"num_object_clones":0,"num_object_copies":21668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7529","reported_seq":11814,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:26.034315+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:26.034315+0000","last_peered":"2024-09-18T11:49:26.034315+0000","last_clean":"2024-09-18T11:49:26.034315+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:26.034315+0000","last_undegraded":"2024-09-18T11:49:26.034315+0000","last_fullsized":"2024-09-18T11:49:26.034315+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44020982,"num_objects":5374,"num_object_clones":0,"num_object_copies":21496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7440","reported_seq":7515,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:27.205246+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:27.205246+0000","last_peered":"2024-09-18T11:49:27.205246+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:27.205246+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7440,"log_dups_size":0,"ondisk_log_size":7440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43144438,"num_objects":5267,"num_object_clones":0,"num_object_copies":21068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7438,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":705897868,"num_objects":86175,"num_object_clones":0,"num_object_copies":344700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120912,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":32636928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1352765440,"data_stored":1352765440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120939,"ondisk_log_size":120939,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651852,"kb_used_data":345664,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93719988,"statfs":{"total":96636764160,"available":95969267712,"internally_reserved":0,"allocated":353959936,"data_stored":353792751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,1,5,2,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":19.792000000000002},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":19.847999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":6.7329999999999997},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":10.821999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":12.084},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.94499999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642228,"kb_used_data":344876,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93729612,"statfs":{"total":96636764160,"available":95979122688,"internally_reserved":0,"allocated":353153024,"data_stored":352985839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.68600000000000005}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":17.045999999999999},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":16.968}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":17.001000000000001},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":7.4939999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585852,"kb_used_data":288052,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93785988,"statfs":{"total":96636764160,"available":96036851712,"internally_reserved":0,"allocated":294965248,"data_stored":294798063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.9780000000000002},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.77700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":1.8919999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.0609999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":1.8540000000000001},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.9380000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642096,"kb_used_data":344292,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93729744,"statfs":{"total":96636764160,"available":95979257856,"internally_reserved":0,"allocated":352555008,"data_stored":352387823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":26.189},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":2.5470000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":2.5750000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":26.149999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":10.975}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352174080,"data_stored":352174080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294617088,"data_stored":294617088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352518144,"data_stored":352518144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":353456128,"data_stored":353456128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:34.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:35.036 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:35.038 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:35.068 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-18T11:49:33.950601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":696042892,"num_objects":84972,"num_object_clones":0,"num_object_copies":339888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122115,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4256,"num_bytes_recovered":33316864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122142,"ondisk_log_size":122142,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2524120,"kb_used_data":1308328,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374963240,"statfs":{"total":386547056640,"available":383962357760,"internally_reserved":0,"allocated":1339727872,"data_stored":1339059132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,5,8,9,4,0,0,0,1,0,0,2,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23846912,"num_objects":-2911,"num_object_clones":0,"num_object_copies":-11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2911,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004825"},"pg_stats":[{"pgid":"1.f","version":"33'7798","reported_seq":7865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.046507+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:31.046507+0000","last_peered":"2024-09-18T11:49:31.046507+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:31.046507+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44662784,"num_objects":5452,"num_object_clones":0,"num_object_copies":21808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7710","reported_seq":11732,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.323422+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:32.323422+0000","last_peered":"2024-09-18T11:49:32.323422+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:32.323422+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43791606,"num_objects":5346,"num_object_clones":0,"num_object_copies":21384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7708,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":16187392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7543","reported_seq":7652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.935391+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:31.935391+0000","last_peered":"2024-09-18T11:49:31.935391+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:31.935391+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43129491,"num_objects":5266,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7542,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7632","reported_seq":7708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.236307+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:32.236307+0000","last_peered":"2024-09-18T11:49:32.236307+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:32.236307+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7632,"log_dups_size":0,"ondisk_log_size":7632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43365622,"num_objects":5294,"num_object_clones":0,"num_object_copies":21176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7630,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7606","reported_seq":7689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.932121+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:30.932121+0000","last_peered":"2024-09-18T11:49:30.932121+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:30.932121+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7606,"log_dups_size":0,"ondisk_log_size":7606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43324662,"num_objects":5289,"num_object_clones":0,"num_object_copies":21156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7600","reported_seq":7662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.077682+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:31.077682+0000","last_peered":"2024-09-18T11:49:31.077682+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:31.077682+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7600,"log_dups_size":0,"ondisk_log_size":7600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42483712,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7600,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7719","reported_seq":7808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.232407+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:32.232407+0000","last_peered":"2024-09-18T11:49:32.232407+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:32.232407+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43390198,"num_objects":5297,"num_object_clones":0,"num_object_copies":21188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3174}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7557","reported_seq":7652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.286259+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:32.286259+0000","last_peered":"2024-09-18T11:49:32.286259+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:32.286259+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7557,"log_dups_size":0,"ondisk_log_size":7557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43562230,"num_objects":5318,"num_object_clones":0,"num_object_copies":21272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7781","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.981878+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:30.981878+0000","last_peered":"2024-09-18T11:49:30.981878+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:30.981878+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7781,"log_dups_size":0,"ondisk_log_size":7781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44190188,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7425","reported_seq":7529,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.919258+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:31.919258+0000","last_peered":"2024-09-18T11:49:31.919258+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:31.919258+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7425,"log_dups_size":0,"ondisk_log_size":7425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42341622,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7648","reported_seq":7741,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.030474+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:31.030474+0000","last_peered":"2024-09-18T11:49:31.030474+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:31.030474+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44109681,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7670","reported_seq":7733,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.875167+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:30.875167+0000","last_peered":"2024-09-18T11:49:30.875167+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:30.875167+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43915899,"num_objects":5361,"num_object_clones":0,"num_object_copies":21444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7669,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3264}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7602","reported_seq":7686,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.829568+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:31.829568+0000","last_peered":"2024-09-18T11:49:31.829568+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:31.829568+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43948667,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7601,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3314}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7724","reported_seq":7811,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.892207+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:30.892207+0000","last_peered":"2024-09-18T11:49:30.892207+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:30.892207+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7724,"log_dups_size":0,"ondisk_log_size":7724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43955446,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3226}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7618","reported_seq":11903,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:30.819250+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:30.819250+0000","last_peered":"2024-09-18T11:49:30.819250+0000","last_clean":"2024-09-18T11:49:30.819250+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:30.819250+0000","last_undegraded":"2024-09-18T11:49:30.819250+0000","last_fullsized":"2024-09-18T11:49:30.819250+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7618,"log_dups_size":0,"ondisk_log_size":7618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43291894,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7616,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7509","reported_seq":7585,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.274642+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:32.274642+0000","last_peered":"2024-09-18T11:49:32.274642+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:32.274642+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7509,"log_dups_size":0,"ondisk_log_size":7509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42579190,"num_objects":5198,"num_object_clones":0,"num_object_copies":20792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7507,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":696042892,"num_objects":84972,"num_object_clones":0,"num_object_copies":339888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122115,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4256,"num_bytes_recovered":33316864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1337761792,"data_stored":1337761792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122142,"ondisk_log_size":122142,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651224,"kb_used_data":340880,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93720616,"statfs":{"total":96636764160,"available":95969910784,"internally_reserved":0,"allocated":349061120,"data_stored":348893935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1,2,0,0,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":4.9470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":5.4329999999999998},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.002}]}]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641688,"kb_used_data":340160,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93730152,"statfs":{"total":96636764160,"available":95979675648,"internally_reserved":0,"allocated":348323840,"data_stored":348156655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,5,1,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":11.199999999999999},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":11.221}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.6360000000000001},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":11.157999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":11.348000000000001},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.8650000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589656,"kb_used_data":287696,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93782184,"statfs":{"total":96636764160,"available":96032956416,"internally_reserved":0,"allocated":294600704,"data_stored":294433519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":15.220000000000001},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":5.1310000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.5449999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":1.097}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.247}]}]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641552,"kb_used_data":339592,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93730288,"statfs":{"total":96636764160,"available":95979814912,"internally_reserved":0,"allocated":347742208,"data_stored":347575023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":0.96099999999999997},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":13.901999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":6.1310000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.7930000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":5.9619999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347295744,"data_stored":347295744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294305792,"data_stored":294305792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":347652096,"data_stored":347652096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":348508160,"data_stored":348508160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:35.069 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:35.597 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:35.598 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:35.622 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-18T11:49:33.950601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":696042892,"num_objects":84972,"num_object_clones":0,"num_object_copies":339888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122115,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4256,"num_bytes_recovered":33316864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122142,"ondisk_log_size":122142,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2524120,"kb_used_data":1308328,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374963240,"statfs":{"total":386547056640,"available":383962357760,"internally_reserved":0,"allocated":1339727872,"data_stored":1339059132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,5,8,9,4,0,0,0,1,0,0,2,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23846912,"num_objects":-2911,"num_object_clones":0,"num_object_copies":-11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2911,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004825"},"pg_stats":[{"pgid":"1.f","version":"33'7798","reported_seq":7865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.046507+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:31.046507+0000","last_peered":"2024-09-18T11:49:31.046507+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:31.046507+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44662784,"num_objects":5452,"num_object_clones":0,"num_object_copies":21808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7710","reported_seq":11732,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.323422+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:32.323422+0000","last_peered":"2024-09-18T11:49:32.323422+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:32.323422+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43791606,"num_objects":5346,"num_object_clones":0,"num_object_copies":21384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7708,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":16187392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7543","reported_seq":7652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.935391+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:31.935391+0000","last_peered":"2024-09-18T11:49:31.935391+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:31.935391+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43129491,"num_objects":5266,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7542,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7632","reported_seq":7708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.236307+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:32.236307+0000","last_peered":"2024-09-18T11:49:32.236307+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:32.236307+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7632,"log_dups_size":0,"ondisk_log_size":7632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43365622,"num_objects":5294,"num_object_clones":0,"num_object_copies":21176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7630,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7606","reported_seq":7689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.932121+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:30.932121+0000","last_peered":"2024-09-18T11:49:30.932121+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:30.932121+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7606,"log_dups_size":0,"ondisk_log_size":7606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43324662,"num_objects":5289,"num_object_clones":0,"num_object_copies":21156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7600","reported_seq":7662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.077682+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:31.077682+0000","last_peered":"2024-09-18T11:49:31.077682+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:31.077682+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7600,"log_dups_size":0,"ondisk_log_size":7600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42483712,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7600,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7719","reported_seq":7808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.232407+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:32.232407+0000","last_peered":"2024-09-18T11:49:32.232407+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:32.232407+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43390198,"num_objects":5297,"num_object_clones":0,"num_object_copies":21188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3174}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7557","reported_seq":7652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.286259+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:32.286259+0000","last_peered":"2024-09-18T11:49:32.286259+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:32.286259+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7557,"log_dups_size":0,"ondisk_log_size":7557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43562230,"num_objects":5318,"num_object_clones":0,"num_object_copies":21272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7781","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.981878+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:30.981878+0000","last_peered":"2024-09-18T11:49:30.981878+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:30.981878+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7781,"log_dups_size":0,"ondisk_log_size":7781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44190188,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7425","reported_seq":7529,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.919258+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:31.919258+0000","last_peered":"2024-09-18T11:49:31.919258+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:31.919258+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7425,"log_dups_size":0,"ondisk_log_size":7425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42341622,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7648","reported_seq":7741,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.030474+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:31.030474+0000","last_peered":"2024-09-18T11:49:31.030474+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:31.030474+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44109681,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7670","reported_seq":7733,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.875167+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:30.875167+0000","last_peered":"2024-09-18T11:49:30.875167+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:30.875167+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43915899,"num_objects":5361,"num_object_clones":0,"num_object_copies":21444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7669,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3264}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7602","reported_seq":7686,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.829568+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:31.829568+0000","last_peered":"2024-09-18T11:49:31.829568+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:31.829568+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43948667,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7601,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3314}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7724","reported_seq":7811,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.892207+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:30.892207+0000","last_peered":"2024-09-18T11:49:30.892207+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:30.892207+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7724,"log_dups_size":0,"ondisk_log_size":7724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43955446,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3226}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7618","reported_seq":11903,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:30.819250+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:30.819250+0000","last_peered":"2024-09-18T11:49:30.819250+0000","last_clean":"2024-09-18T11:49:30.819250+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:30.819250+0000","last_undegraded":"2024-09-18T11:49:30.819250+0000","last_fullsized":"2024-09-18T11:49:30.819250+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7618,"log_dups_size":0,"ondisk_log_size":7618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43291894,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7616,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7509","reported_seq":7585,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.274642+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:32.274642+0000","last_peered":"2024-09-18T11:49:32.274642+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:32.274642+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7509,"log_dups_size":0,"ondisk_log_size":7509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42579190,"num_objects":5198,"num_object_clones":0,"num_object_copies":20792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7507,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":696042892,"num_objects":84972,"num_object_clones":0,"num_object_copies":339888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122115,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4256,"num_bytes_recovered":33316864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1337761792,"data_stored":1337761792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122142,"ondisk_log_size":122142,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651224,"kb_used_data":340880,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93720616,"statfs":{"total":96636764160,"available":95969910784,"internally_reserved":0,"allocated":349061120,"data_stored":348893935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1,2,0,0,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":4.9470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":5.4329999999999998},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.002}]}]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641688,"kb_used_data":340160,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93730152,"statfs":{"total":96636764160,"available":95979675648,"internally_reserved":0,"allocated":348323840,"data_stored":348156655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,5,1,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":11.199999999999999},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":11.221}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.6360000000000001},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":11.157999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":11.348000000000001},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.8650000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589656,"kb_used_data":287696,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93782184,"statfs":{"total":96636764160,"available":96032956416,"internally_reserved":0,"allocated":294600704,"data_stored":294433519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":15.220000000000001},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":5.1310000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.5449999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":1.097}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.247}]}]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641552,"kb_used_data":339592,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93730288,"statfs":{"total":96636764160,"available":95979814912,"internally_reserved":0,"allocated":347742208,"data_stored":347575023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":0.96099999999999997},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":13.901999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":6.1310000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.7930000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":5.9619999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347295744,"data_stored":347295744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294305792,"data_stored":294305792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":347652096,"data_stored":347652096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":348508160,"data_stored":348508160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:35.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:35.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7798", 'reported_seq': 7865, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:31.046507+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:31.046507+0000', 'last_peered': '2024-09-18T11:49:31.046507+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:31.046507+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7798, 'log_dups_size': 0, 'ondisk_log_size': 7798, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44662784, 'num_objects': 5452, 'num_object_clones': 0, 'num_object_copies': 21808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7798, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3334}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:35.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:49:35.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7710", 'reported_seq': 11732, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:32.323422+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:49:32.323422+0000', 'last_peered': '2024-09-18T11:49:32.323422+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:49:32.323422+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7710, 'log_dups_size': 0, 'ondisk_log_size': 7710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43791606, 'num_objects': 5346, 'num_object_clones': 0, 'num_object_copies': 21384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 126, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7708, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1980, 'num_bytes_recovered': 16187392, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:35.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:35.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7543", 'reported_seq': 7652, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:31.935391+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:31.935391+0000', 'last_peered': '2024-09-18T11:49:31.935391+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:31.935391+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7543, 'log_dups_size': 0, 'ondisk_log_size': 7543, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43129491, 'num_objects': 5266, 'num_object_clones': 0, 'num_object_copies': 21064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5266, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7542, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3224}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:35.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:35.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7632", 'reported_seq': 7708, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:32.236307+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:32.236307+0000', 'last_peered': '2024-09-18T11:49:32.236307+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:32.236307+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7632, 'log_dups_size': 0, 'ondisk_log_size': 7632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43365622, 'num_objects': 5294, 'num_object_clones': 0, 'num_object_copies': 21176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7630, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:35.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:35.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7606", 'reported_seq': 7689, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:30.932121+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:30.932121+0000', 'last_peered': '2024-09-18T11:49:30.932121+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:30.932121+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7606, 'log_dups_size': 0, 'ondisk_log_size': 7606, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43324662, 'num_objects': 5289, 'num_object_clones': 0, 'num_object_copies': 21156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2071, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7604, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3220}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:35.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:35.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7600", 'reported_seq': 7662, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:31.077682+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:31.077682+0000', 'last_peered': '2024-09-18T11:49:31.077682+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:31.077682+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7600, 'log_dups_size': 0, 'ondisk_log_size': 7600, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42483712, 'num_objects': 5186, 'num_object_clones': 0, 'num_object_copies': 20744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7600, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3041}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:35.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:35.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7719", 'reported_seq': 7808, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:32.232407+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:32.232407+0000', 'last_peered': '2024-09-18T11:49:32.232407+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:32.232407+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7719, 'log_dups_size': 0, 'ondisk_log_size': 7719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43390198, 'num_objects': 5297, 'num_object_clones': 0, 'num_object_copies': 21188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5297, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7717, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3174}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7557", 'reported_seq': 7652, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:32.286259+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:32.286259+0000', 'last_peered': '2024-09-18T11:49:32.286259+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:32.286259+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7557, 'log_dups_size': 0, 'ondisk_log_size': 7557, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43562230, 'num_objects': 5318, 'num_object_clones': 0, 'num_object_copies': 21272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2021, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7555, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7781", 'reported_seq': 7871, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:30.981878+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:49:30.981878+0000', 'last_peered': '2024-09-18T11:49:30.981878+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:30.981878+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7781, 'log_dups_size': 0, 'ondisk_log_size': 7781, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44190188, 'num_objects': 5395, 'num_object_clones': 0, 'num_object_copies': 21580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7778, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3275}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7425", 'reported_seq': 7529, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:31.919258+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:31.919258+0000', 'last_peered': '2024-09-18T11:49:31.919258+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:31.919258+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7425, 'log_dups_size': 0, 'ondisk_log_size': 7425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42341622, 'num_objects': 5169, 'num_object_clones': 0, 'num_object_copies': 20676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1968, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7423, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3203}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7648", 'reported_seq': 7741, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:31.030474+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:31.030474+0000', 'last_peered': '2024-09-18T11:49:31.030474+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:31.030474+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7648, 'log_dups_size': 0, 'ondisk_log_size': 7648, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44109681, 'num_objects': 5385, 'num_object_clones': 0, 'num_object_copies': 21540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7645, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3339}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7670", 'reported_seq': 7733, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:30.875167+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:30.875167+0000', 'last_peered': '2024-09-18T11:49:30.875167+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:30.875167+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7670, 'log_dups_size': 0, 'ondisk_log_size': 7670, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43915899, 'num_objects': 5361, 'num_object_clones': 0, 'num_object_copies': 21444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2098, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7669, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7602", 'reported_seq': 7686, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:31.829568+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:31.829568+0000', 'last_peered': '2024-09-18T11:49:31.829568+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:31.829568+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7602, 'log_dups_size': 0, 'ondisk_log_size': 7602, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43948667, 'num_objects': 5365, 'num_object_clones': 0, 'num_object_copies': 21460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2052, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7601, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7724", 'reported_seq': 7811, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:30.892207+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:30.892207+0000', 'last_peered': '2024-09-18T11:49:30.892207+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:30.892207+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7724, 'log_dups_size': 0, 'ondisk_log_size': 7724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43955446, 'num_objects': 5366, 'num_object_clones': 0, 'num_object_copies': 21464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7722, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7509", 'reported_seq': 7585, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:32.274642+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:32.274642+0000', 'last_peered': '2024-09-18T11:49:32.274642+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:32.274642+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7509, 'log_dups_size': 0, 'ondisk_log_size': 7509, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42579190, 'num_objects': 5198, 'num_object_clones': 0, 'num_object_copies': 20792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7507, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3191}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:35.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:36.364 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:36.364 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29200/339888 objects degraded (8.591%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":374,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84972,"data_bytes":696042892,"bytes_used":2584698880,"bytes_avail":383962357760,"bytes_total":386547056640,"degraded_objects":29200,"degraded_total":339888,"degraded_ratio":0.085910652920962199,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":118736,"recovering_keys_per_sec":0,"num_objects_recovered":191,"num_bytes_recovered":1425408,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":194},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:36.382 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:36.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:36.917 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:36.917 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:36.936 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-18T11:49:35.951453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":696042892,"num_objects":84972,"num_object_clones":0,"num_object_copies":339888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122115,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4256,"num_bytes_recovered":33316864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122142,"ondisk_log_size":122142,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2524120,"kb_used_data":1308328,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374963240,"statfs":{"total":386547056640,"available":383962357760,"internally_reserved":0,"allocated":1339727872,"data_stored":1339059132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,5,8,9,4,0,0,0,1,0,0,2,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19152896,"num_objects":-2338,"num_object_clones":0,"num_object_copies":-9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1425408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004823"},"pg_stats":[{"pgid":"1.f","version":"33'7798","reported_seq":7865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.046507+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:31.046507+0000","last_peered":"2024-09-18T11:49:31.046507+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:31.046507+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44662784,"num_objects":5452,"num_object_clones":0,"num_object_copies":21808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7710","reported_seq":11732,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.323422+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:32.323422+0000","last_peered":"2024-09-18T11:49:32.323422+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:32.323422+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43791606,"num_objects":5346,"num_object_clones":0,"num_object_copies":21384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7708,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":16187392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7543","reported_seq":7652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.935391+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:31.935391+0000","last_peered":"2024-09-18T11:49:31.935391+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:31.935391+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43129491,"num_objects":5266,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7542,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7632","reported_seq":7708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.236307+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:32.236307+0000","last_peered":"2024-09-18T11:49:32.236307+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:32.236307+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7632,"log_dups_size":0,"ondisk_log_size":7632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43365622,"num_objects":5294,"num_object_clones":0,"num_object_copies":21176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7630,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7606","reported_seq":7689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.932121+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:30.932121+0000","last_peered":"2024-09-18T11:49:30.932121+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:30.932121+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7606,"log_dups_size":0,"ondisk_log_size":7606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43324662,"num_objects":5289,"num_object_clones":0,"num_object_copies":21156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7600","reported_seq":7662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.077682+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:31.077682+0000","last_peered":"2024-09-18T11:49:31.077682+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:31.077682+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7600,"log_dups_size":0,"ondisk_log_size":7600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42483712,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7600,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7719","reported_seq":7808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.232407+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:32.232407+0000","last_peered":"2024-09-18T11:49:32.232407+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:32.232407+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43390198,"num_objects":5297,"num_object_clones":0,"num_object_copies":21188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3174}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7557","reported_seq":7652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.286259+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:32.286259+0000","last_peered":"2024-09-18T11:49:32.286259+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:32.286259+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7557,"log_dups_size":0,"ondisk_log_size":7557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43562230,"num_objects":5318,"num_object_clones":0,"num_object_copies":21272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7781","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.981878+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:30.981878+0000","last_peered":"2024-09-18T11:49:30.981878+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:30.981878+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7781,"log_dups_size":0,"ondisk_log_size":7781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44190188,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7425","reported_seq":7529,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.919258+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:31.919258+0000","last_peered":"2024-09-18T11:49:31.919258+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:31.919258+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7425,"log_dups_size":0,"ondisk_log_size":7425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42341622,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7648","reported_seq":7741,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.030474+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:31.030474+0000","last_peered":"2024-09-18T11:49:31.030474+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:31.030474+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44109681,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7670","reported_seq":7733,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.875167+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:30.875167+0000","last_peered":"2024-09-18T11:49:30.875167+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:30.875167+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43915899,"num_objects":5361,"num_object_clones":0,"num_object_copies":21444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7669,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3264}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7602","reported_seq":7686,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:31.829568+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:31.829568+0000","last_peered":"2024-09-18T11:49:31.829568+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:31.829568+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43948667,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7601,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3314}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7724","reported_seq":7811,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:30.892207+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:30.892207+0000","last_peered":"2024-09-18T11:49:30.892207+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:30.892207+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7724,"log_dups_size":0,"ondisk_log_size":7724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43955446,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3226}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7618","reported_seq":11903,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:30.819250+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:30.819250+0000","last_peered":"2024-09-18T11:49:30.819250+0000","last_clean":"2024-09-18T11:49:30.819250+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:30.819250+0000","last_undegraded":"2024-09-18T11:49:30.819250+0000","last_fullsized":"2024-09-18T11:49:30.819250+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7618,"log_dups_size":0,"ondisk_log_size":7618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43291894,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7616,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7509","reported_seq":7585,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:32.274642+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:32.274642+0000","last_peered":"2024-09-18T11:49:32.274642+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:32.274642+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7509,"log_dups_size":0,"ondisk_log_size":7509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42579190,"num_objects":5198,"num_object_clones":0,"num_object_copies":20792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7507,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":696042892,"num_objects":84972,"num_object_clones":0,"num_object_copies":339888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122115,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4256,"num_bytes_recovered":33316864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1337761792,"data_stored":1337761792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122142,"ondisk_log_size":122142,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651224,"kb_used_data":340880,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93720616,"statfs":{"total":96636764160,"available":95969910784,"internally_reserved":0,"allocated":349061120,"data_stored":348893935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1,2,0,0,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":4.9470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":5.4329999999999998},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.002}]}]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641688,"kb_used_data":340160,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93730152,"statfs":{"total":96636764160,"available":95979675648,"internally_reserved":0,"allocated":348323840,"data_stored":348156655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,5,1,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":11.199999999999999},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":11.221}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.6360000000000001},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":11.157999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":11.348000000000001},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.8650000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589656,"kb_used_data":287696,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93782184,"statfs":{"total":96636764160,"available":96032956416,"internally_reserved":0,"allocated":294600704,"data_stored":294433519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":15.220000000000001},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":5.1310000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":5.5449999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":1.097}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.247}]}]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641552,"kb_used_data":339592,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93730288,"statfs":{"total":96636764160,"available":95979814912,"internally_reserved":0,"allocated":347742208,"data_stored":347575023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":0.96099999999999997},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":13.901999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":6.1310000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.7930000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":5.9619999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347295744,"data_stored":347295744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294305792,"data_stored":294305792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":347652096,"data_stored":347652096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":348508160,"data_stored":348508160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:39.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:40.488 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:40.490 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:40.511 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-18T11:49:39.952882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":686343564,"num_objects":83788,"num_object_clones":0,"num_object_copies":335152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83788,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123299,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4345,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123326,"ondisk_log_size":123326,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2509780,"kb_used_data":1293976,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374977580,"statfs":{"total":386547056640,"available":383977041920,"internally_reserved":0,"allocated":1325031424,"data_stored":1324362684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,4,15,6,8,0,0,0,1,0,0,1,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19554304,"num_objects":-2387,"num_object_clones":0,"num_object_copies":-9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2387,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004686"},"pg_stats":[{"pgid":"1.f","version":"33'7867","reported_seq":7940,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.001315+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:36.001315+0000","last_peered":"2024-09-18T11:49:36.001315+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:36.001315+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7867,"log_dups_size":0,"ondisk_log_size":7867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44097536,"num_objects":5383,"num_object_clones":0,"num_object_copies":21532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7867,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7784","reported_seq":11959,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.261299+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:37.261299+0000","last_peered":"2024-09-18T11:49:37.261299+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:37.261299+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185398,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7782,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7630","reported_seq":7739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.891227+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:36.891227+0000","last_peered":"2024-09-18T11:49:36.891227+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:36.891227+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7630,"log_dups_size":0,"ondisk_log_size":7630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42416787,"num_objects":5179,"num_object_clones":0,"num_object_copies":20716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7629,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3137}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7697","reported_seq":7773,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.148272+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:37.148272+0000","last_peered":"2024-09-18T11:49:37.148272+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:37.148272+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42833142,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7673","reported_seq":7758,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.006678+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:36.006678+0000","last_peered":"2024-09-18T11:49:36.006678+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:36.006678+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42775798,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3154}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7678","reported_seq":7750,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.103577+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:36.103577+0000","last_peered":"2024-09-18T11:49:36.103577+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:36.103577+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7678,"log_dups_size":0,"ondisk_log_size":7678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41844736,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7678,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7798","reported_seq":7887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.307893+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:37.307893+0000","last_peered":"2024-09-18T11:49:37.307893+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:37.307893+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42743030,"num_objects":5218,"num_object_clones":0,"num_object_copies":20872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3095}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7631","reported_seq":7726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.261979+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:37.261979+0000","last_peered":"2024-09-18T11:49:37.261979+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:37.261979+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42956022,"num_objects":5244,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7629,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7849","reported_seq":7943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.061542+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:36.061542+0000","last_peered":"2024-09-18T11:49:36.061542+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:36.061542+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43633132,"num_objects":5327,"num_object_clones":0,"num_object_copies":21308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7490","reported_seq":7593,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.948333+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:36.948333+0000","last_peered":"2024-09-18T11:49:36.948333+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:36.948333+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41809142,"num_objects":5104,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3138}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7729","reported_seq":7820,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.079030+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:36.079030+0000","last_peered":"2024-09-18T11:49:36.079030+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:36.079030+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43446129,"num_objects":5304,"num_object_clones":0,"num_object_copies":21216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7726,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7751","reported_seq":7814,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.054689+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:36.054689+0000","last_peered":"2024-09-18T11:49:36.054689+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:36.054689+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7751,"log_dups_size":0,"ondisk_log_size":7751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43252347,"num_objects":5280,"num_object_clones":0,"num_object_copies":21120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7750,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7669","reported_seq":7753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.875694+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:36.875694+0000","last_peered":"2024-09-18T11:49:36.875694+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:36.875694+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43399803,"num_objects":5298,"num_object_clones":0,"num_object_copies":21192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7668,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3247}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7817","reported_seq":7905,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.118538+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:36.118538+0000","last_peered":"2024-09-18T11:49:36.118538+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:36.118538+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7817,"log_dups_size":0,"ondisk_log_size":7817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43193590,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7815,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3134}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7690","reported_seq":11975,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:36.021400+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:36.021400+0000","last_peered":"2024-09-18T11:49:36.021400+0000","last_clean":"2024-09-18T11:49:36.021400+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:36.021400+0000","last_undegraded":"2024-09-18T11:49:36.021400+0000","last_fullsized":"2024-09-18T11:49:36.021400+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7690,"log_dups_size":0,"ondisk_log_size":7690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42702070,"num_objects":5213,"num_object_clones":0,"num_object_copies":20852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7688,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7573","reported_seq":7649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.028816+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:37.028816+0000","last_peered":"2024-09-18T11:49:37.028816+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:37.028816+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42054902,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7571,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3127}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":686343564,"num_objects":83788,"num_object_clones":0,"num_object_copies":335152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83788,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123299,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4345,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1322946560,"data_stored":1322946560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123326,"ondisk_log_size":123326,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634088,"kb_used_data":336216,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93737752,"statfs":{"total":96636764160,"available":95987458048,"internally_reserved":0,"allocated":344285184,"data_stored":344117999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,5,5,3,0,0,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.1840000000000002},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.0149999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.89500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":4.6539999999999999},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.86799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641084,"kb_used_data":335404,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93730756,"statfs":{"total":96636764160,"available":95980294144,"internally_reserved":0,"allocated":343453696,"data_stored":343286511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,5,1,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":11.199999999999999},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":11.221}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.6360000000000001},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":11.157999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":11.348000000000001},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.8650000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593544,"kb_used_data":287424,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93778296,"statfs":{"total":96636764160,"available":96028975104,"internally_reserved":0,"allocated":294322176,"data_stored":294154991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.9500000000000002},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":3.734}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":12.718},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.4539999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":3.7679999999999998},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":13.718}]}]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641064,"kb_used_data":334932,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93730776,"statfs":{"total":96636764160,"available":95980314624,"internally_reserved":0,"allocated":342970368,"data_stored":342803183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,0,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":0.311},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":8.0790000000000006}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":13.93},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.9459999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":5.3159999999999998},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":10.919}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":342474752,"data_stored":342474752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294019072,"data_stored":294019072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":342831104,"data_stored":342831104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343621632,"data_stored":343621632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:40.513 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:41.031 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:41.031 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:41.059 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-18T11:49:39.952882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":686343564,"num_objects":83788,"num_object_clones":0,"num_object_copies":335152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83788,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123299,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4345,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123326,"ondisk_log_size":123326,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2509780,"kb_used_data":1293976,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":374977580,"statfs":{"total":386547056640,"available":383977041920,"internally_reserved":0,"allocated":1325031424,"data_stored":1324362684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,4,15,6,8,0,0,0,1,0,0,1,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19554304,"num_objects":-2387,"num_object_clones":0,"num_object_copies":-9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2387,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004686"},"pg_stats":[{"pgid":"1.f","version":"33'7867","reported_seq":7940,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.001315+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:36.001315+0000","last_peered":"2024-09-18T11:49:36.001315+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:36.001315+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7867,"log_dups_size":0,"ondisk_log_size":7867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44097536,"num_objects":5383,"num_object_clones":0,"num_object_copies":21532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7867,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7784","reported_seq":11959,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.261299+0000","last_change":"2024-09-18T11:47:00.180576+0000","last_active":"2024-09-18T11:49:37.261299+0000","last_peered":"2024-09-18T11:49:37.261299+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:37.261299+0000","last_undegraded":"2024-09-18T11:44:47.158336+0000","last_fullsized":"2024-09-18T11:44:47.121834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185398,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7782,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":5223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7630","reported_seq":7739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.891227+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:36.891227+0000","last_peered":"2024-09-18T11:49:36.891227+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:36.891227+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7630,"log_dups_size":0,"ondisk_log_size":7630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42416787,"num_objects":5179,"num_object_clones":0,"num_object_copies":20716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7629,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3137}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7697","reported_seq":7773,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.148272+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:37.148272+0000","last_peered":"2024-09-18T11:49:37.148272+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:37.148272+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42833142,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7673","reported_seq":7758,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.006678+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:36.006678+0000","last_peered":"2024-09-18T11:49:36.006678+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:36.006678+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42775798,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3154}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7678","reported_seq":7750,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.103577+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:36.103577+0000","last_peered":"2024-09-18T11:49:36.103577+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:36.103577+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7678,"log_dups_size":0,"ondisk_log_size":7678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41844736,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7678,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7798","reported_seq":7887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.307893+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:37.307893+0000","last_peered":"2024-09-18T11:49:37.307893+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:37.307893+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42743030,"num_objects":5218,"num_object_clones":0,"num_object_copies":20872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3095}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7631","reported_seq":7726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.261979+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:37.261979+0000","last_peered":"2024-09-18T11:49:37.261979+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:37.261979+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42956022,"num_objects":5244,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7629,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7849","reported_seq":7943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.061542+0000","last_change":"2024-09-18T11:44:47.360659+0000","last_active":"2024-09-18T11:49:36.061542+0000","last_peered":"2024-09-18T11:49:36.061542+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:36.061542+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43633132,"num_objects":5327,"num_object_clones":0,"num_object_copies":21308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7490","reported_seq":7593,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.948333+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:36.948333+0000","last_peered":"2024-09-18T11:49:36.948333+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:36.948333+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41809142,"num_objects":5104,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3138}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7729","reported_seq":7820,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.079030+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:36.079030+0000","last_peered":"2024-09-18T11:49:36.079030+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:36.079030+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43446129,"num_objects":5304,"num_object_clones":0,"num_object_copies":21216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7726,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7751","reported_seq":7814,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.054689+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:36.054689+0000","last_peered":"2024-09-18T11:49:36.054689+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:36.054689+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7751,"log_dups_size":0,"ondisk_log_size":7751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43252347,"num_objects":5280,"num_object_clones":0,"num_object_copies":21120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7750,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7669","reported_seq":7753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.875694+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:36.875694+0000","last_peered":"2024-09-18T11:49:36.875694+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:36.875694+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43399803,"num_objects":5298,"num_object_clones":0,"num_object_copies":21192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7668,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3247}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7817","reported_seq":7905,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:36.118538+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:36.118538+0000","last_peered":"2024-09-18T11:49:36.118538+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:36.118538+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7817,"log_dups_size":0,"ondisk_log_size":7817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43193590,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7815,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3134}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7690","reported_seq":11975,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T11:49:36.021400+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:36.021400+0000","last_peered":"2024-09-18T11:49:36.021400+0000","last_clean":"2024-09-18T11:49:36.021400+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:36.021400+0000","last_undegraded":"2024-09-18T11:49:36.021400+0000","last_fullsized":"2024-09-18T11:49:36.021400+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7690,"log_dups_size":0,"ondisk_log_size":7690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42702070,"num_objects":5213,"num_object_clones":0,"num_object_copies":20852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7688,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7573","reported_seq":7649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.028816+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:37.028816+0000","last_peered":"2024-09-18T11:49:37.028816+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:37.028816+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42054902,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7571,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3127}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":686343564,"num_objects":83788,"num_object_clones":0,"num_object_copies":335152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83788,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123299,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4345,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1322946560,"data_stored":1322946560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123326,"ondisk_log_size":123326,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634088,"kb_used_data":336216,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93737752,"statfs":{"total":96636764160,"available":95987458048,"internally_reserved":0,"allocated":344285184,"data_stored":344117999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,5,5,3,0,0,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.1840000000000002},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.0149999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.89500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":4.6539999999999999},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.86799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641084,"kb_used_data":335404,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93730756,"statfs":{"total":96636764160,"available":95980294144,"internally_reserved":0,"allocated":343453696,"data_stored":343286511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,5,1,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":11.199999999999999},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":11.221}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.6360000000000001},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":11.157999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":11.348000000000001},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.8650000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593544,"kb_used_data":287424,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93778296,"statfs":{"total":96636764160,"available":96028975104,"internally_reserved":0,"allocated":294322176,"data_stored":294154991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.9500000000000002},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":3.734}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":12.718},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.4539999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":3.7679999999999998},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":13.718}]}]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641064,"kb_used_data":334932,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93730776,"statfs":{"total":96636764160,"available":95980314624,"internally_reserved":0,"allocated":342970368,"data_stored":342803183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,0,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":0.311},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":8.0790000000000006}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":13.93},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.9459999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":5.3159999999999998},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":10.919}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":342474752,"data_stored":342474752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294019072,"data_stored":294019072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":342831104,"data_stored":342831104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343621632,"data_stored":343621632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:41.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:41.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7867", 'reported_seq': 7940, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:36.001315+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:36.001315+0000', 'last_peered': '2024-09-18T11:49:36.001315+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:36.001315+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7867, 'log_dups_size': 0, 'ondisk_log_size': 7867, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44097536, 'num_objects': 5383, 'num_object_clones': 0, 'num_object_copies': 21532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2115, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7867, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3268}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:41.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:49:41.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7784", 'reported_seq': 11959, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:37.261299+0000', 'last_change': '2024-09-18T11:47:00.180576+0000', 'last_active': '2024-09-18T11:49:37.261299+0000', 'last_peered': '2024-09-18T11:49:37.261299+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:49:37.261299+0000', 'last_undegraded': '2024-09-18T11:44:47.158336+0000', 'last_fullsized': '2024-09-18T11:44:47.121834+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7784, 'log_dups_size': 0, 'ondisk_log_size': 7784, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43185398, 'num_objects': 5272, 'num_object_clones': 0, 'num_object_copies': 21088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 49, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7782, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2057, 'num_bytes_recovered': 16818176, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 5223}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:41.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:41.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7630", 'reported_seq': 7739, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:36.891227+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:36.891227+0000', 'last_peered': '2024-09-18T11:49:36.891227+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:36.891227+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7630, 'log_dups_size': 0, 'ondisk_log_size': 7630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42416787, 'num_objects': 5179, 'num_object_clones': 0, 'num_object_copies': 20716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5179, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7629, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3137}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:41.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:41.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7697", 'reported_seq': 7773, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:37.148272+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:37.148272+0000', 'last_peered': '2024-09-18T11:49:37.148272+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:37.148272+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7697, 'log_dups_size': 0, 'ondisk_log_size': 7697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42833142, 'num_objects': 5229, 'num_object_clones': 0, 'num_object_copies': 20916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7695, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:41.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:41.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7673", 'reported_seq': 7758, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:36.006678+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:36.006678+0000', 'last_peered': '2024-09-18T11:49:36.006678+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:36.006678+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7673, 'log_dups_size': 0, 'ondisk_log_size': 7673, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42775798, 'num_objects': 5222, 'num_object_clones': 0, 'num_object_copies': 20888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2070, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7671, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3154}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:41.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:41.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7678", 'reported_seq': 7750, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:36.103577+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:36.103577+0000', 'last_peered': '2024-09-18T11:49:36.103577+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:36.103577+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7678, 'log_dups_size': 0, 'ondisk_log_size': 7678, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41844736, 'num_objects': 5108, 'num_object_clones': 0, 'num_object_copies': 20432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7678, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2968}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:41.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:41.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7798", 'reported_seq': 7887, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:37.307893+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:37.307893+0000', 'last_peered': '2024-09-18T11:49:37.307893+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:37.307893+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7798, 'log_dups_size': 0, 'ondisk_log_size': 7798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42743030, 'num_objects': 5218, 'num_object_clones': 0, 'num_object_copies': 20872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7796, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3095}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:41.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:41.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7631", 'reported_seq': 7726, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:37.261979+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:37.261979+0000', 'last_peered': '2024-09-18T11:49:37.261979+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:37.261979+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7631, 'log_dups_size': 0, 'ondisk_log_size': 7631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42956022, 'num_objects': 5244, 'num_object_clones': 0, 'num_object_copies': 20976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2021, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7629, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3225}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:41.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:41.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7849", 'reported_seq': 7943, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:36.061542+0000', 'last_change': '2024-09-18T11:44:47.360659+0000', 'last_active': '2024-09-18T11:49:36.061542+0000', 'last_peered': '2024-09-18T11:49:36.061542+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:36.061542+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7849, 'log_dups_size': 0, 'ondisk_log_size': 7849, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43633132, 'num_objects': 5327, 'num_object_clones': 0, 'num_object_copies': 21308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7846, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3209}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:41.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:41.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7490", 'reported_seq': 7593, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:36.948333+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:36.948333+0000', 'last_peered': '2024-09-18T11:49:36.948333+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:36.948333+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7490, 'log_dups_size': 0, 'ondisk_log_size': 7490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41809142, 'num_objects': 5104, 'num_object_clones': 0, 'num_object_copies': 20416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1968, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7488, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3138}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:41.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:41.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7729", 'reported_seq': 7820, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:36.079030+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:36.079030+0000', 'last_peered': '2024-09-18T11:49:36.079030+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:36.079030+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7729, 'log_dups_size': 0, 'ondisk_log_size': 7729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43446129, 'num_objects': 5304, 'num_object_clones': 0, 'num_object_copies': 21216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7726, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3258}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:41.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:41.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7751", 'reported_seq': 7814, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:36.054689+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:36.054689+0000', 'last_peered': '2024-09-18T11:49:36.054689+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:36.054689+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7751, 'log_dups_size': 0, 'ondisk_log_size': 7751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43252347, 'num_objects': 5280, 'num_object_clones': 0, 'num_object_copies': 21120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2098, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7750, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3183}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:41.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:41.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7669", 'reported_seq': 7753, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:36.875694+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:36.875694+0000', 'last_peered': '2024-09-18T11:49:36.875694+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:36.875694+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7669, 'log_dups_size': 0, 'ondisk_log_size': 7669, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43399803, 'num_objects': 5298, 'num_object_clones': 0, 'num_object_copies': 21192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2052, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7668, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3247}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:41.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:41.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7817", 'reported_seq': 7905, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:36.118538+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:36.118538+0000', 'last_peered': '2024-09-18T11:49:36.118538+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:36.118538+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7817, 'log_dups_size': 0, 'ondisk_log_size': 7817, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43193590, 'num_objects': 5273, 'num_object_clones': 0, 'num_object_copies': 21092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2141, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7815, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:41.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:41.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7573", 'reported_seq': 7649, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:37.028816+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:37.028816+0000', 'last_peered': '2024-09-18T11:49:37.028816+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:37.028816+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7573, 'log_dups_size': 0, 'ondisk_log_size': 7573, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42054902, 'num_objects': 5134, 'num_object_clones': 0, 'num_object_copies': 20536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7571, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3127}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:41.063 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:41.691 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:41.691 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29111/335152 objects degraded (8.686%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":379,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":83788,"data_bytes":686343564,"bytes_used":2570014720,"bytes_avail":383977041920,"bytes_total":386547056640,"degraded_objects":29111,"degraded_total":335152,"degraded_ratio":0.086859096768033606,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":109184,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":1310720,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":198},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:41.719 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:41.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:42.146 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:42.165 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-18T11:49:42.030526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":678929804,"num_objects":82883,"num_object_clones":0,"num_object_copies":331532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82883,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124204,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4401,"num_bytes_recovered":34373632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124231,"ondisk_log_size":124231,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2512176,"kb_used_data":1283892,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":374975184,"statfs":{"total":386547056640,"available":383974588416,"internally_reserved":0,"allocated":1314705408,"data_stored":1314032538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,5,6,7,10,1,0,0,0,0,3,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17113088,"num_objects":-2089,"num_object_clones":0,"num_object_copies":-8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2089,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.079925"},"pg_stats":[{"pgid":"1.f","version":"34'7958","reported_seq":8032,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.100429+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:41.100429+0000","last_peered":"2024-09-18T11:49:41.100429+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:41.100429+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7958,"log_dups_size":0,"ondisk_log_size":7958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43352064,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7852","reported_seq":12132,"reported_epoch":34,"state":"peering","last_fresh":"2024-09-18T11:49:41.051330+0000","last_change":"2024-09-18T11:49:41.051149+0000","last_active":"2024-09-18T11:49:41.050911+0000","last_peered":"2024-09-18T11:49:41.027220+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:41.051330+0000","last_undegraded":"2024-09-18T11:49:41.051330+0000","last_fullsized":"2024-09-18T11:49:41.051330+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42628342,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"34'7713","reported_seq":7823,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.746613+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:41.746613+0000","last_peered":"2024-09-18T11:49:41.746613+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:41.746613+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41736851,"num_objects":5096,"num_object_clones":0,"num_object_copies":20384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7712,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7697","reported_seq":7773,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.148272+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:37.148272+0000","last_peered":"2024-09-18T11:49:37.148272+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:37.148272+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42833142,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'7758","reported_seq":7844,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087664+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:41.087664+0000","last_peered":"2024-09-18T11:49:41.087664+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:41.087664+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42079478,"num_objects":5137,"num_object_clones":0,"num_object_copies":20548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3069}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'7762","reported_seq":7835,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.103224+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:41.103224+0000","last_peered":"2024-09-18T11:49:41.103224+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:41.103224+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":5024,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7798","reported_seq":7887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.307893+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:37.307893+0000","last_peered":"2024-09-18T11:49:37.307893+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:37.307893+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42743030,"num_objects":5218,"num_object_clones":0,"num_object_copies":20872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3095}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7631","reported_seq":7726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.261979+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:37.261979+0000","last_peered":"2024-09-18T11:49:37.261979+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:37.261979+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42956022,"num_objects":5244,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7629,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'7934","reported_seq":8042,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.130539+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:49:41.130539+0000","last_peered":"2024-09-18T11:49:41.130539+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:41.130539+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7934,"log_dups_size":0,"ondisk_log_size":7934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42936812,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'7551","reported_seq":7656,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.679701+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:41.679701+0000","last_peered":"2024-09-18T11:49:41.679701+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:41.679701+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7551,"log_dups_size":0,"ondisk_log_size":7551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41309430,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'7795","reported_seq":7889,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087068+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:41.087068+0000","last_peered":"2024-09-18T11:49:41.087068+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:41.087068+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7795,"log_dups_size":0,"ondisk_log_size":7795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42905457,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7792,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'7830","reported_seq":7894,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.052925+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:41.052925+0000","last_peered":"2024-09-18T11:49:41.052925+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:41.052925+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42605179,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7829,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'7726","reported_seq":7813,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.850336+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:41.850336+0000","last_peered":"2024-09-18T11:49:41.850336+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:41.850336+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7726,"log_dups_size":0,"ondisk_log_size":7726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42932859,"num_objects":5241,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7900","reported_seq":7990,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087541+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:41.087541+0000","last_peered":"2024-09-18T11:49:41.087541+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:41.087541+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7900,"log_dups_size":0,"ondisk_log_size":7900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42513654,"num_objects":5190,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7898,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3051}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'7753","reported_seq":12039,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:49:41.088272+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:41.088272+0000","last_peered":"2024-09-18T11:49:41.088272+0000","last_clean":"2024-09-18T11:49:41.088272+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:41.088272+0000","last_undegraded":"2024-09-18T11:49:41.088272+0000","last_fullsized":"2024-09-18T11:49:41.088272+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7753,"log_dups_size":0,"ondisk_log_size":7753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42185974,"num_objects":5150,"num_object_clones":0,"num_object_copies":20600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7751,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"33'7573","reported_seq":7649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:37.028816+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:37.028816+0000","last_peered":"2024-09-18T11:49:37.028816+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:37.028816+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42054902,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7571,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3127}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":678929804,"num_objects":82883,"num_object_clones":0,"num_object_copies":331532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82883,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124204,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4401,"num_bytes_recovered":34373632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1312616448,"data_stored":1312616448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124231,"ondisk_log_size":124231,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633284,"kb_used_data":331248,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93738556,"statfs":{"total":96636764160,"available":95988281344,"internally_reserved":0,"allocated":339197952,"data_stored":339030767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,3,3,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.2759999999999998},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.246},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.41999999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":2.286},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.26200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640420,"kb_used_data":330584,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93731420,"statfs":{"total":96636764160,"available":95980974080,"internally_reserved":0,"allocated":338518016,"data_stored":338346701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,1,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.0999999999999996},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.1289999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":5.3010000000000002},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":0.28599999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.501},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.46899999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597408,"kb_used_data":287128,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93774432,"statfs":{"total":96636764160,"available":96025018368,"internally_reserved":0,"allocated":294019072,"data_stored":293851887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":14.518000000000001},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":12.859}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":12.827999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":5.1059999999999999},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.0270000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641064,"kb_used_data":334932,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93730776,"statfs":{"total":96636764160,"available":95980314624,"internally_reserved":0,"allocated":342970368,"data_stored":342803183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,0,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":0.311},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":8.0790000000000006}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":13.93},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.9459999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":5.3159999999999998},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":10.919}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":342474752,"data_stored":342474752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293654528,"data_stored":293654528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337948672,"data_stored":337948672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":338538496,"data_stored":338538496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:45.168 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:45.720 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:45.721 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:45.749 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-18T11:49:43.954237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":676619660,"num_objects":82601,"num_object_clones":0,"num_object_copies":330404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124486,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4401,"num_bytes_recovered":34373632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124513,"ondisk_log_size":124513,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2511640,"kb_used_data":1279200,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":374975720,"statfs":{"total":386547056640,"available":383975137280,"internally_reserved":0,"allocated":1309900800,"data_stored":1309219649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,4,13,8,10,1,0,0,0,0,3,1,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19423232,"num_objects":-2371,"num_object_clones":0,"num_object_copies":-9484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2371,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002784"},"pg_stats":[{"pgid":"1.f","version":"34'7958","reported_seq":8032,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.100429+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:41.100429+0000","last_peered":"2024-09-18T11:49:41.100429+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:41.100429+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7958,"log_dups_size":0,"ondisk_log_size":7958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43352064,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7852","reported_seq":12132,"reported_epoch":34,"state":"peering","last_fresh":"2024-09-18T11:49:41.051330+0000","last_change":"2024-09-18T11:49:41.051149+0000","last_active":"2024-09-18T11:49:41.050911+0000","last_peered":"2024-09-18T11:49:41.027220+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:41.051330+0000","last_undegraded":"2024-09-18T11:49:41.051330+0000","last_fullsized":"2024-09-18T11:49:41.051330+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42628342,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"34'7713","reported_seq":7823,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.746613+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:41.746613+0000","last_peered":"2024-09-18T11:49:41.746613+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:41.746613+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41736851,"num_objects":5096,"num_object_clones":0,"num_object_copies":20384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7712,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7764","reported_seq":7842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.254497+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:42.254497+0000","last_peered":"2024-09-18T11:49:42.254497+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:42.254497+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42284278,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'7758","reported_seq":7844,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087664+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:41.087664+0000","last_peered":"2024-09-18T11:49:41.087664+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:41.087664+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42079478,"num_objects":5137,"num_object_clones":0,"num_object_copies":20548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3069}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'7762","reported_seq":7835,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.103224+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:41.103224+0000","last_peered":"2024-09-18T11:49:41.103224+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:41.103224+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":5024,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7871","reported_seq":7962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.206475+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:42.206475+0000","last_peered":"2024-09-18T11:49:42.206475+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:42.206475+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7871,"log_dups_size":0,"ondisk_log_size":7871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42145014,"num_objects":5145,"num_object_clones":0,"num_object_copies":20580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3022}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7707","reported_seq":7802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.286755+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:42.286755+0000","last_peered":"2024-09-18T11:49:42.286755+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:42.286755+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42333430,"num_objects":5168,"num_object_clones":0,"num_object_copies":20672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7705,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'7934","reported_seq":8042,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.130539+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:49:41.130539+0000","last_peered":"2024-09-18T11:49:41.130539+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:41.130539+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7934,"log_dups_size":0,"ondisk_log_size":7934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42936812,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'7551","reported_seq":7656,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.679701+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:41.679701+0000","last_peered":"2024-09-18T11:49:41.679701+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:41.679701+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7551,"log_dups_size":0,"ondisk_log_size":7551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41309430,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'7795","reported_seq":7889,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087068+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:41.087068+0000","last_peered":"2024-09-18T11:49:41.087068+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:41.087068+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7795,"log_dups_size":0,"ondisk_log_size":7795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42905457,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7792,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'7830","reported_seq":7894,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.052925+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:41.052925+0000","last_peered":"2024-09-18T11:49:41.052925+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:41.052925+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42605179,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7829,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'7726","reported_seq":7813,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.850336+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:41.850336+0000","last_peered":"2024-09-18T11:49:41.850336+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:41.850336+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7726,"log_dups_size":0,"ondisk_log_size":7726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42932859,"num_objects":5241,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7900","reported_seq":7990,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087541+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:41.087541+0000","last_peered":"2024-09-18T11:49:41.087541+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:41.087541+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7900,"log_dups_size":0,"ondisk_log_size":7900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42513654,"num_objects":5190,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7898,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3051}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'7753","reported_seq":12039,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:49:41.088272+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:41.088272+0000","last_peered":"2024-09-18T11:49:41.088272+0000","last_clean":"2024-09-18T11:49:41.088272+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:41.088272+0000","last_undegraded":"2024-09-18T11:49:41.088272+0000","last_fullsized":"2024-09-18T11:49:41.088272+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7753,"log_dups_size":0,"ondisk_log_size":7753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42185974,"num_objects":5150,"num_object_clones":0,"num_object_copies":20600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7751,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7639","reported_seq":7717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.204944+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:42.204944+0000","last_peered":"2024-09-18T11:49:42.204944+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:42.204944+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7639,"log_dups_size":0,"ondisk_log_size":7639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41514230,"num_objects":5068,"num_object_clones":0,"num_object_copies":20272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7637,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":676619660,"num_objects":82601,"num_object_clones":0,"num_object_copies":330404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124486,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4401,"num_bytes_recovered":34373632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1307844608,"data_stored":1307844608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124513,"ondisk_log_size":124513,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633284,"kb_used_data":331248,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93738556,"statfs":{"total":96636764160,"available":95988281344,"internally_reserved":0,"allocated":339197952,"data_stored":339030767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,3,3,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.2759999999999998},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.246},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.41999999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":2.286},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.26200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640420,"kb_used_data":330584,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93731420,"statfs":{"total":96636764160,"available":95980974080,"internally_reserved":0,"allocated":338518016,"data_stored":338346701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,1,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.0999999999999996},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.1289999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":5.3010000000000002},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":0.28599999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.501},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.46899999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597408,"kb_used_data":287128,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93774432,"statfs":{"total":96636764160,"available":96025018368,"internally_reserved":0,"allocated":294019072,"data_stored":293851887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":14.518000000000001},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":12.859}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":12.827999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":5.1059999999999999},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.0270000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640528,"kb_used_data":330240,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93731312,"statfs":{"total":96636764160,"available":95980863488,"internally_reserved":0,"allocated":338165760,"data_stored":337990294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,8,1,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":3.6880000000000002},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":15.632999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":21.366},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":15.611000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.734},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":15.574}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":337702912,"data_stored":337702912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293654528,"data_stored":293654528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337948672,"data_stored":337948672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":338538496,"data_stored":338538496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:45.750 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:46.279 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:46.280 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:46.308 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-18T11:49:45.958128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":676619660,"num_objects":82601,"num_object_clones":0,"num_object_copies":330404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124486,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4401,"num_bytes_recovered":34373632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124513,"ondisk_log_size":124513,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2511640,"kb_used_data":1279200,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":374975720,"statfs":{"total":386547056640,"available":383975137280,"internally_reserved":0,"allocated":1309900800,"data_stored":1309219649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,4,13,8,10,1,0,0,0,0,3,1,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9723904,"num_objects":-1187,"num_object_clones":0,"num_object_copies":-4748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-56,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1187,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006017"},"pg_stats":[{"pgid":"1.f","version":"34'7958","reported_seq":8032,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.100429+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:41.100429+0000","last_peered":"2024-09-18T11:49:41.100429+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:41.100429+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7958,"log_dups_size":0,"ondisk_log_size":7958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43352064,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7852","reported_seq":12132,"reported_epoch":34,"state":"peering","last_fresh":"2024-09-18T11:49:41.051330+0000","last_change":"2024-09-18T11:49:41.051149+0000","last_active":"2024-09-18T11:49:41.050911+0000","last_peered":"2024-09-18T11:49:41.027220+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:41.051330+0000","last_undegraded":"2024-09-18T11:49:41.051330+0000","last_fullsized":"2024-09-18T11:49:41.051330+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42628342,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"34'7713","reported_seq":7823,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.746613+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:41.746613+0000","last_peered":"2024-09-18T11:49:41.746613+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:41.746613+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41736851,"num_objects":5096,"num_object_clones":0,"num_object_copies":20384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7712,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7764","reported_seq":7842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.254497+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:42.254497+0000","last_peered":"2024-09-18T11:49:42.254497+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:42.254497+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42284278,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'7758","reported_seq":7844,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087664+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:41.087664+0000","last_peered":"2024-09-18T11:49:41.087664+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:41.087664+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42079478,"num_objects":5137,"num_object_clones":0,"num_object_copies":20548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3069}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'7762","reported_seq":7835,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.103224+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:41.103224+0000","last_peered":"2024-09-18T11:49:41.103224+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:41.103224+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":5024,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7871","reported_seq":7962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.206475+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:42.206475+0000","last_peered":"2024-09-18T11:49:42.206475+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:42.206475+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7871,"log_dups_size":0,"ondisk_log_size":7871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42145014,"num_objects":5145,"num_object_clones":0,"num_object_copies":20580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3022}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7707","reported_seq":7802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.286755+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:42.286755+0000","last_peered":"2024-09-18T11:49:42.286755+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:42.286755+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42333430,"num_objects":5168,"num_object_clones":0,"num_object_copies":20672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7705,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'7934","reported_seq":8042,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.130539+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:49:41.130539+0000","last_peered":"2024-09-18T11:49:41.130539+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:41.130539+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7934,"log_dups_size":0,"ondisk_log_size":7934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42936812,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'7551","reported_seq":7656,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.679701+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:41.679701+0000","last_peered":"2024-09-18T11:49:41.679701+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:41.679701+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7551,"log_dups_size":0,"ondisk_log_size":7551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41309430,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'7795","reported_seq":7889,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087068+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:41.087068+0000","last_peered":"2024-09-18T11:49:41.087068+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:41.087068+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7795,"log_dups_size":0,"ondisk_log_size":7795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42905457,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7792,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'7830","reported_seq":7894,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.052925+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:41.052925+0000","last_peered":"2024-09-18T11:49:41.052925+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:41.052925+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42605179,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7829,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'7726","reported_seq":7813,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.850336+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:41.850336+0000","last_peered":"2024-09-18T11:49:41.850336+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:41.850336+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7726,"log_dups_size":0,"ondisk_log_size":7726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42932859,"num_objects":5241,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7900","reported_seq":7990,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087541+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:41.087541+0000","last_peered":"2024-09-18T11:49:41.087541+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:41.087541+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7900,"log_dups_size":0,"ondisk_log_size":7900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42513654,"num_objects":5190,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7898,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3051}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'7753","reported_seq":12039,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:49:41.088272+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:41.088272+0000","last_peered":"2024-09-18T11:49:41.088272+0000","last_clean":"2024-09-18T11:49:41.088272+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:41.088272+0000","last_undegraded":"2024-09-18T11:49:41.088272+0000","last_fullsized":"2024-09-18T11:49:41.088272+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7753,"log_dups_size":0,"ondisk_log_size":7753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42185974,"num_objects":5150,"num_object_clones":0,"num_object_copies":20600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7751,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7639","reported_seq":7717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.204944+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:42.204944+0000","last_peered":"2024-09-18T11:49:42.204944+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:42.204944+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7639,"log_dups_size":0,"ondisk_log_size":7639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41514230,"num_objects":5068,"num_object_clones":0,"num_object_copies":20272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7637,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":676619660,"num_objects":82601,"num_object_clones":0,"num_object_copies":330404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124486,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4401,"num_bytes_recovered":34373632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1307844608,"data_stored":1307844608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124513,"ondisk_log_size":124513,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633284,"kb_used_data":331248,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93738556,"statfs":{"total":96636764160,"available":95988281344,"internally_reserved":0,"allocated":339197952,"data_stored":339030767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,3,3,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.2759999999999998},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.246},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.41999999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":2.286},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.26200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640420,"kb_used_data":330584,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93731420,"statfs":{"total":96636764160,"available":95980974080,"internally_reserved":0,"allocated":338518016,"data_stored":338346701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,1,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.0999999999999996},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.1289999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":5.3010000000000002},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":0.28599999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.501},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.46899999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597408,"kb_used_data":287128,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93774432,"statfs":{"total":96636764160,"available":96025018368,"internally_reserved":0,"allocated":294019072,"data_stored":293851887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":14.518000000000001},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":12.859}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":12.827999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":5.1059999999999999},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.0270000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640528,"kb_used_data":330240,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93731312,"statfs":{"total":96636764160,"available":95980863488,"internally_reserved":0,"allocated":338165760,"data_stored":337990294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,8,1,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":3.6880000000000002},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":15.632999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":21.366},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":15.611000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.734},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":15.574}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":337702912,"data_stored":337702912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293654528,"data_stored":293654528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337948672,"data_stored":337948672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":338538496,"data_stored":338538496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:46.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:46.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'7958", 'reported_seq': 8032, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:41.100429+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:41.100429+0000', 'last_peered': '2024-09-18T11:49:41.100429+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:41.100429+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7958, 'log_dups_size': 0, 'ondisk_log_size': 7958, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43352064, 'num_objects': 5292, 'num_object_clones': 0, 'num_object_copies': 21168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2115, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7958, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3177}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:46.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-18T11:49:46.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7852", 'reported_seq': 12132, 'reported_epoch': 34, 'state': 'peering', 'last_fresh': '2024-09-18T11:49:41.051330+0000', 'last_change': '2024-09-18T11:49:41.051149+0000', 'last_active': '2024-09-18T11:49:41.050911+0000', 'last_peered': '2024-09-18T11:49:41.027220+0000', 'last_clean': '2024-09-18T11:43:44.365240+0000', 'last_became_active': '2024-09-18T11:44:47.400915+0000', 'last_became_peered': '2024-09-18T11:44:47.400915+0000', 'last_unstale': '2024-09-18T11:49:41.051330+0000', 'last_undegraded': '2024-09-18T11:49:41.051330+0000', 'last_fullsized': '2024-09-18T11:49:41.051330+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7852, 'log_dups_size': 0, 'ondisk_log_size': 7852, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42628342, 'num_objects': 5204, 'num_object_clones': 0, 'num_object_copies': 20816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7850, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2106, 'num_bytes_recovered': 17219584, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [1, 0, 3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-18T11:49:46.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:46.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'7713", 'reported_seq': 7823, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:41.746613+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:41.746613+0000', 'last_peered': '2024-09-18T11:49:41.746613+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:41.746613+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7713, 'log_dups_size': 0, 'ondisk_log_size': 7713, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41736851, 'num_objects': 5096, 'num_object_clones': 0, 'num_object_copies': 20384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5096, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7712, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3054}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:46.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:46.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'7764", 'reported_seq': 7842, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:42.254497+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:42.254497+0000', 'last_peered': '2024-09-18T11:49:42.254497+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:42.254497+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7764, 'log_dups_size': 0, 'ondisk_log_size': 7764, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42284278, 'num_objects': 5162, 'num_object_clones': 0, 'num_object_copies': 20648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7762, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:46.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:46.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'7758", 'reported_seq': 7844, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:41.087664+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:41.087664+0000', 'last_peered': '2024-09-18T11:49:41.087664+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:41.087664+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7758, 'log_dups_size': 0, 'ondisk_log_size': 7758, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42079478, 'num_objects': 5137, 'num_object_clones': 0, 'num_object_copies': 20548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2070, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7756, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3069}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:46.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:46.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'7762", 'reported_seq': 7835, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:41.103224+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:41.103224+0000', 'last_peered': '2024-09-18T11:49:41.103224+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:41.103224+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7762, 'log_dups_size': 0, 'ondisk_log_size': 7762, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41156608, 'num_objects': 5024, 'num_object_clones': 0, 'num_object_copies': 20096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7762, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2884}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:46.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:46.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7871", 'reported_seq': 7962, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:42.206475+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:42.206475+0000', 'last_peered': '2024-09-18T11:49:42.206475+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:42.206475+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7871, 'log_dups_size': 0, 'ondisk_log_size': 7871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42145014, 'num_objects': 5145, 'num_object_clones': 0, 'num_object_copies': 20580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7869, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3022}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:46.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:46.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7707", 'reported_seq': 7802, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:42.286755+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:42.286755+0000', 'last_peered': '2024-09-18T11:49:42.286755+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:42.286755+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7707, 'log_dups_size': 0, 'ondisk_log_size': 7707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42333430, 'num_objects': 5168, 'num_object_clones': 0, 'num_object_copies': 20672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2021, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7705, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3149}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:46.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:46.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'7934", 'reported_seq': 8042, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:41.130539+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:49:41.130539+0000', 'last_peered': '2024-09-18T11:49:41.130539+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:41.130539+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7934, 'log_dups_size': 0, 'ondisk_log_size': 7934, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42936812, 'num_objects': 5242, 'num_object_clones': 0, 'num_object_copies': 20968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2115, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7931, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 24576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3127}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:46.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:46.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'7551", 'reported_seq': 7656, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:41.679701+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:41.679701+0000', 'last_peered': '2024-09-18T11:49:41.679701+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:41.679701+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7551, 'log_dups_size': 0, 'ondisk_log_size': 7551, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41309430, 'num_objects': 5043, 'num_object_clones': 0, 'num_object_copies': 20172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1968, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7549, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3077}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:46.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:46.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'7795", 'reported_seq': 7889, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:41.087068+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:41.087068+0000', 'last_peered': '2024-09-18T11:49:41.087068+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:41.087068+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7795, 'log_dups_size': 0, 'ondisk_log_size': 7795, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42905457, 'num_objects': 5238, 'num_object_clones': 0, 'num_object_copies': 20952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7792, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3192}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:46.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:46.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'7830", 'reported_seq': 7894, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:41.052925+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:41.052925+0000', 'last_peered': '2024-09-18T11:49:41.052925+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:41.052925+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7830, 'log_dups_size': 0, 'ondisk_log_size': 7830, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42605179, 'num_objects': 5201, 'num_object_clones': 0, 'num_object_copies': 20804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2098, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7829, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:46.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:46.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'7726", 'reported_seq': 7813, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:41.850336+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:41.850336+0000', 'last_peered': '2024-09-18T11:49:41.850336+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:41.850336+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7726, 'log_dups_size': 0, 'ondisk_log_size': 7726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42932859, 'num_objects': 5241, 'num_object_clones': 0, 'num_object_copies': 20964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7725, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3191}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:46.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:46.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7900", 'reported_seq': 7990, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:41.087541+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:41.087541+0000', 'last_peered': '2024-09-18T11:49:41.087541+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:41.087541+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7900, 'log_dups_size': 0, 'ondisk_log_size': 7900, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42513654, 'num_objects': 5190, 'num_object_clones': 0, 'num_object_copies': 20760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2141, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7898, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3051}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:46.314 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:46.314 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7639", 'reported_seq': 7717, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:42.204944+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:42.204944+0000', 'last_peered': '2024-09-18T11:49:42.204944+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:42.204944+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7639, 'log_dups_size': 0, 'ondisk_log_size': 7639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41514230, 'num_objects': 5068, 'num_object_clones': 0, 'num_object_copies': 20272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7637, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3061}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:46.314 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:46.992 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:46.992 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29055/330404 objects degraded (8.794%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":385,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":82601,"data_bytes":676619660,"bytes_used":2571919360,"bytes_avail":383975137280,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":29055,"degraded_total":330404,"degraded_ratio":0.087937797363228043,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":53207,"recovering_keys_per_sec":0,"num_objects_recovered":56,"num_bytes_recovered":425984,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":148},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:47.016 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:47.016 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:47.570 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:47.571 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:47.588 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-18T11:49:45.958128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":676619660,"num_objects":82601,"num_object_clones":0,"num_object_copies":330404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124486,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4401,"num_bytes_recovered":34373632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124513,"ondisk_log_size":124513,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2511640,"kb_used_data":1279200,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":374975720,"statfs":{"total":386547056640,"available":383975137280,"internally_reserved":0,"allocated":1309900800,"data_stored":1309219649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,4,13,8,10,1,0,0,0,0,3,1,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9723904,"num_objects":-1187,"num_object_clones":0,"num_object_copies":-4748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-56,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1187,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006017"},"pg_stats":[{"pgid":"1.f","version":"34'7958","reported_seq":8032,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.100429+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:41.100429+0000","last_peered":"2024-09-18T11:49:41.100429+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:41.100429+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7958,"log_dups_size":0,"ondisk_log_size":7958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43352064,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7852","reported_seq":12132,"reported_epoch":34,"state":"peering","last_fresh":"2024-09-18T11:49:41.051330+0000","last_change":"2024-09-18T11:49:41.051149+0000","last_active":"2024-09-18T11:49:41.050911+0000","last_peered":"2024-09-18T11:49:41.027220+0000","last_clean":"2024-09-18T11:43:44.365240+0000","last_became_active":"2024-09-18T11:44:47.400915+0000","last_became_peered":"2024-09-18T11:44:47.400915+0000","last_unstale":"2024-09-18T11:49:41.051330+0000","last_undegraded":"2024-09-18T11:49:41.051330+0000","last_fullsized":"2024-09-18T11:49:41.051330+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42628342,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"34'7713","reported_seq":7823,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.746613+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:41.746613+0000","last_peered":"2024-09-18T11:49:41.746613+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:41.746613+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41736851,"num_objects":5096,"num_object_clones":0,"num_object_copies":20384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7712,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7764","reported_seq":7842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.254497+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:42.254497+0000","last_peered":"2024-09-18T11:49:42.254497+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:42.254497+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42284278,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'7758","reported_seq":7844,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087664+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:41.087664+0000","last_peered":"2024-09-18T11:49:41.087664+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:41.087664+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42079478,"num_objects":5137,"num_object_clones":0,"num_object_copies":20548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3069}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'7762","reported_seq":7835,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.103224+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:41.103224+0000","last_peered":"2024-09-18T11:49:41.103224+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:41.103224+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":5024,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7871","reported_seq":7962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.206475+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:42.206475+0000","last_peered":"2024-09-18T11:49:42.206475+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:42.206475+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7871,"log_dups_size":0,"ondisk_log_size":7871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42145014,"num_objects":5145,"num_object_clones":0,"num_object_copies":20580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3022}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7707","reported_seq":7802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.286755+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:42.286755+0000","last_peered":"2024-09-18T11:49:42.286755+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:42.286755+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42333430,"num_objects":5168,"num_object_clones":0,"num_object_copies":20672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7705,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'7934","reported_seq":8042,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.130539+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:49:41.130539+0000","last_peered":"2024-09-18T11:49:41.130539+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:41.130539+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7934,"log_dups_size":0,"ondisk_log_size":7934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42936812,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'7551","reported_seq":7656,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.679701+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:41.679701+0000","last_peered":"2024-09-18T11:49:41.679701+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:41.679701+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7551,"log_dups_size":0,"ondisk_log_size":7551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41309430,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'7795","reported_seq":7889,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087068+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:41.087068+0000","last_peered":"2024-09-18T11:49:41.087068+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:41.087068+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7795,"log_dups_size":0,"ondisk_log_size":7795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42905457,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7792,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'7830","reported_seq":7894,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.052925+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:41.052925+0000","last_peered":"2024-09-18T11:49:41.052925+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:41.052925+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42605179,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7829,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'7726","reported_seq":7813,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.850336+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:41.850336+0000","last_peered":"2024-09-18T11:49:41.850336+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:41.850336+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7726,"log_dups_size":0,"ondisk_log_size":7726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42932859,"num_objects":5241,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7900","reported_seq":7990,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:41.087541+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:41.087541+0000","last_peered":"2024-09-18T11:49:41.087541+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:41.087541+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7900,"log_dups_size":0,"ondisk_log_size":7900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42513654,"num_objects":5190,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7898,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3051}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'7753","reported_seq":12039,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:49:41.088272+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:41.088272+0000","last_peered":"2024-09-18T11:49:41.088272+0000","last_clean":"2024-09-18T11:49:41.088272+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:41.088272+0000","last_undegraded":"2024-09-18T11:49:41.088272+0000","last_fullsized":"2024-09-18T11:49:41.088272+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7753,"log_dups_size":0,"ondisk_log_size":7753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42185974,"num_objects":5150,"num_object_clones":0,"num_object_copies":20600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7751,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7639","reported_seq":7717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:42.204944+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:42.204944+0000","last_peered":"2024-09-18T11:49:42.204944+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:42.204944+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7639,"log_dups_size":0,"ondisk_log_size":7639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41514230,"num_objects":5068,"num_object_clones":0,"num_object_copies":20272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7637,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":676619660,"num_objects":82601,"num_object_clones":0,"num_object_copies":330404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124486,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4401,"num_bytes_recovered":34373632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1307844608,"data_stored":1307844608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124513,"ondisk_log_size":124513,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633284,"kb_used_data":331248,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93738556,"statfs":{"total":96636764160,"available":95988281344,"internally_reserved":0,"allocated":339197952,"data_stored":339030767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,3,3,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.2759999999999998},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.246},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.41999999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":2.286},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.26200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640420,"kb_used_data":330584,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93731420,"statfs":{"total":96636764160,"available":95980974080,"internally_reserved":0,"allocated":338518016,"data_stored":338346701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,1,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.0999999999999996},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.1289999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":5.3010000000000002},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":0.28599999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.501},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.46899999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597408,"kb_used_data":287128,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93774432,"statfs":{"total":96636764160,"available":96025018368,"internally_reserved":0,"allocated":294019072,"data_stored":293851887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":14.518000000000001},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":12.859}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":12.827999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":5.1059999999999999},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.0270000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640528,"kb_used_data":330240,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93731312,"statfs":{"total":96636764160,"available":95980863488,"internally_reserved":0,"allocated":338165760,"data_stored":337990294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,8,1,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":3.6880000000000002},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":15.632999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":21.366},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":15.611000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.734},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":15.574}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":337702912,"data_stored":337702912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293654528,"data_stored":293654528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337948672,"data_stored":337948672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":338538496,"data_stored":338538496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:50.590 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:51.159 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:51.160 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:51.185 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-18T11:49:49.959199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666830220,"num_objects":81406,"num_object_clones":0,"num_object_copies":325624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81406,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125681,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":34971648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125708,"ondisk_log_size":125708,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2476048,"kb_used_data":1264428,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375011312,"statfs":{"total":386547056640,"available":384011583488,"internally_reserved":0,"allocated":1294774272,"data_stored":1294072408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,8,11,5,0,0,0,0,4,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19513344,"num_objects":-2382,"num_object_clones":0,"num_object_copies":-9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.931211"},"pg_stats":[{"pgid":"1.f","version":"35'8030","reported_seq":8109,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:45.984810+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:45.984810+0000","last_peered":"2024-09-18T11:49:45.984810+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:45.984810+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42762240,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8030,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7929","reported_seq":12219,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:46.289568+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:49:46.289568+0000","last_peered":"2024-09-18T11:49:46.289568+0000","last_clean":"2024-09-18T11:49:46.289568+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:49:46.289568+0000","last_undegraded":"2024-09-18T11:49:46.289568+0000","last_fullsized":"2024-09-18T11:49:46.289568+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997558,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7927,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'7782","reported_seq":7891,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.965166+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:46.965166+0000","last_peered":"2024-09-18T11:49:46.965166+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:46.965166+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7782,"log_dups_size":0,"ondisk_log_size":7782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41171603,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7781,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2985}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7848","reported_seq":7926,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.239127+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:47.239127+0000","last_peered":"2024-09-18T11:49:47.239127+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:47.239127+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7848,"log_dups_size":0,"ondisk_log_size":7848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41596150,"num_objects":5078,"num_object_clones":0,"num_object_copies":20312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'7838","reported_seq":7931,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.081576+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:46.081576+0000","last_peered":"2024-09-18T11:49:46.081576+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:46.081576+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7838,"log_dups_size":0,"ondisk_log_size":7838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41424118,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7836,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7836","reported_seq":7913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:45.872732+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:45.872732+0000","last_peered":"2024-09-18T11:49:45.872732+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:45.872732+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7836,"log_dups_size":0,"ondisk_log_size":7836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40550400,"num_objects":4950,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7836,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7953","reported_seq":8044,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.282123+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:47.282123+0000","last_peered":"2024-09-18T11:49:47.282123+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:47.282123+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7953,"log_dups_size":0,"ondisk_log_size":7953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41473270,"num_objects":5063,"num_object_clones":0,"num_object_copies":20252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7951,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2940}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7781","reported_seq":7878,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.304221+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:47.304221+0000","last_peered":"2024-09-18T11:49:47.304221+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:47.304221+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7781,"log_dups_size":0,"ondisk_log_size":7781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727222,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7779,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8015","reported_seq":8268,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.174076+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:49:46.174076+0000","last_peered":"2024-09-18T11:49:46.174076+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:46.174076+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42273260,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8012,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7622","reported_seq":7728,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.953909+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:46.953909+0000","last_peered":"2024-09-18T11:49:46.953909+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:46.953909+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40727798,"num_objects":4972,"num_object_clones":0,"num_object_copies":19888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7620,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3006}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7869","reported_seq":7964,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.123218+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:46.123218+0000","last_peered":"2024-09-18T11:49:46.123218+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:46.123218+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7869,"log_dups_size":0,"ondisk_log_size":7869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42299249,"num_objects":5164,"num_object_clones":0,"num_object_copies":20656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7866,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7896","reported_seq":7965,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:45.988263+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:45.988263+0000","last_peered":"2024-09-18T11:49:45.988263+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:45.988263+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42064507,"num_objects":5135,"num_object_clones":0,"num_object_copies":20540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7895,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3040}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7801","reported_seq":7889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.889676+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:46.889676+0000","last_peered":"2024-09-18T11:49:46.889676+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:46.889676+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42318459,"num_objects":5166,"num_object_clones":0,"num_object_copies":20664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3116}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7975","reported_seq":8068,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.066196+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:46.066196+0000","last_peered":"2024-09-18T11:49:46.066196+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:46.066196+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41899254,"num_objects":5115,"num_object_clones":0,"num_object_copies":20460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7832","reported_seq":12119,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:46.210929+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:46.210929+0000","last_peered":"2024-09-18T11:49:46.210929+0000","last_clean":"2024-09-18T11:49:46.210929+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:46.210929+0000","last_undegraded":"2024-09-18T11:49:46.210929+0000","last_fullsized":"2024-09-18T11:49:46.210929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7832,"log_dups_size":0,"ondisk_log_size":7832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41538806,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7701","reported_seq":7779,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.178221+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:47.178221+0000","last_peered":"2024-09-18T11:49:47.178221+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:47.178221+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006326,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7699,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":666830220,"num_objects":81406,"num_object_clones":0,"num_object_copies":325624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81406,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125681,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":34971648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1292546048,"data_stored":1292546048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125708,"ondisk_log_size":125708,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637000,"kb_used_data":326652,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93734840,"statfs":{"total":96636764160,"available":95984476160,"internally_reserved":0,"allocated":334491648,"data_stored":334316182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,0,0,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.39800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.232},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":1.8260000000000001},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.38700000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627332,"kb_used_data":325820,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93744508,"statfs":{"total":96636764160,"available":95994376192,"internally_reserved":0,"allocated":333639680,"data_stored":333464214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.9260000000000002},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.9580000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.157},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":1.452}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":3.746},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.0650000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588528,"kb_used_data":286568,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93783312,"statfs":{"total":96636764160,"available":96034111488,"internally_reserved":0,"allocated":293445632,"data_stored":293270166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.238},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.95299999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.78000000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.0429999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623188,"kb_used_data":325388,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93748652,"statfs":{"total":96636764160,"available":95998619648,"internally_reserved":0,"allocated":333197312,"data_stored":333021846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":14.119},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.7460000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.7750000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":11.651999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":4.8049999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332746752,"data_stored":332746752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293048320,"data_stored":293048320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333004800,"data_stored":333004800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":333746176,"data_stored":333746176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:51.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:51.746 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:51.747 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:51.784 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-18T11:49:49.959199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666830220,"num_objects":81406,"num_object_clones":0,"num_object_copies":325624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81406,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125681,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":34971648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125708,"ondisk_log_size":125708,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2476048,"kb_used_data":1264428,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375011312,"statfs":{"total":386547056640,"available":384011583488,"internally_reserved":0,"allocated":1294774272,"data_stored":1294072408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,8,11,5,0,0,0,0,4,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19513344,"num_objects":-2382,"num_object_clones":0,"num_object_copies":-9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.931211"},"pg_stats":[{"pgid":"1.f","version":"35'8030","reported_seq":8109,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:45.984810+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:45.984810+0000","last_peered":"2024-09-18T11:49:45.984810+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:45.984810+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42762240,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8030,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7929","reported_seq":12219,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:46.289568+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:49:46.289568+0000","last_peered":"2024-09-18T11:49:46.289568+0000","last_clean":"2024-09-18T11:49:46.289568+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:49:46.289568+0000","last_undegraded":"2024-09-18T11:49:46.289568+0000","last_fullsized":"2024-09-18T11:49:46.289568+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997558,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7927,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'7782","reported_seq":7891,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.965166+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:46.965166+0000","last_peered":"2024-09-18T11:49:46.965166+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:46.965166+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7782,"log_dups_size":0,"ondisk_log_size":7782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41171603,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7781,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2985}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7848","reported_seq":7926,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.239127+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:47.239127+0000","last_peered":"2024-09-18T11:49:47.239127+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:47.239127+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7848,"log_dups_size":0,"ondisk_log_size":7848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41596150,"num_objects":5078,"num_object_clones":0,"num_object_copies":20312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'7838","reported_seq":7931,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.081576+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:46.081576+0000","last_peered":"2024-09-18T11:49:46.081576+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:46.081576+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7838,"log_dups_size":0,"ondisk_log_size":7838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41424118,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7836,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7836","reported_seq":7913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:45.872732+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:45.872732+0000","last_peered":"2024-09-18T11:49:45.872732+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:45.872732+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7836,"log_dups_size":0,"ondisk_log_size":7836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40550400,"num_objects":4950,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7836,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7953","reported_seq":8044,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.282123+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:47.282123+0000","last_peered":"2024-09-18T11:49:47.282123+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:47.282123+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7953,"log_dups_size":0,"ondisk_log_size":7953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41473270,"num_objects":5063,"num_object_clones":0,"num_object_copies":20252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7951,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2940}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7781","reported_seq":7878,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.304221+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:47.304221+0000","last_peered":"2024-09-18T11:49:47.304221+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:47.304221+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7781,"log_dups_size":0,"ondisk_log_size":7781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727222,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7779,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8015","reported_seq":8268,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.174076+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:49:46.174076+0000","last_peered":"2024-09-18T11:49:46.174076+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:46.174076+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42273260,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8012,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7622","reported_seq":7728,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.953909+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:46.953909+0000","last_peered":"2024-09-18T11:49:46.953909+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:46.953909+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40727798,"num_objects":4972,"num_object_clones":0,"num_object_copies":19888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7620,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3006}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7869","reported_seq":7964,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.123218+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:46.123218+0000","last_peered":"2024-09-18T11:49:46.123218+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:46.123218+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7869,"log_dups_size":0,"ondisk_log_size":7869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42299249,"num_objects":5164,"num_object_clones":0,"num_object_copies":20656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7866,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7896","reported_seq":7965,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:45.988263+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:45.988263+0000","last_peered":"2024-09-18T11:49:45.988263+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:45.988263+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42064507,"num_objects":5135,"num_object_clones":0,"num_object_copies":20540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7895,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3040}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7801","reported_seq":7889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.889676+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:46.889676+0000","last_peered":"2024-09-18T11:49:46.889676+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:46.889676+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42318459,"num_objects":5166,"num_object_clones":0,"num_object_copies":20664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3116}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7975","reported_seq":8068,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.066196+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:46.066196+0000","last_peered":"2024-09-18T11:49:46.066196+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:46.066196+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41899254,"num_objects":5115,"num_object_clones":0,"num_object_copies":20460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7832","reported_seq":12119,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:46.210929+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:46.210929+0000","last_peered":"2024-09-18T11:49:46.210929+0000","last_clean":"2024-09-18T11:49:46.210929+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:46.210929+0000","last_undegraded":"2024-09-18T11:49:46.210929+0000","last_fullsized":"2024-09-18T11:49:46.210929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7832,"log_dups_size":0,"ondisk_log_size":7832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41538806,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7701","reported_seq":7779,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.178221+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:47.178221+0000","last_peered":"2024-09-18T11:49:47.178221+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:47.178221+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006326,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7699,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":666830220,"num_objects":81406,"num_object_clones":0,"num_object_copies":325624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81406,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125681,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":34971648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1292546048,"data_stored":1292546048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125708,"ondisk_log_size":125708,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637000,"kb_used_data":326652,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93734840,"statfs":{"total":96636764160,"available":95984476160,"internally_reserved":0,"allocated":334491648,"data_stored":334316182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,0,0,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.39800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.232},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":1.8260000000000001},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.38700000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627332,"kb_used_data":325820,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93744508,"statfs":{"total":96636764160,"available":95994376192,"internally_reserved":0,"allocated":333639680,"data_stored":333464214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.9260000000000002},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.9580000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.157},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":1.452}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":3.746},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.0650000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588528,"kb_used_data":286568,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93783312,"statfs":{"total":96636764160,"available":96034111488,"internally_reserved":0,"allocated":293445632,"data_stored":293270166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.238},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":0.95299999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.78000000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.0429999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623188,"kb_used_data":325388,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93748652,"statfs":{"total":96636764160,"available":95998619648,"internally_reserved":0,"allocated":333197312,"data_stored":333021846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":14.119},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.7460000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.7750000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":11.651999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":4.8049999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332746752,"data_stored":332746752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293048320,"data_stored":293048320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333004800,"data_stored":333004800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":333746176,"data_stored":333746176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:51.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:51.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8030", 'reported_seq': 8109, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:45.984810+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:45.984810+0000', 'last_peered': '2024-09-18T11:49:45.984810+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:45.984810+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8030, 'log_dups_size': 0, 'ondisk_log_size': 8030, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42762240, 'num_objects': 5220, 'num_object_clones': 0, 'num_object_copies': 20880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2113, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8030, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3107}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:51.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:51.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7782", 'reported_seq': 7891, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:46.965166+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:46.965166+0000', 'last_peered': '2024-09-18T11:49:46.965166+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:46.965166+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7782, 'log_dups_size': 0, 'ondisk_log_size': 7782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41171603, 'num_objects': 5027, 'num_object_clones': 0, 'num_object_copies': 20108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5027, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7781, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2985}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:51.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:51.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'7848", 'reported_seq': 7926, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:47.239127+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:47.239127+0000', 'last_peered': '2024-09-18T11:49:47.239127+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:47.239127+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7848, 'log_dups_size': 0, 'ondisk_log_size': 7848, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41596150, 'num_objects': 5078, 'num_object_clones': 0, 'num_object_copies': 20312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7846, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2970}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:51.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:51.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7838", 'reported_seq': 7931, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:46.081576+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:46.081576+0000', 'last_peered': '2024-09-18T11:49:46.081576+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:46.081576+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7838, 'log_dups_size': 0, 'ondisk_log_size': 7838, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41424118, 'num_objects': 5057, 'num_object_clones': 0, 'num_object_copies': 20228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2067, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7836, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2992}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:51.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:51.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7836", 'reported_seq': 7913, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:45.872732+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:45.872732+0000', 'last_peered': '2024-09-18T11:49:45.872732+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:45.872732+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7836, 'log_dups_size': 0, 'ondisk_log_size': 7836, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40550400, 'num_objects': 4950, 'num_object_clones': 0, 'num_object_copies': 19800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2139, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7836, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:51.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:51.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7953", 'reported_seq': 8044, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:47.282123+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:47.282123+0000', 'last_peered': '2024-09-18T11:49:47.282123+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:47.282123+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7953, 'log_dups_size': 0, 'ondisk_log_size': 7953, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41473270, 'num_objects': 5063, 'num_object_clones': 0, 'num_object_copies': 20252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7951, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2940}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:51.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:51.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7781", 'reported_seq': 7878, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:47.304221+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:47.304221+0000', 'last_peered': '2024-09-18T11:49:47.304221+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:47.304221+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7781, 'log_dups_size': 0, 'ondisk_log_size': 7781, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41727222, 'num_objects': 5094, 'num_object_clones': 0, 'num_object_copies': 20376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2021, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7779, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3075}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:51.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:51.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8015", 'reported_seq': 8268, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:46.174076+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:49:46.174076+0000', 'last_peered': '2024-09-18T11:49:46.174076+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:46.174076+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8015, 'log_dups_size': 0, 'ondisk_log_size': 8015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42273260, 'num_objects': 5161, 'num_object_clones': 0, 'num_object_copies': 20644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2042, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8012, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 622592, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:51.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:51.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7622", 'reported_seq': 7728, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:46.953909+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:46.953909+0000', 'last_peered': '2024-09-18T11:49:46.953909+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:46.953909+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7622, 'log_dups_size': 0, 'ondisk_log_size': 7622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40727798, 'num_objects': 4972, 'num_object_clones': 0, 'num_object_copies': 19888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1968, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7620, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3006}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:51.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:51.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7869", 'reported_seq': 7964, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:46.123218+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:46.123218+0000', 'last_peered': '2024-09-18T11:49:46.123218+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:46.123218+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7869, 'log_dups_size': 0, 'ondisk_log_size': 7869, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42299249, 'num_objects': 5164, 'num_object_clones': 0, 'num_object_copies': 20656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7866, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3118}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:51.788 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:51.788 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7896", 'reported_seq': 7965, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:45.988263+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:45.988263+0000', 'last_peered': '2024-09-18T11:49:45.988263+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:45.988263+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7896, 'log_dups_size': 0, 'ondisk_log_size': 7896, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42064507, 'num_objects': 5135, 'num_object_clones': 0, 'num_object_copies': 20540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2096, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7895, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3040}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:51.788 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:51.788 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7801", 'reported_seq': 7889, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:46.889676+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:46.889676+0000', 'last_peered': '2024-09-18T11:49:46.889676+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:46.889676+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7801, 'log_dups_size': 0, 'ondisk_log_size': 7801, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42318459, 'num_objects': 5166, 'num_object_clones': 0, 'num_object_copies': 20664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7800, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3116}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:51.788 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:51.788 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7975", 'reported_seq': 8068, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:46.066196+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:46.066196+0000', 'last_peered': '2024-09-18T11:49:46.066196+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:46.066196+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7975, 'log_dups_size': 0, 'ondisk_log_size': 7975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41899254, 'num_objects': 5115, 'num_object_clones': 0, 'num_object_copies': 20460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5115, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7973, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:51.788 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:51.788 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7701", 'reported_seq': 7779, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:47.178221+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:47.178221+0000', 'last_peered': '2024-09-18T11:49:47.178221+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:47.178221+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7701, 'log_dups_size': 0, 'ondisk_log_size': 7701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41006326, 'num_objects': 5006, 'num_object_clones': 0, 'num_object_copies': 20024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7699, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2999}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:51.788 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:52.489 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:52.489 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28919/322784 objects degraded (8.959%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":390,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":80696,"data_bytes":661013900,"bytes_used":2538553344,"bytes_avail":384008503296,"bytes_total":386547056640,"degraded_objects":28919,"degraded_total":322784,"degraded_ratio":0.089592420937840783,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":103973,"recovering_keys_per_sec":0,"num_objects_recovered":137,"num_bytes_recovered":1040384,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":238},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:52.512 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:52.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:53.060 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:53.061 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:53.089 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-18T11:49:51.959743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661013900,"num_objects":80696,"num_object_clones":0,"num_object_copies":322784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80696,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126391,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":35414016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126418,"ondisk_log_size":126418,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2479056,"kb_used_data":1259120,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375008304,"statfs":{"total":386547056640,"available":384008503296,"internally_reserved":0,"allocated":1289338880,"data_stored":1288637016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,9,13,7,1,0,0,0,0,0,4,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19562496,"num_objects":-2388,"num_object_clones":0,"num_object_copies":-9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2388,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006211"},"pg_stats":[{"pgid":"1.f","version":"35'8114","reported_seq":8193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.242221+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:51.242221+0000","last_peered":"2024-09-18T11:49:51.242221+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:51.242221+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8114,"log_dups_size":0,"ondisk_log_size":8114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42074112,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8114,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3023}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8006","reported_seq":12296,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:51.258857+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:49:51.258857+0000","last_peered":"2024-09-18T11:49:51.258857+0000","last_clean":"2024-09-18T11:49:51.258857+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:49:51.258857+0000","last_undegraded":"2024-09-18T11:49:51.258857+0000","last_fullsized":"2024-09-18T11:49:51.258857+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8006,"log_dups_size":0,"ondisk_log_size":8006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41366774,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8004,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'7782","reported_seq":7891,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.965166+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:46.965166+0000","last_peered":"2024-09-18T11:49:46.965166+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:46.965166+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7782,"log_dups_size":0,"ondisk_log_size":7782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41171603,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7781,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2985}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7848","reported_seq":7926,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.239127+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:47.239127+0000","last_peered":"2024-09-18T11:49:47.239127+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:47.239127+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7848,"log_dups_size":0,"ondisk_log_size":7848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41596150,"num_objects":5078,"num_object_clones":0,"num_object_copies":20312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'7922","reported_seq":8015,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.139712+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:51.139712+0000","last_peered":"2024-09-18T11:49:51.139712+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:51.139712+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7922,"log_dups_size":0,"ondisk_log_size":7922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40735990,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7920,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7907","reported_seq":7984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.172636+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:51.172636+0000","last_peered":"2024-09-18T11:49:51.172636+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:51.172636+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7907,"log_dups_size":0,"ondisk_log_size":7907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7953","reported_seq":8044,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.282123+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:47.282123+0000","last_peered":"2024-09-18T11:49:47.282123+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:47.282123+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7953,"log_dups_size":0,"ondisk_log_size":7953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41473270,"num_objects":5063,"num_object_clones":0,"num_object_copies":20252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7951,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2940}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7781","reported_seq":7878,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.304221+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:47.304221+0000","last_peered":"2024-09-18T11:49:47.304221+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:47.304221+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7781,"log_dups_size":0,"ondisk_log_size":7781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727222,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7779,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8101","reported_seq":8464,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.191695+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:49:51.191695+0000","last_peered":"2024-09-18T11:49:51.191695+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:51.191695+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8101,"log_dups_size":0,"ondisk_log_size":8101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41568748,"num_objects":5075,"num_object_clones":0,"num_object_copies":20300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1988,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8098,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7622","reported_seq":7728,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.953909+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:46.953909+0000","last_peered":"2024-09-18T11:49:46.953909+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:46.953909+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40727798,"num_objects":4972,"num_object_clones":0,"num_object_copies":19888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7620,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3006}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7945","reported_seq":8040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.192393+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:51.192393+0000","last_peered":"2024-09-18T11:49:51.192393+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:51.192393+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7945,"log_dups_size":0,"ondisk_log_size":7945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41676657,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7942,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3042}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7973","reported_seq":8042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.165686+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:51.165686+0000","last_peered":"2024-09-18T11:49:51.165686+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:51.165686+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7973,"log_dups_size":0,"ondisk_log_size":7973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41433723,"num_objects":5058,"num_object_clones":0,"num_object_copies":20232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7972,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7801","reported_seq":7889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:46.889676+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:46.889676+0000","last_peered":"2024-09-18T11:49:46.889676+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:46.889676+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42318459,"num_objects":5166,"num_object_clones":0,"num_object_copies":20664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3116}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8052","reported_seq":8145,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.108060+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:51.108060+0000","last_peered":"2024-09-18T11:49:51.108060+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:51.108060+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8052,"log_dups_size":0,"ondisk_log_size":8052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268470,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8050,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2900}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7910","reported_seq":12197,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:51.217266+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:51.217266+0000","last_peered":"2024-09-18T11:49:51.217266+0000","last_clean":"2024-09-18T11:49:51.217266+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:51.217266+0000","last_undegraded":"2024-09-18T11:49:51.217266+0000","last_fullsized":"2024-09-18T11:49:51.217266+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40899830,"num_objects":4993,"num_object_clones":0,"num_object_copies":19972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7701","reported_seq":7779,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:47.178221+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:47.178221+0000","last_peered":"2024-09-18T11:49:47.178221+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:47.178221+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006326,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7699,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":661013900,"num_objects":80696,"num_object_clones":0,"num_object_copies":322784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80696,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126391,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":35414016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1287073792,"data_stored":1287073792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126418,"ondisk_log_size":126418,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636396,"kb_used_data":321892,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93735444,"statfs":{"total":96636764160,"available":95985094656,"internally_reserved":0,"allocated":329617408,"data_stored":329441942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,4,0,2,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.7330000000000001},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":5.0999999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":2.7589999999999999},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":10.741}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":0.19500000000000001},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.2589999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627332,"kb_used_data":325820,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93744508,"statfs":{"total":96636764160,"available":95994376192,"internally_reserved":0,"allocated":333639680,"data_stored":333464214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.9260000000000002},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.9580000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.157},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":1.452}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":3.746},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.0650000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592140,"kb_used_data":286020,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93779700,"statfs":{"total":96636764160,"available":96030412800,"internally_reserved":0,"allocated":292884480,"data_stored":292709014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.0229999999999997},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":25.187000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.0570000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.23899999999999999},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.56699999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623188,"kb_used_data":325388,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93748652,"statfs":{"total":96636764160,"available":95998619648,"internally_reserved":0,"allocated":333197312,"data_stored":333021846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":14.119},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.7460000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.7750000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":11.651999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":4.8049999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332746752,"data_stored":332746752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292483072,"data_stored":292483072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333004800,"data_stored":333004800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328839168,"data_stored":328839168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:56.092 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:56.620 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:56.621 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:56.642 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-18T11:49:55.961241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657057164,"num_objects":80213,"num_object_clones":0,"num_object_copies":320852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126874,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4546,"num_bytes_recovered":35414016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126901,"ondisk_log_size":126901,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2477788,"kb_used_data":1249524,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375009572,"statfs":{"total":386547056640,"available":384009801728,"internally_reserved":0,"allocated":1279512576,"data_stored":1278810712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,6,11,14,8,1,0,0,0,2,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19562496,"num_objects":-2388,"num_object_clones":0,"num_object_copies":-9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2388,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007003"},"pg_stats":[{"pgid":"1.f","version":"35'8114","reported_seq":8193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.242221+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:51.242221+0000","last_peered":"2024-09-18T11:49:51.242221+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:51.242221+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8114,"log_dups_size":0,"ondisk_log_size":8114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42074112,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8114,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3023}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8006","reported_seq":12296,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:51.258857+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:49:51.258857+0000","last_peered":"2024-09-18T11:49:51.258857+0000","last_clean":"2024-09-18T11:49:51.258857+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:49:51.258857+0000","last_undegraded":"2024-09-18T11:49:51.258857+0000","last_fullsized":"2024-09-18T11:49:51.258857+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8006,"log_dups_size":0,"ondisk_log_size":8006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41366774,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8004,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'7843","reported_seq":7954,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.993854+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:51.993854+0000","last_peered":"2024-09-18T11:49:51.993854+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:51.993854+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40671891,"num_objects":4966,"num_object_clones":0,"num_object_copies":19864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7842,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2924}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7908","reported_seq":7990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:52.193395+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:52.193395+0000","last_peered":"2024-09-18T11:49:52.193395+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:52.193395+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7908,"log_dups_size":0,"ondisk_log_size":7908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41104630,"num_objects":5018,"num_object_clones":0,"num_object_copies":20072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7906,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2912}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'7922","reported_seq":8015,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.139712+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:51.139712+0000","last_peered":"2024-09-18T11:49:51.139712+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:51.139712+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7922,"log_dups_size":0,"ondisk_log_size":7922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40735990,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7920,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7907","reported_seq":7984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.172636+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:51.172636+0000","last_peered":"2024-09-18T11:49:51.172636+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:51.172636+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7907,"log_dups_size":0,"ondisk_log_size":7907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8020","reported_seq":8119,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:52.340125+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:52.340125+0000","last_peered":"2024-09-18T11:49:52.340125+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:52.340125+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8020,"log_dups_size":0,"ondisk_log_size":8020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40924406,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8018,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2877}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7862","reported_seq":7963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:52.293515+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:52.293515+0000","last_peered":"2024-09-18T11:49:52.293515+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:52.293515+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41063670,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7860,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2996}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8101","reported_seq":8464,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.191695+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:49:51.191695+0000","last_peered":"2024-09-18T11:49:51.191695+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:51.191695+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8101,"log_dups_size":0,"ondisk_log_size":8101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41568748,"num_objects":5075,"num_object_clones":0,"num_object_copies":20300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1988,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8098,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7691","reported_seq":7796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:52.032577+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:52.032577+0000","last_peered":"2024-09-18T11:49:52.032577+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:52.032577+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7691,"log_dups_size":0,"ondisk_log_size":7691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40162550,"num_objects":4903,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7689,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7945","reported_seq":8040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.192393+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:51.192393+0000","last_peered":"2024-09-18T11:49:51.192393+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:51.192393+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7945,"log_dups_size":0,"ondisk_log_size":7945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41676657,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7942,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3042}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7973","reported_seq":8042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.165686+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:51.165686+0000","last_peered":"2024-09-18T11:49:51.165686+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:51.165686+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7973,"log_dups_size":0,"ondisk_log_size":7973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41433723,"num_objects":5058,"num_object_clones":0,"num_object_copies":20232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7972,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7884","reported_seq":7972,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.891569+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:51.891569+0000","last_peered":"2024-09-18T11:49:51.891569+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:51.891569+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41638523,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8052","reported_seq":8145,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.108060+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:51.108060+0000","last_peered":"2024-09-18T11:49:51.108060+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:51.108060+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8052,"log_dups_size":0,"ondisk_log_size":8052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268470,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8050,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2900}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7910","reported_seq":12197,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:51.217266+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:51.217266+0000","last_peered":"2024-09-18T11:49:51.217266+0000","last_clean":"2024-09-18T11:49:51.217266+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:51.217266+0000","last_undegraded":"2024-09-18T11:49:51.217266+0000","last_fullsized":"2024-09-18T11:49:51.217266+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40899830,"num_objects":4993,"num_object_clones":0,"num_object_copies":19972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7763","reported_seq":7843,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:52.278345+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:52.278345+0000","last_peered":"2024-09-18T11:49:52.278345+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:52.278345+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7763,"log_dups_size":0,"ondisk_log_size":7763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40498422,"num_objects":4944,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7761,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":657057164,"num_objects":80213,"num_object_clones":0,"num_object_copies":320852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126874,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4546,"num_bytes_recovered":35414016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1277362176,"data_stored":1277362176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126901,"ondisk_log_size":126901,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636396,"kb_used_data":321892,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93735444,"statfs":{"total":96636764160,"available":95985094656,"internally_reserved":0,"allocated":329617408,"data_stored":329441942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,4,0,2,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.7330000000000001},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":5.0999999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":2.7589999999999999},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":10.741}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":0.19500000000000001},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.2589999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626644,"kb_used_data":320972,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93745196,"statfs":{"total":96636764160,"available":95995080704,"internally_reserved":0,"allocated":328675328,"data_stored":328499862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":3.8199999999999998},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.8420000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":10.688000000000001},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":3.7970000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":3.762},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.53800000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592140,"kb_used_data":286020,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93779700,"statfs":{"total":96636764160,"available":96030412800,"internally_reserved":0,"allocated":292884480,"data_stored":292709014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.0229999999999997},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":25.187000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.0570000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.23899999999999999},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.56699999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622608,"kb_used_data":320640,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93749232,"statfs":{"total":96636764160,"available":95999213568,"internally_reserved":0,"allocated":328335360,"data_stored":328159894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":10.028},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":2.6539999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.6850000000000001},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.867}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":2.9430000000000001},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":2.6819999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327888896,"data_stored":327888896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292483072,"data_stored":292483072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":328151040,"data_stored":328151040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328839168,"data_stored":328839168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:56.644 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:57.188 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:57.189 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:57.206 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-18T11:49:55.961241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657057164,"num_objects":80213,"num_object_clones":0,"num_object_copies":320852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126874,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4546,"num_bytes_recovered":35414016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126901,"ondisk_log_size":126901,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2477788,"kb_used_data":1249524,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375009572,"statfs":{"total":386547056640,"available":384009801728,"internally_reserved":0,"allocated":1279512576,"data_stored":1278810712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,6,11,14,8,1,0,0,0,2,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19562496,"num_objects":-2388,"num_object_clones":0,"num_object_copies":-9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2388,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007003"},"pg_stats":[{"pgid":"1.f","version":"35'8114","reported_seq":8193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.242221+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:51.242221+0000","last_peered":"2024-09-18T11:49:51.242221+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:51.242221+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8114,"log_dups_size":0,"ondisk_log_size":8114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42074112,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8114,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3023}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8006","reported_seq":12296,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:51.258857+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:49:51.258857+0000","last_peered":"2024-09-18T11:49:51.258857+0000","last_clean":"2024-09-18T11:49:51.258857+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:49:51.258857+0000","last_undegraded":"2024-09-18T11:49:51.258857+0000","last_fullsized":"2024-09-18T11:49:51.258857+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8006,"log_dups_size":0,"ondisk_log_size":8006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41366774,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8004,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'7843","reported_seq":7954,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.993854+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:51.993854+0000","last_peered":"2024-09-18T11:49:51.993854+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:51.993854+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40671891,"num_objects":4966,"num_object_clones":0,"num_object_copies":19864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7842,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2924}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7908","reported_seq":7990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:52.193395+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:52.193395+0000","last_peered":"2024-09-18T11:49:52.193395+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:52.193395+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7908,"log_dups_size":0,"ondisk_log_size":7908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41104630,"num_objects":5018,"num_object_clones":0,"num_object_copies":20072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7906,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2912}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'7922","reported_seq":8015,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.139712+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:51.139712+0000","last_peered":"2024-09-18T11:49:51.139712+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:51.139712+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7922,"log_dups_size":0,"ondisk_log_size":7922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40735990,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7920,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7907","reported_seq":7984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.172636+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:51.172636+0000","last_peered":"2024-09-18T11:49:51.172636+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:51.172636+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7907,"log_dups_size":0,"ondisk_log_size":7907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8020","reported_seq":8119,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:52.340125+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:52.340125+0000","last_peered":"2024-09-18T11:49:52.340125+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:52.340125+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8020,"log_dups_size":0,"ondisk_log_size":8020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40924406,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8018,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2877}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7862","reported_seq":7963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:52.293515+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:52.293515+0000","last_peered":"2024-09-18T11:49:52.293515+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:52.293515+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41063670,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7860,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2996}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8101","reported_seq":8464,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.191695+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:49:51.191695+0000","last_peered":"2024-09-18T11:49:51.191695+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:51.191695+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8101,"log_dups_size":0,"ondisk_log_size":8101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41568748,"num_objects":5075,"num_object_clones":0,"num_object_copies":20300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1988,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8098,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7691","reported_seq":7796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:52.032577+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:52.032577+0000","last_peered":"2024-09-18T11:49:52.032577+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:52.032577+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7691,"log_dups_size":0,"ondisk_log_size":7691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40162550,"num_objects":4903,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7689,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7945","reported_seq":8040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.192393+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:51.192393+0000","last_peered":"2024-09-18T11:49:51.192393+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:51.192393+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7945,"log_dups_size":0,"ondisk_log_size":7945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41676657,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7942,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3042}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7973","reported_seq":8042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.165686+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:51.165686+0000","last_peered":"2024-09-18T11:49:51.165686+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:51.165686+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7973,"log_dups_size":0,"ondisk_log_size":7973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41433723,"num_objects":5058,"num_object_clones":0,"num_object_copies":20232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7972,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7884","reported_seq":7972,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.891569+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:51.891569+0000","last_peered":"2024-09-18T11:49:51.891569+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:51.891569+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41638523,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8052","reported_seq":8145,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:51.108060+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:51.108060+0000","last_peered":"2024-09-18T11:49:51.108060+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:51.108060+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8052,"log_dups_size":0,"ondisk_log_size":8052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268470,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8050,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2900}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7910","reported_seq":12197,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:51.217266+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:51.217266+0000","last_peered":"2024-09-18T11:49:51.217266+0000","last_clean":"2024-09-18T11:49:51.217266+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:51.217266+0000","last_undegraded":"2024-09-18T11:49:51.217266+0000","last_fullsized":"2024-09-18T11:49:51.217266+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40899830,"num_objects":4993,"num_object_clones":0,"num_object_copies":19972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7763","reported_seq":7843,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:52.278345+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:52.278345+0000","last_peered":"2024-09-18T11:49:52.278345+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:52.278345+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7763,"log_dups_size":0,"ondisk_log_size":7763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40498422,"num_objects":4944,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7761,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":657057164,"num_objects":80213,"num_object_clones":0,"num_object_copies":320852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126874,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4546,"num_bytes_recovered":35414016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1277362176,"data_stored":1277362176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126901,"ondisk_log_size":126901,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636396,"kb_used_data":321892,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93735444,"statfs":{"total":96636764160,"available":95985094656,"internally_reserved":0,"allocated":329617408,"data_stored":329441942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,4,0,2,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.7330000000000001},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":5.0999999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.9089999999999998,"5min":5.4960000000000004,"15min":5.4870000000000001},"min":{"1min":0.55900000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":10.887,"5min":21.501000000000001,"15min":21.501000000000001},"last":2.7589999999999999},{"interface":"front","average":{"1min":5.702,"5min":4.407,"15min":4.2690000000000001},"min":{"1min":0.66600000000000004,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":14.118,"5min":21.132000000000001,"15min":21.132000000000001},"last":10.741}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":0.19500000000000001},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.2589999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626644,"kb_used_data":320972,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93745196,"statfs":{"total":96636764160,"available":95995080704,"internally_reserved":0,"allocated":328675328,"data_stored":328499862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":3.8199999999999998},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":3.8420000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:48:51 2024","interfaces":[{"interface":"back","average":{"1min":4.242,"5min":5.8449999999999998,"15min":5.9009999999999998},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.361000000000001,"5min":21.626999999999999,"15min":21.626999999999999},"last":10.688000000000001},{"interface":"front","average":{"1min":5.0670000000000002,"5min":4.0460000000000003,"15min":3.786},"min":{"1min":0.39200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"last":3.7970000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":3.762},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.53800000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592140,"kb_used_data":286020,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93779700,"statfs":{"total":96636764160,"available":96030412800,"internally_reserved":0,"allocated":292884480,"data_stored":292709014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.0229999999999997},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":25.187000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.0570000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.23899999999999999},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.56699999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622608,"kb_used_data":320640,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93749232,"statfs":{"total":96636764160,"available":95999213568,"internally_reserved":0,"allocated":328335360,"data_stored":328159894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:48:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":3.9199999999999999,"15min":3.7599999999999998},"min":{"1min":0.19,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":10.028},{"interface":"front","average":{"1min":4.0869999999999997,"5min":4.165,"15min":3.8959999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":14.557,"5min":19.077999999999999,"15min":19.077999999999999},"last":2.6539999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.6850000000000001},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.867}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":2.9430000000000001},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":2.6819999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327888896,"data_stored":327888896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292483072,"data_stored":292483072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":328151040,"data_stored":328151040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328839168,"data_stored":328839168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:49:57.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:49:57.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8114", 'reported_seq': 8193, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:51.242221+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:49:51.242221+0000', 'last_peered': '2024-09-18T11:49:51.242221+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:49:51.242221+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8114, 'log_dups_size': 0, 'ondisk_log_size': 8114, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42074112, 'num_objects': 5136, 'num_object_clones': 0, 'num_object_copies': 20544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2113, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8114, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3023}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:57.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:49:57.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7843", 'reported_seq': 7954, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:51.993854+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:51.993854+0000', 'last_peered': '2024-09-18T11:49:51.993854+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:51.993854+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7843, 'log_dups_size': 0, 'ondisk_log_size': 7843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40671891, 'num_objects': 4966, 'num_object_clones': 0, 'num_object_copies': 19864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4966, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7842, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2924}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:57.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:49:57.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'7908", 'reported_seq': 7990, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:52.193395+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:52.193395+0000', 'last_peered': '2024-09-18T11:49:52.193395+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:52.193395+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7908, 'log_dups_size': 0, 'ondisk_log_size': 7908, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41104630, 'num_objects': 5018, 'num_object_clones': 0, 'num_object_copies': 20072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2108, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7906, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2912}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:57.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:49:57.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7922", 'reported_seq': 8015, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:51.139712+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:49:51.139712+0000', 'last_peered': '2024-09-18T11:49:51.139712+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:49:51.139712+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7922, 'log_dups_size': 0, 'ondisk_log_size': 7922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40735990, 'num_objects': 4973, 'num_object_clones': 0, 'num_object_copies': 19892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2067, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7920, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2908}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:57.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:49:57.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7907", 'reported_seq': 7984, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:51.172636+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:49:51.172636+0000', 'last_peered': '2024-09-18T11:49:51.172636+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:49:51.172636+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7907, 'log_dups_size': 0, 'ondisk_log_size': 7907, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39968768, 'num_objects': 4879, 'num_object_clones': 0, 'num_object_copies': 19516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2139, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7907, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2740}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:57.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:49:57.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8020", 'reported_seq': 8119, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:52.340125+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:52.340125+0000', 'last_peered': '2024-09-18T11:49:52.340125+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:52.340125+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8020, 'log_dups_size': 0, 'ondisk_log_size': 8020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40924406, 'num_objects': 4996, 'num_object_clones': 0, 'num_object_copies': 19984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8018, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2877}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:57.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:49:57.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7862", 'reported_seq': 7963, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:52.293515+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:52.293515+0000', 'last_peered': '2024-09-18T11:49:52.293515+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:52.293515+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7862, 'log_dups_size': 0, 'ondisk_log_size': 7862, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41063670, 'num_objects': 5013, 'num_object_clones': 0, 'num_object_copies': 20052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2019, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7860, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2996}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:57.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:49:57.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8101", 'reported_seq': 8464, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:51.191695+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:49:51.191695+0000', 'last_peered': '2024-09-18T11:49:51.191695+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:49:51.191695+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8101, 'log_dups_size': 0, 'ondisk_log_size': 8101, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41568748, 'num_objects': 5075, 'num_object_clones': 0, 'num_object_copies': 20300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1988, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8098, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 153, 'num_bytes_recovered': 1064960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3087}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:57.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:49:57.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7691", 'reported_seq': 7796, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:52.032577+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:52.032577+0000', 'last_peered': '2024-09-18T11:49:52.032577+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:52.032577+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7691, 'log_dups_size': 0, 'ondisk_log_size': 7691, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40162550, 'num_objects': 4903, 'num_object_clones': 0, 'num_object_copies': 19612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1968, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7689, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2937}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:57.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:49:57.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7945", 'reported_seq': 8040, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:51.192393+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:49:51.192393+0000', 'last_peered': '2024-09-18T11:49:51.192393+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:49:51.192393+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7945, 'log_dups_size': 0, 'ondisk_log_size': 7945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41676657, 'num_objects': 5088, 'num_object_clones': 0, 'num_object_copies': 20352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7942, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3042}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:57.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:49:57.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7973", 'reported_seq': 8042, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:51.165686+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:49:51.165686+0000', 'last_peered': '2024-09-18T11:49:51.165686+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:49:51.165686+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7973, 'log_dups_size': 0, 'ondisk_log_size': 7973, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41433723, 'num_objects': 5058, 'num_object_clones': 0, 'num_object_copies': 20232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2096, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7972, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2963}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:57.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:49:57.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7884", 'reported_seq': 7972, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:51.891569+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:51.891569+0000', 'last_peered': '2024-09-18T11:49:51.891569+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:51.891569+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7884, 'log_dups_size': 0, 'ondisk_log_size': 7884, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41638523, 'num_objects': 5083, 'num_object_clones': 0, 'num_object_copies': 20332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7883, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3033}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:49:57.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:49:57.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8052", 'reported_seq': 8145, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:51.108060+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:49:51.108060+0000', 'last_peered': '2024-09-18T11:49:51.108060+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:49:51.108060+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8052, 'log_dups_size': 0, 'ondisk_log_size': 8052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41268470, 'num_objects': 5038, 'num_object_clones': 0, 'num_object_copies': 20152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8050, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:49:57.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:49:57.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7763", 'reported_seq': 7843, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:52.278345+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:52.278345+0000', 'last_peered': '2024-09-18T11:49:52.278345+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:52.278345+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7763, 'log_dups_size': 0, 'ondisk_log_size': 7763, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40498422, 'num_objects': 4944, 'num_object_clones': 0, 'num_object_copies': 19776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2008, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7761, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:49:57.211 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:49:57.864 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:57.864 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28910/320852 objects degraded (9.010%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":395,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":80213,"data_bytes":657057164,"bytes_used":2537254912,"bytes_avail":384009801728,"bytes_total":386547056640,"degraded_objects":28910,"degraded_total":320852,"degraded_ratio":0.090103848503359799,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":86648,"recovering_keys_per_sec":0,"num_objects_recovered":145,"num_bytes_recovered":1040384,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":198},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:49:57.892 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:49:57.892 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:49:58.405 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:49:58.406 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:49:58.427 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-18T11:49:57.961938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":647685516,"num_objects":79069,"num_object_clones":0,"num_object_copies":316276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128018,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4620,"num_bytes_recovered":35913728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128045,"ondisk_log_size":128045,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2463336,"kb_used_data":1235060,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375024024,"statfs":{"total":386547056640,"available":384024600576,"internally_reserved":0,"allocated":1264701440,"data_stored":1263999576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,7,16,5,2,5,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28934144,"num_objects":-3532,"num_object_clones":0,"num_object_copies":-14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3532,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003809"},"pg_stats":[{"pgid":"1.f","version":"35'8182","reported_seq":8265,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.292801+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:49:56.292801+0000","last_peered":"2024-09-18T11:49:56.292801+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:49:56.292801+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8182,"log_dups_size":0,"ondisk_log_size":8182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41517056,"num_objects":5068,"num_object_clones":0,"num_object_copies":20272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8182,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2957}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8083","reported_seq":12373,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:56.257222+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:49:56.257222+0000","last_peered":"2024-09-18T11:49:56.257222+0000","last_clean":"2024-09-18T11:49:56.257222+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:49:56.257222+0000","last_undegraded":"2024-09-18T11:49:56.257222+0000","last_fullsized":"2024-09-18T11:49:56.257222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8083,"log_dups_size":0,"ondisk_log_size":8083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40735990,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8081,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'7910","reported_seq":8021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.026042+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:57.026042+0000","last_peered":"2024-09-18T11:49:57.026042+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:57.026042+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40123027,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7909,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2857}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7999","reported_seq":8081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.300267+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:57.300267+0000","last_peered":"2024-09-18T11:49:57.300267+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:57.300267+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7999,"log_dups_size":0,"ondisk_log_size":7999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40359158,"num_objects":4927,"num_object_clones":0,"num_object_copies":19708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7997,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'7994","reported_seq":8091,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.196672+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:49:56.196672+0000","last_peered":"2024-09-18T11:49:56.196672+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:49:56.196672+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7994,"log_dups_size":0,"ondisk_log_size":7994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40146166,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7984","reported_seq":8069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.288887+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:49:56.288887+0000","last_peered":"2024-09-18T11:49:56.288887+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:49:56.288887+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7984,"log_dups_size":0,"ondisk_log_size":7984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39337984,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7984,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8086","reported_seq":8185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.188201+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:57.188201+0000","last_peered":"2024-09-18T11:49:57.188201+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:57.188201+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40383734,"num_objects":4930,"num_object_clones":0,"num_object_copies":19720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8084,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2811}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7935","reported_seq":8036,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.210684+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:57.210684+0000","last_peered":"2024-09-18T11:49:57.210684+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:57.210684+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40465654,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2923}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8171","reported_seq":8658,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.258802+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:49:56.258802+0000","last_peered":"2024-09-18T11:49:56.258802+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:49:56.258802+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8171,"log_dups_size":0,"ondisk_log_size":8171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40995308,"num_objects":5005,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8168,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":1564672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7762","reported_seq":7868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.863168+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:56.863168+0000","last_peered":"2024-09-18T11:49:56.863168+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:56.863168+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39580918,"num_objects":4832,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7760,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8026","reported_seq":8125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.289566+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:49:56.289566+0000","last_peered":"2024-09-18T11:49:56.289566+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:49:56.289566+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8026,"log_dups_size":0,"ondisk_log_size":8026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41013105,"num_objects":5007,"num_object_clones":0,"num_object_copies":20028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8023,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8035","reported_seq":8104,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.097140+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:49:56.097140+0000","last_peered":"2024-09-18T11:49:56.097140+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:49:56.097140+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8035,"log_dups_size":0,"ondisk_log_size":8035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40925819,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8034,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2901}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7942","reported_seq":8030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.823217+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:56.823217+0000","last_peered":"2024-09-18T11:49:56.823217+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:56.823217+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41163387,"num_objects":5025,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8119","reported_seq":8216,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:55.915393+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:49:55.915393+0000","last_peered":"2024-09-18T11:49:55.915393+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:49:55.915393+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8119,"log_dups_size":0,"ondisk_log_size":8119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719606,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2835}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7976","reported_seq":12263,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:49:56.345849+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:49:56.345849+0000","last_peered":"2024-09-18T11:49:56.345849+0000","last_clean":"2024-09-18T11:49:56.345849+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:49:56.345849+0000","last_undegraded":"2024-09-18T11:49:56.345849+0000","last_fullsized":"2024-09-18T11:49:56.345849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7976,"log_dups_size":0,"ondisk_log_size":7976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40359158,"num_objects":4927,"num_object_clones":0,"num_object_copies":19708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7974,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7841","reported_seq":7921,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.364716+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:57.364716+0000","last_peered":"2024-09-18T11:49:57.364716+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:57.364716+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7841,"log_dups_size":0,"ondisk_log_size":7841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39859446,"num_objects":4866,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7839,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":647685516,"num_objects":79069,"num_object_clones":0,"num_object_copies":316276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128018,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4620,"num_bytes_recovered":35913728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1262608384,"data_stored":1262608384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128045,"ondisk_log_size":128045,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619240,"kb_used_data":317216,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93752600,"statfs":{"total":96636764160,"available":96002662400,"internally_reserved":0,"allocated":324829184,"data_stored":324653718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,1,2,0,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":4.8339999999999996},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":8.5429999999999993}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.375},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":9.0790000000000006}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":0.39000000000000001},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":9.1150000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626240,"kb_used_data":316392,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93745600,"statfs":{"total":96636764160,"available":95995494400,"internally_reserved":0,"allocated":323985408,"data_stored":323809942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.498}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":3.944},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.5680000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.52900000000000003},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.55500000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595744,"kb_used_data":285464,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93776096,"statfs":{"total":96636764160,"available":96026722304,"internally_reserved":0,"allocated":292315136,"data_stored":292139670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":3.8650000000000002,"15min":3.4809999999999999},"min":{"1min":0.24299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":8.1750000000000007,"5min":45.462000000000003,"15min":45.462000000000003},"last":21.030999999999999},{"interface":"front","average":{"1min":5.3330000000000002,"5min":4.798,"15min":5.0010000000000003},"min":{"1min":0.19400000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":22.608000000000001,"5min":22.608000000000001,"15min":22.608000000000001},"last":21.073}]},{"osd":2,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.2590000000000003,"5min":4.9199999999999999,"15min":4.2610000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.877000000000001,"5min":24.600000000000001,"15min":24.600000000000001},"last":19.382999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":4.2670000000000003,"15min":3.9670000000000001},"min":{"1min":0.24099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":13.706,"5min":26.719000000000001,"15min":26.719000000000001},"last":6.8879999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:48:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5170000000000003,"5min":4.3129999999999997,"15min":4.2249999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":17.390999999999998,"5min":17.390999999999998,"15min":17.390999999999998},"last":9.093},{"interface":"front","average":{"1min":6.0960000000000001,"5min":5.2850000000000001,"15min":4.8369999999999997},"min":{"1min":0.222,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":20.361000000000001,"5min":25.440000000000001,"15min":25.440000000000001},"last":19.280000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622112,"kb_used_data":315988,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93749728,"statfs":{"total":96636764160,"available":95999721472,"internally_reserved":0,"allocated":323571712,"data_stored":323396246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":3.0630000000000002},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.6219999999999999},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.3479999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.1480000000000001},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323166208,"data_stored":323166208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291913728,"data_stored":291913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323424256,"data_stored":323424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":324104192,"data_stored":324104192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:01.430 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:01.987 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:01.987 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:02.011 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-18T11:50:01.963471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642090380,"num_objects":78386,"num_object_clones":0,"num_object_copies":313544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128701,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4673,"num_bytes_recovered":36282368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128728,"ondisk_log_size":128728,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2449660,"kb_used_data":1229696,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375037700,"statfs":{"total":386547056640,"available":384038604800,"internally_reserved":0,"allocated":1259208704,"data_stored":1258506840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,8,13,15,3,7,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24739840,"num_objects":-3020,"num_object_clones":0,"num_object_copies":-12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004271"},"pg_stats":[{"pgid":"1.f","version":"35'8244","reported_seq":8327,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.141324+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:01.141324+0000","last_peered":"2024-09-18T11:50:01.141324+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:01.141324+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41009152,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2895}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8160","reported_seq":12450,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:01.328708+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:01.328708+0000","last_peered":"2024-09-18T11:50:01.328708+0000","last_clean":"2024-09-18T11:50:01.328708+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:01.328708+0000","last_undegraded":"2024-09-18T11:50:01.328708+0000","last_fullsized":"2024-09-18T11:50:01.328708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40105206,"num_objects":4896,"num_object_clones":0,"num_object_copies":19584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'7910","reported_seq":8021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.026042+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:57.026042+0000","last_peered":"2024-09-18T11:49:57.026042+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:57.026042+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40123027,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7909,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2857}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7999","reported_seq":8081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.300267+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:57.300267+0000","last_peered":"2024-09-18T11:49:57.300267+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:57.300267+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7999,"log_dups_size":0,"ondisk_log_size":7999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40359158,"num_objects":4927,"num_object_clones":0,"num_object_copies":19708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7997,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8075","reported_seq":8174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.085230+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:01.085230+0000","last_peered":"2024-09-18T11:50:01.085230+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:01.085230+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8075,"log_dups_size":0,"ondisk_log_size":8075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39482614,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8056","reported_seq":8147,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.219088+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:01.219088+0000","last_peered":"2024-09-18T11:50:01.219088+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:01.219088+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8056,"log_dups_size":0,"ondisk_log_size":8056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38748160,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8086","reported_seq":8185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.188201+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:57.188201+0000","last_peered":"2024-09-18T11:49:57.188201+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:57.188201+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40383734,"num_objects":4930,"num_object_clones":0,"num_object_copies":19720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8084,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2811}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7935","reported_seq":8036,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.210684+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:57.210684+0000","last_peered":"2024-09-18T11:49:57.210684+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:57.210684+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40465654,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2923}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8265","reported_seq":8842,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.357757+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:01.357757+0000","last_peered":"2024-09-18T11:50:01.357757+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:01.357757+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8265,"log_dups_size":0,"ondisk_log_size":8265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40225260,"num_objects":4911,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":1933312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7762","reported_seq":7868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.863168+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:56.863168+0000","last_peered":"2024-09-18T11:49:56.863168+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:56.863168+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39580918,"num_objects":4832,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7760,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8096","reported_seq":8195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.236623+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:01.236623+0000","last_peered":"2024-09-18T11:50:01.236623+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:01.236623+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40439665,"num_objects":4937,"num_object_clones":0,"num_object_copies":19748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8093,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8130","reported_seq":8201,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.350039+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:01.350039+0000","last_peered":"2024-09-18T11:50:01.350039+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:01.350039+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8130,"log_dups_size":0,"ondisk_log_size":8130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40147579,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7942","reported_seq":8030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.823217+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:56.823217+0000","last_peered":"2024-09-18T11:49:56.823217+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:56.823217+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41163387,"num_objects":5025,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8174","reported_seq":8277,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.289932+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:01.289932+0000","last_peered":"2024-09-18T11:50:01.289932+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:01.289932+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40269046,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8172,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2783}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8053","reported_seq":12340,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:01.373233+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:01.373233+0000","last_peered":"2024-09-18T11:50:01.373233+0000","last_clean":"2024-09-18T11:50:01.373233+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:01.373233+0000","last_undegraded":"2024-09-18T11:50:01.373233+0000","last_fullsized":"2024-09-18T11:50:01.373233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8053,"log_dups_size":0,"ondisk_log_size":8053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728374,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8051,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7841","reported_seq":7921,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.364716+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:57.364716+0000","last_peered":"2024-09-18T11:49:57.364716+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:57.364716+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7841,"log_dups_size":0,"ondisk_log_size":7841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39859446,"num_objects":4866,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7839,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":642090380,"num_objects":78386,"num_object_clones":0,"num_object_copies":313544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128701,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4673,"num_bytes_recovered":36282368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1257172992,"data_stored":1257172992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128728,"ondisk_log_size":128728,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618592,"kb_used_data":312408,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93753248,"statfs":{"total":96636764160,"available":96003325952,"internally_reserved":0,"allocated":319905792,"data_stored":319730326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,10,3,2,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.923},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.6150000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":7.6950000000000003},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":5.2619999999999996},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.30199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626240,"kb_used_data":316392,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93745600,"statfs":{"total":96636764160,"available":95995494400,"internally_reserved":0,"allocated":323985408,"data_stored":323809942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.498}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":3.944},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.5680000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.52900000000000003},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.55500000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582716,"kb_used_data":284908,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93789124,"statfs":{"total":96636764160,"available":96040062976,"internally_reserved":0,"allocated":291745792,"data_stored":291570326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":6.7560000000000002},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.1210000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":15.701000000000001},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":8.3109999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":16.193000000000001},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":15.782999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622112,"kb_used_data":315988,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93749728,"statfs":{"total":96636764160,"available":95999721472,"internally_reserved":0,"allocated":323571712,"data_stored":323396246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":3.0630000000000002},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.6219999999999999},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.3479999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.1480000000000001},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323166208,"data_stored":323166208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291364864,"data_stored":291364864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323424256,"data_stored":323424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319217664,"data_stored":319217664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:02.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:02.616 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:02.617 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:02.643 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-18T11:50:01.963471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642090380,"num_objects":78386,"num_object_clones":0,"num_object_copies":313544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128701,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4673,"num_bytes_recovered":36282368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128728,"ondisk_log_size":128728,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2449660,"kb_used_data":1229696,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375037700,"statfs":{"total":386547056640,"available":384038604800,"internally_reserved":0,"allocated":1259208704,"data_stored":1258506840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,8,13,15,3,7,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24739840,"num_objects":-3020,"num_object_clones":0,"num_object_copies":-12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004271"},"pg_stats":[{"pgid":"1.f","version":"35'8244","reported_seq":8327,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.141324+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:01.141324+0000","last_peered":"2024-09-18T11:50:01.141324+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:01.141324+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41009152,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2895}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8160","reported_seq":12450,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:01.328708+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:01.328708+0000","last_peered":"2024-09-18T11:50:01.328708+0000","last_clean":"2024-09-18T11:50:01.328708+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:01.328708+0000","last_undegraded":"2024-09-18T11:50:01.328708+0000","last_fullsized":"2024-09-18T11:50:01.328708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40105206,"num_objects":4896,"num_object_clones":0,"num_object_copies":19584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'7910","reported_seq":8021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.026042+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:57.026042+0000","last_peered":"2024-09-18T11:49:57.026042+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:57.026042+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40123027,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7909,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2857}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7999","reported_seq":8081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.300267+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:57.300267+0000","last_peered":"2024-09-18T11:49:57.300267+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:57.300267+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7999,"log_dups_size":0,"ondisk_log_size":7999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40359158,"num_objects":4927,"num_object_clones":0,"num_object_copies":19708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7997,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8075","reported_seq":8174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.085230+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:01.085230+0000","last_peered":"2024-09-18T11:50:01.085230+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:01.085230+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8075,"log_dups_size":0,"ondisk_log_size":8075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39482614,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8056","reported_seq":8147,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.219088+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:01.219088+0000","last_peered":"2024-09-18T11:50:01.219088+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:01.219088+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8056,"log_dups_size":0,"ondisk_log_size":8056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38748160,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8086","reported_seq":8185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.188201+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:57.188201+0000","last_peered":"2024-09-18T11:49:57.188201+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:57.188201+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40383734,"num_objects":4930,"num_object_clones":0,"num_object_copies":19720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8084,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2811}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7935","reported_seq":8036,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.210684+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:57.210684+0000","last_peered":"2024-09-18T11:49:57.210684+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:57.210684+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40465654,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2923}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8265","reported_seq":8842,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.357757+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:01.357757+0000","last_peered":"2024-09-18T11:50:01.357757+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:01.357757+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8265,"log_dups_size":0,"ondisk_log_size":8265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40225260,"num_objects":4911,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":1933312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7762","reported_seq":7868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.863168+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:56.863168+0000","last_peered":"2024-09-18T11:49:56.863168+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:56.863168+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39580918,"num_objects":4832,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7760,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8096","reported_seq":8195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.236623+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:01.236623+0000","last_peered":"2024-09-18T11:50:01.236623+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:01.236623+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40439665,"num_objects":4937,"num_object_clones":0,"num_object_copies":19748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8093,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8130","reported_seq":8201,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.350039+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:01.350039+0000","last_peered":"2024-09-18T11:50:01.350039+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:01.350039+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8130,"log_dups_size":0,"ondisk_log_size":8130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40147579,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7942","reported_seq":8030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.823217+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:56.823217+0000","last_peered":"2024-09-18T11:49:56.823217+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:56.823217+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41163387,"num_objects":5025,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8174","reported_seq":8277,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.289932+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:01.289932+0000","last_peered":"2024-09-18T11:50:01.289932+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:01.289932+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40269046,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8172,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2783}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8053","reported_seq":12340,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:01.373233+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:01.373233+0000","last_peered":"2024-09-18T11:50:01.373233+0000","last_clean":"2024-09-18T11:50:01.373233+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:01.373233+0000","last_undegraded":"2024-09-18T11:50:01.373233+0000","last_fullsized":"2024-09-18T11:50:01.373233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8053,"log_dups_size":0,"ondisk_log_size":8053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728374,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8051,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7841","reported_seq":7921,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.364716+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:57.364716+0000","last_peered":"2024-09-18T11:49:57.364716+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:57.364716+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7841,"log_dups_size":0,"ondisk_log_size":7841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39859446,"num_objects":4866,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7839,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":642090380,"num_objects":78386,"num_object_clones":0,"num_object_copies":313544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128701,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4673,"num_bytes_recovered":36282368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1257172992,"data_stored":1257172992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128728,"ondisk_log_size":128728,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618592,"kb_used_data":312408,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93753248,"statfs":{"total":96636764160,"available":96003325952,"internally_reserved":0,"allocated":319905792,"data_stored":319730326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,10,3,2,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.923},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.6150000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":7.6950000000000003},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":5.2619999999999996},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.30199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626240,"kb_used_data":316392,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93745600,"statfs":{"total":96636764160,"available":95995494400,"internally_reserved":0,"allocated":323985408,"data_stored":323809942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.498}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":3.944},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.5680000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.52900000000000003},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.55500000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582716,"kb_used_data":284908,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93789124,"statfs":{"total":96636764160,"available":96040062976,"internally_reserved":0,"allocated":291745792,"data_stored":291570326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":6.7560000000000002},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.1210000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":15.701000000000001},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":8.3109999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":16.193000000000001},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":15.782999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622112,"kb_used_data":315988,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93749728,"statfs":{"total":96636764160,"available":95999721472,"internally_reserved":0,"allocated":323571712,"data_stored":323396246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":3.0630000000000002},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.6219999999999999},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.3479999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.1480000000000001},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323166208,"data_stored":323166208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291364864,"data_stored":291364864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323424256,"data_stored":323424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319217664,"data_stored":319217664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:02.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:02.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8244", 'reported_seq': 8327, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:01.141324+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:01.141324+0000', 'last_peered': '2024-09-18T11:50:01.141324+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:01.141324+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8244, 'log_dups_size': 0, 'ondisk_log_size': 8244, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41009152, 'num_objects': 5006, 'num_object_clones': 0, 'num_object_copies': 20024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2111, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8244, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2895}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:02.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:02.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7910", 'reported_seq': 8021, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:57.026042+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:49:57.026042+0000', 'last_peered': '2024-09-18T11:49:57.026042+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:49:57.026042+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7910, 'log_dups_size': 0, 'ondisk_log_size': 7910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40123027, 'num_objects': 4899, 'num_object_clones': 0, 'num_object_copies': 19596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4899, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7909, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2857}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:02.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:02.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'7999", 'reported_seq': 8081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:57.300267+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:49:57.300267+0000', 'last_peered': '2024-09-18T11:49:57.300267+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:49:57.300267+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7999, 'log_dups_size': 0, 'ondisk_log_size': 7999, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40359158, 'num_objects': 4927, 'num_object_clones': 0, 'num_object_copies': 19708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2108, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7997, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2821}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:02.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:02.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8075", 'reported_seq': 8174, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:01.085230+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:01.085230+0000', 'last_peered': '2024-09-18T11:50:01.085230+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:01.085230+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8075, 'log_dups_size': 0, 'ondisk_log_size': 8075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39482614, 'num_objects': 4820, 'num_object_clones': 0, 'num_object_copies': 19280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2064, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8073, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2758}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:02.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:02.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8056", 'reported_seq': 8147, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:01.219088+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:01.219088+0000', 'last_peered': '2024-09-18T11:50:01.219088+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:01.219088+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8056, 'log_dups_size': 0, 'ondisk_log_size': 8056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38748160, 'num_objects': 4730, 'num_object_clones': 0, 'num_object_copies': 18920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8056, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2598}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:02.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:02.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8086", 'reported_seq': 8185, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:57.188201+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:49:57.188201+0000', 'last_peered': '2024-09-18T11:49:57.188201+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:49:57.188201+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8086, 'log_dups_size': 0, 'ondisk_log_size': 8086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40383734, 'num_objects': 4930, 'num_object_clones': 0, 'num_object_copies': 19720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8084, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2811}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:02.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:02.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7935", 'reported_seq': 8036, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:57.210684+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:49:57.210684+0000', 'last_peered': '2024-09-18T11:49:57.210684+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:49:57.210684+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7935, 'log_dups_size': 0, 'ondisk_log_size': 7935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40465654, 'num_objects': 4940, 'num_object_clones': 0, 'num_object_copies': 19760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2019, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7933, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2923}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:02.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:02.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8265", 'reported_seq': 8842, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:01.357757+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:01.357757+0000', 'last_peered': '2024-09-18T11:50:01.357757+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:01.357757+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8265, 'log_dups_size': 0, 'ondisk_log_size': 8265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40225260, 'num_objects': 4911, 'num_object_clones': 0, 'num_object_copies': 19644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8262, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 260, 'num_bytes_recovered': 1933312, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3030}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:02.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:02.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7762", 'reported_seq': 7868, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:56.863168+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:49:56.863168+0000', 'last_peered': '2024-09-18T11:49:56.863168+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:49:56.863168+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7762, 'log_dups_size': 0, 'ondisk_log_size': 7762, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39580918, 'num_objects': 4832, 'num_object_clones': 0, 'num_object_copies': 19328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1968, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7760, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2866}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:02.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:02.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8096", 'reported_seq': 8195, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:01.236623+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:01.236623+0000', 'last_peered': '2024-09-18T11:50:01.236623+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:01.236623+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8096, 'log_dups_size': 0, 'ondisk_log_size': 8096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40439665, 'num_objects': 4937, 'num_object_clones': 0, 'num_object_copies': 19748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8093, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2893}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:02.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:02.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8130", 'reported_seq': 8201, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:01.350039+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:01.350039+0000', 'last_peered': '2024-09-18T11:50:01.350039+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:01.350039+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8130, 'log_dups_size': 0, 'ondisk_log_size': 8130, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40147579, 'num_objects': 4901, 'num_object_clones': 0, 'num_object_copies': 19604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8129, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2807}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:02.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:02.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7942", 'reported_seq': 8030, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:56.823217+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:49:56.823217+0000', 'last_peered': '2024-09-18T11:49:56.823217+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:49:56.823217+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7942, 'log_dups_size': 0, 'ondisk_log_size': 7942, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41163387, 'num_objects': 5025, 'num_object_clones': 0, 'num_object_copies': 20100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7941, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:02.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:02.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8174", 'reported_seq': 8277, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:01.289932+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:01.289932+0000', 'last_peered': '2024-09-18T11:50:01.289932+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:01.289932+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8174, 'log_dups_size': 0, 'ondisk_log_size': 8174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40269046, 'num_objects': 4916, 'num_object_clones': 0, 'num_object_copies': 19664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2135, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8172, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:02.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:02.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7841", 'reported_seq': 7921, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:49:57.364716+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:49:57.364716+0000', 'last_peered': '2024-09-18T11:49:57.364716+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:49:57.364716+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7841, 'log_dups_size': 0, 'ondisk_log_size': 7841, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39859446, 'num_objects': 4866, 'num_object_clones': 0, 'num_object_copies': 19464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2008, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7839, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:02.648 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:03.281 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:03.281 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28783/313544 objects degraded (9.180%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":401,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":78386,"data_bytes":642090380,"bytes_used":2508451840,"bytes_avail":384038604800,"bytes_total":386547056640,"degraded_objects":28783,"degraded_total":313544,"degraded_ratio":0.09179891817416376,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":109187,"recovering_keys_per_sec":0,"num_objects_recovered":190,"num_bytes_recovered":1310720,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":251},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:03.303 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:03.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:03.832 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:03.832 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:03.855 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-18T11:50:01.963471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642090380,"num_objects":78386,"num_object_clones":0,"num_object_copies":313544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128701,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4673,"num_bytes_recovered":36282368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128728,"ondisk_log_size":128728,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2449660,"kb_used_data":1229696,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375037700,"statfs":{"total":386547056640,"available":384038604800,"internally_reserved":0,"allocated":1259208704,"data_stored":1258506840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,8,13,15,3,7,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24739840,"num_objects":-3020,"num_object_clones":0,"num_object_copies":-12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004271"},"pg_stats":[{"pgid":"1.f","version":"35'8244","reported_seq":8327,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.141324+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:01.141324+0000","last_peered":"2024-09-18T11:50:01.141324+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:01.141324+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41009152,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2895}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8160","reported_seq":12450,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:01.328708+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:01.328708+0000","last_peered":"2024-09-18T11:50:01.328708+0000","last_clean":"2024-09-18T11:50:01.328708+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:01.328708+0000","last_undegraded":"2024-09-18T11:50:01.328708+0000","last_fullsized":"2024-09-18T11:50:01.328708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40105206,"num_objects":4896,"num_object_clones":0,"num_object_copies":19584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'7910","reported_seq":8021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.026042+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:49:57.026042+0000","last_peered":"2024-09-18T11:49:57.026042+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:49:57.026042+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40123027,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7909,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2857}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7999","reported_seq":8081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.300267+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:49:57.300267+0000","last_peered":"2024-09-18T11:49:57.300267+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:49:57.300267+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7999,"log_dups_size":0,"ondisk_log_size":7999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40359158,"num_objects":4927,"num_object_clones":0,"num_object_copies":19708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7997,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8075","reported_seq":8174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.085230+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:01.085230+0000","last_peered":"2024-09-18T11:50:01.085230+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:01.085230+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8075,"log_dups_size":0,"ondisk_log_size":8075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39482614,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8056","reported_seq":8147,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.219088+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:01.219088+0000","last_peered":"2024-09-18T11:50:01.219088+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:01.219088+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8056,"log_dups_size":0,"ondisk_log_size":8056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38748160,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8086","reported_seq":8185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.188201+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:49:57.188201+0000","last_peered":"2024-09-18T11:49:57.188201+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:49:57.188201+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40383734,"num_objects":4930,"num_object_clones":0,"num_object_copies":19720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8084,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2811}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7935","reported_seq":8036,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.210684+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:49:57.210684+0000","last_peered":"2024-09-18T11:49:57.210684+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:49:57.210684+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40465654,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2923}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8265","reported_seq":8842,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.357757+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:01.357757+0000","last_peered":"2024-09-18T11:50:01.357757+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:01.357757+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8265,"log_dups_size":0,"ondisk_log_size":8265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40225260,"num_objects":4911,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":1933312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7762","reported_seq":7868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.863168+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:49:56.863168+0000","last_peered":"2024-09-18T11:49:56.863168+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:49:56.863168+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39580918,"num_objects":4832,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7760,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8096","reported_seq":8195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.236623+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:01.236623+0000","last_peered":"2024-09-18T11:50:01.236623+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:01.236623+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40439665,"num_objects":4937,"num_object_clones":0,"num_object_copies":19748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8093,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8130","reported_seq":8201,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.350039+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:01.350039+0000","last_peered":"2024-09-18T11:50:01.350039+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:01.350039+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8130,"log_dups_size":0,"ondisk_log_size":8130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40147579,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7942","reported_seq":8030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:56.823217+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:49:56.823217+0000","last_peered":"2024-09-18T11:49:56.823217+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:49:56.823217+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41163387,"num_objects":5025,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8174","reported_seq":8277,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.289932+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:01.289932+0000","last_peered":"2024-09-18T11:50:01.289932+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:01.289932+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40269046,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8172,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2783}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8053","reported_seq":12340,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:01.373233+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:01.373233+0000","last_peered":"2024-09-18T11:50:01.373233+0000","last_clean":"2024-09-18T11:50:01.373233+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:01.373233+0000","last_undegraded":"2024-09-18T11:50:01.373233+0000","last_fullsized":"2024-09-18T11:50:01.373233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8053,"log_dups_size":0,"ondisk_log_size":8053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728374,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8051,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7841","reported_seq":7921,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:49:57.364716+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:49:57.364716+0000","last_peered":"2024-09-18T11:49:57.364716+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:49:57.364716+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7841,"log_dups_size":0,"ondisk_log_size":7841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39859446,"num_objects":4866,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7839,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":642090380,"num_objects":78386,"num_object_clones":0,"num_object_copies":313544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128701,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4673,"num_bytes_recovered":36282368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1257172992,"data_stored":1257172992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128728,"ondisk_log_size":128728,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618592,"kb_used_data":312408,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93753248,"statfs":{"total":96636764160,"available":96003325952,"internally_reserved":0,"allocated":319905792,"data_stored":319730326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,10,3,2,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.923},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.6150000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":7.6950000000000003},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":5.2619999999999996},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.30199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626240,"kb_used_data":316392,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93745600,"statfs":{"total":96636764160,"available":95995494400,"internally_reserved":0,"allocated":323985408,"data_stored":323809942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.498}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":3.944},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.5680000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.52900000000000003},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.55500000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582716,"kb_used_data":284908,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93789124,"statfs":{"total":96636764160,"available":96040062976,"internally_reserved":0,"allocated":291745792,"data_stored":291570326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":6.7560000000000002},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.1210000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":15.701000000000001},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":8.3109999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":16.193000000000001},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":15.782999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622112,"kb_used_data":315988,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93749728,"statfs":{"total":96636764160,"available":95999721472,"internally_reserved":0,"allocated":323571712,"data_stored":323396246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":3.0630000000000002},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.6219999999999999},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.3479999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.1480000000000001},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323166208,"data_stored":323166208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291364864,"data_stored":291364864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323424256,"data_stored":323424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319217664,"data_stored":319217664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:06.857 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:07.426 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:07.427 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:07.445 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-18T11:50:05.964859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637797772,"num_objects":77862,"num_object_clones":0,"num_object_copies":311448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77862,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129225,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":36282368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129252,"ondisk_log_size":129252,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2431956,"kb_used_data":1220328,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375055404,"statfs":{"total":386547056640,"available":384056733696,"internally_reserved":0,"allocated":1249615872,"data_stored":1248914008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,8,14,15,6,2,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19259392,"num_objects":-2351,"num_object_clones":0,"num_object_copies":-9404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004134"},"pg_stats":[{"pgid":"1.f","version":"35'8244","reported_seq":8327,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.141324+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:01.141324+0000","last_peered":"2024-09-18T11:50:01.141324+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:01.141324+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41009152,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2895}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8160","reported_seq":12450,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:01.328708+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:01.328708+0000","last_peered":"2024-09-18T11:50:01.328708+0000","last_clean":"2024-09-18T11:50:01.328708+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:01.328708+0000","last_undegraded":"2024-09-18T11:50:01.328708+0000","last_fullsized":"2024-09-18T11:50:01.328708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40105206,"num_objects":4896,"num_object_clones":0,"num_object_copies":19584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'7997","reported_seq":8112,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:02.033504+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:02.033504+0000","last_peered":"2024-09-18T11:50:02.033504+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:02.033504+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7997,"log_dups_size":0,"ondisk_log_size":7997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39410323,"num_objects":4812,"num_object_clones":0,"num_object_copies":19248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4812,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7996,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2772}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8075","reported_seq":8157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:02.311444+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:02.311444+0000","last_peered":"2024-09-18T11:50:02.311444+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:02.311444+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8075,"log_dups_size":0,"ondisk_log_size":8075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39736566,"num_objects":4851,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8075","reported_seq":8174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.085230+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:01.085230+0000","last_peered":"2024-09-18T11:50:01.085230+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:01.085230+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8075,"log_dups_size":0,"ondisk_log_size":8075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39482614,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8056","reported_seq":8147,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.219088+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:01.219088+0000","last_peered":"2024-09-18T11:50:01.219088+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:01.219088+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8056,"log_dups_size":0,"ondisk_log_size":8056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38748160,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8153","reported_seq":8252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:02.375294+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:02.375294+0000","last_peered":"2024-09-18T11:50:02.375294+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:02.375294+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39834870,"num_objects":4863,"num_object_clones":0,"num_object_copies":19452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2744}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8013","reported_seq":8114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:02.307797+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:02.307797+0000","last_peered":"2024-09-18T11:50:02.307797+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:02.307797+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39826678,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2845}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8265","reported_seq":8842,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.357757+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:01.357757+0000","last_peered":"2024-09-18T11:50:01.357757+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:01.357757+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8265,"log_dups_size":0,"ondisk_log_size":8265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40225260,"num_objects":4911,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":1933312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7836","reported_seq":7948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:02.020874+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:02.020874+0000","last_peered":"2024-09-18T11:50:02.020874+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:02.020874+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7836,"log_dups_size":0,"ondisk_log_size":7836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38974710,"num_objects":4758,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7834,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2795}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8096","reported_seq":8195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.236623+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:01.236623+0000","last_peered":"2024-09-18T11:50:01.236623+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:01.236623+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40439665,"num_objects":4937,"num_object_clones":0,"num_object_copies":19748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8093,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8130","reported_seq":8201,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.350039+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:01.350039+0000","last_peered":"2024-09-18T11:50:01.350039+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:01.350039+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8130,"log_dups_size":0,"ondisk_log_size":8130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40147579,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8016","reported_seq":8106,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.817623+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:01.817623+0000","last_peered":"2024-09-18T11:50:01.817623+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:01.817623+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40557179,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8174","reported_seq":8277,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:01.289932+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:01.289932+0000","last_peered":"2024-09-18T11:50:01.289932+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:01.289932+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40269046,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8172,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2783}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8053","reported_seq":12340,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:01.373233+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:01.373233+0000","last_peered":"2024-09-18T11:50:01.373233+0000","last_clean":"2024-09-18T11:50:01.373233+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:01.373233+0000","last_undegraded":"2024-09-18T11:50:01.373233+0000","last_fullsized":"2024-09-18T11:50:01.373233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8053,"log_dups_size":0,"ondisk_log_size":8053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728374,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8051,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7909","reported_seq":7989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:02.295186+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:02.295186+0000","last_peered":"2024-09-18T11:50:02.295186+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:02.295186+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7909,"log_dups_size":0,"ondisk_log_size":7909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39302390,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":637797772,"num_objects":77862,"num_object_clones":0,"num_object_copies":311448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77862,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129225,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":36282368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1247449088,"data_stored":1247449088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129252,"ondisk_log_size":129252,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618592,"kb_used_data":312408,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93753248,"statfs":{"total":96636764160,"available":96003325952,"internally_reserved":0,"allocated":319905792,"data_stored":319730326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,10,3,2,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.923},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.6150000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":7.6950000000000003},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":5.2619999999999996},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.30199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609084,"kb_used_data":311728,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93762756,"statfs":{"total":96636764160,"available":96013062144,"internally_reserved":0,"allocated":319209472,"data_stored":319034006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:03 2024","interfaces":[{"interface":"back","average":{"1min":5.399,"5min":4.4420000000000002,"15min":2.4980000000000002},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":23.731999999999999,"5min":23.731999999999999,"15min":23.731999999999999},"last":10.148999999999999},{"interface":"front","average":{"1min":5.2119999999999997,"5min":3.319,"15min":2.048},"min":{"1min":0.51600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":17.285,"5min":18.047000000000001,"15min":18.047000000000001},"last":10.182}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":13.500999999999999},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":10.212}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":2.0939999999999999},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.0380000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582716,"kb_used_data":284908,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93789124,"statfs":{"total":96636764160,"available":96040062976,"internally_reserved":0,"allocated":291745792,"data_stored":291570326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":6.7560000000000002},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.1210000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":15.701000000000001},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":8.3109999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":16.193000000000001},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":15.782999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621564,"kb_used_data":311284,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93750276,"statfs":{"total":96636764160,"available":96000282624,"internally_reserved":0,"allocated":318754816,"data_stored":318579350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":6.1130000000000004},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":9.5939999999999994}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":11.666},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":9.5630000000000006}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":0.307},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":19.129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":318296064,"data_stored":318296064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291364864,"data_stored":291364864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318570496,"data_stored":318570496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319217664,"data_stored":319217664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:07.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:07.977 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:07.978 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:08.009 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-18T11:50:07.965827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":628426124,"num_objects":76718,"num_object_clones":0,"num_object_copies":306872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76718,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130369,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4754,"num_bytes_recovered":36741120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130396,"ondisk_log_size":130396,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2417520,"kb_used_data":1205904,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375069840,"statfs":{"total":386547056640,"available":384071516160,"internally_reserved":0,"allocated":1234845696,"data_stored":1234143832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,1,11,11,13,4,1,0,0,2,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28631040,"num_objects":-3495,"num_object_clones":0,"num_object_copies":-13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004585"},"pg_stats":[{"pgid":"1.f","version":"35'8308","reported_seq":8395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.309959+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:06.309959+0000","last_peered":"2024-09-18T11:50:06.309959+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:06.309959+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40484864,"num_objects":4942,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2833}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8238","reported_seq":12528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:06.207300+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:06.207300+0000","last_peered":"2024-09-18T11:50:06.207300+0000","last_clean":"2024-09-18T11:50:06.207300+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:06.207300+0000","last_undegraded":"2024-09-18T11:50:06.207300+0000","last_fullsized":"2024-09-18T11:50:06.207300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8238,"log_dups_size":0,"ondisk_log_size":8238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39466230,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8236,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8059","reported_seq":8173,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.998758+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:06.998758+0000","last_peered":"2024-09-18T11:50:06.998758+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:06.998758+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38902419,"num_objects":4750,"num_object_clones":0,"num_object_copies":19000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8058,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8156","reported_seq":8238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.231405+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:07.231405+0000","last_peered":"2024-09-18T11:50:07.231405+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:07.231405+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39073014,"num_objects":4770,"num_object_clones":0,"num_object_copies":19080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8136","reported_seq":8237,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.230078+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:06.230078+0000","last_peered":"2024-09-18T11:50:06.230078+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:06.230078+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38982902,"num_objects":4759,"num_object_clones":0,"num_object_copies":19036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2063,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8134,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8126","reported_seq":8227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.191492+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:06.191492+0000","last_peered":"2024-09-18T11:50:06.191492+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:06.191492+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8126,"log_dups_size":0,"ondisk_log_size":8126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38174720,"num_objects":4660,"num_object_clones":0,"num_object_copies":18640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8229","reported_seq":8330,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.312335+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:07.312335+0000","last_peered":"2024-09-18T11:50:07.312335+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:07.312335+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8229,"log_dups_size":0,"ondisk_log_size":8229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39212278,"num_objects":4787,"num_object_clones":0,"num_object_copies":19148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8081","reported_seq":8182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.286494+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:07.286494+0000","last_peered":"2024-09-18T11:50:07.286494+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:07.286494+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8081,"log_dups_size":0,"ondisk_log_size":8081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39269622,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2777}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8325","reported_seq":9014,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.336430+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:06.336430+0000","last_peered":"2024-09-18T11:50:06.336430+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:06.336430+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39733740,"num_objects":4851,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8322,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":2392064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7910","reported_seq":8020,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.124009+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:07.124009+0000","last_peered":"2024-09-18T11:50:07.124009+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:07.124009+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38368502,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2721}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8163","reported_seq":8264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.355243+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:06.355243+0000","last_peered":"2024-09-18T11:50:06.355243+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:06.355243+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8163,"log_dups_size":0,"ondisk_log_size":8163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39890801,"num_objects":4870,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2827}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8194","reported_seq":8275,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.301912+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:06.301912+0000","last_peered":"2024-09-18T11:50:06.301912+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:06.301912+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8194,"log_dups_size":0,"ondisk_log_size":8194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39623291,"num_objects":4837,"num_object_clones":0,"num_object_copies":19348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8107","reported_seq":8197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.053390+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:07.053390+0000","last_peered":"2024-09-18T11:50:07.053390+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:07.053390+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8107,"log_dups_size":0,"ondisk_log_size":8107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39811707,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8106,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8241","reported_seq":8348,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.274217+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:06.274217+0000","last_peered":"2024-09-18T11:50:06.274217+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:06.274217+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39720182,"num_objects":4849,"num_object_clones":0,"num_object_copies":19396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8239,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2718}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8135","reported_seq":12422,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:06.288586+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:06.288586+0000","last_peered":"2024-09-18T11:50:06.288586+0000","last_clean":"2024-09-18T11:50:06.288586+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:06.288586+0000","last_undegraded":"2024-09-18T11:50:06.288586+0000","last_fullsized":"2024-09-18T11:50:06.288586+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8135,"log_dups_size":0,"ondisk_log_size":8135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39056630,"num_objects":4768,"num_object_clones":0,"num_object_copies":19072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7988","reported_seq":8072,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.280354+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:07.280354+0000","last_peered":"2024-09-18T11:50:07.280354+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:07.280354+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38655222,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":628426124,"num_objects":76718,"num_object_clones":0,"num_object_copies":306872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76718,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130369,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4754,"num_bytes_recovered":36741120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1232629760,"data_stored":1232629760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130396,"ondisk_log_size":130396,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618184,"kb_used_data":307840,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93753656,"statfs":{"total":96636764160,"available":96003743744,"internally_reserved":0,"allocated":315228160,"data_stored":315052694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,1,1,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.919},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":10.888}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":7.4089999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":19.866},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.60499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608672,"kb_used_data":307160,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93763168,"statfs":{"total":96636764160,"available":96013484032,"internally_reserved":0,"allocated":314531840,"data_stored":314356374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.5419999999999998},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.5709999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.0530000000000008},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.23899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.30199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586180,"kb_used_data":284220,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93785660,"statfs":{"total":96636764160,"available":96036515840,"internally_reserved":0,"allocated":291041280,"data_stored":290865814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.9770000000000003},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.042},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":6.0030000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":5.9160000000000004},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.3940000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604484,"kb_used_data":306684,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93767356,"statfs":{"total":96636764160,"available":96017772544,"internally_reserved":0,"allocated":314044416,"data_stored":313868950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,4,5,4,1,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.6150000000000002},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":10.754}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.593},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":11.007}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.2909999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":10.792999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313544704,"data_stored":313544704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":290742272,"data_stored":290742272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313827328,"data_stored":313827328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":314515456,"data_stored":314515456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:08.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:08.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8308", 'reported_seq': 8395, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:06.309959+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:06.309959+0000', 'last_peered': '2024-09-18T11:50:06.309959+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:06.309959+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8308, 'log_dups_size': 0, 'ondisk_log_size': 8308, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40484864, 'num_objects': 4942, 'num_object_clones': 0, 'num_object_copies': 19768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8308, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2833}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:08.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:08.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8059", 'reported_seq': 8173, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:06.998758+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:06.998758+0000', 'last_peered': '2024-09-18T11:50:06.998758+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:06.998758+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8059, 'log_dups_size': 0, 'ondisk_log_size': 8059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38902419, 'num_objects': 4750, 'num_object_clones': 0, 'num_object_copies': 19000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4750, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8058, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2710}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:08.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:08.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8156", 'reported_seq': 8238, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.231405+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:07.231405+0000', 'last_peered': '2024-09-18T11:50:07.231405+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:07.231405+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8156, 'log_dups_size': 0, 'ondisk_log_size': 8156, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39073014, 'num_objects': 4770, 'num_object_clones': 0, 'num_object_copies': 19080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2108, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8154, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2664}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:08.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8136", 'reported_seq': 8237, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:06.230078+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:06.230078+0000', 'last_peered': '2024-09-18T11:50:06.230078+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:06.230078+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8136, 'log_dups_size': 0, 'ondisk_log_size': 8136, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38982902, 'num_objects': 4759, 'num_object_clones': 0, 'num_object_copies': 19036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2063, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8134, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2698}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8126", 'reported_seq': 8227, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:06.191492+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:06.191492+0000', 'last_peered': '2024-09-18T11:50:06.191492+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:06.191492+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8126, 'log_dups_size': 0, 'ondisk_log_size': 8126, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38174720, 'num_objects': 4660, 'num_object_clones': 0, 'num_object_copies': 18640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2127, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8126, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2533}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8229", 'reported_seq': 8330, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.312335+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:07.312335+0000', 'last_peered': '2024-09-18T11:50:07.312335+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:07.312335+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8229, 'log_dups_size': 0, 'ondisk_log_size': 8229, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39212278, 'num_objects': 4787, 'num_object_clones': 0, 'num_object_copies': 19148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8227, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2669}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8081", 'reported_seq': 8182, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.286494+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:07.286494+0000', 'last_peered': '2024-09-18T11:50:07.286494+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:07.286494+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8081, 'log_dups_size': 0, 'ondisk_log_size': 8081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39269622, 'num_objects': 4794, 'num_object_clones': 0, 'num_object_copies': 19176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2019, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8079, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2777}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8325", 'reported_seq': 9014, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:06.336430+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:06.336430+0000', 'last_peered': '2024-09-18T11:50:06.336430+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:06.336430+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8325, 'log_dups_size': 0, 'ondisk_log_size': 8325, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39733740, 'num_objects': 4851, 'num_object_clones': 0, 'num_object_copies': 19404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1825, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8322, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 316, 'num_bytes_recovered': 2392064, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3026}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:08.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7910", 'reported_seq': 8020, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.124009+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:07.124009+0000', 'last_peered': '2024-09-18T11:50:07.124009+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:07.124009+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7910, 'log_dups_size': 0, 'ondisk_log_size': 7910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38368502, 'num_objects': 4684, 'num_object_clones': 0, 'num_object_copies': 18736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7908, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2721}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:08.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:08.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8163", 'reported_seq': 8264, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:06.355243+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:06.355243+0000', 'last_peered': '2024-09-18T11:50:06.355243+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:06.355243+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8163, 'log_dups_size': 0, 'ondisk_log_size': 8163, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39890801, 'num_objects': 4870, 'num_object_clones': 0, 'num_object_copies': 19480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8160, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2827}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:08.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:08.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8194", 'reported_seq': 8275, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:06.301912+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:06.301912+0000', 'last_peered': '2024-09-18T11:50:06.301912+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:06.301912+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8194, 'log_dups_size': 0, 'ondisk_log_size': 8194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39623291, 'num_objects': 4837, 'num_object_clones': 0, 'num_object_copies': 19348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2090, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8193, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2748}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:08.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:08.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8107", 'reported_seq': 8197, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.053390+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:07.053390+0000', 'last_peered': '2024-09-18T11:50:07.053390+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:07.053390+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8107, 'log_dups_size': 0, 'ondisk_log_size': 8107, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39811707, 'num_objects': 4860, 'num_object_clones': 0, 'num_object_copies': 19440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8106, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2811}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:08.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:08.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8241", 'reported_seq': 8348, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:06.274217+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:06.274217+0000', 'last_peered': '2024-09-18T11:50:06.274217+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:06.274217+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8241, 'log_dups_size': 0, 'ondisk_log_size': 8241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39720182, 'num_objects': 4849, 'num_object_clones': 0, 'num_object_copies': 19396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8239, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2718}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:08.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:08.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7988", 'reported_seq': 8072, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.280354+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:07.280354+0000', 'last_peered': '2024-09-18T11:50:07.280354+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:07.280354+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7988, 'log_dups_size': 0, 'ondisk_log_size': 7988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38655222, 'num_objects': 4719, 'num_object_clones': 0, 'num_object_copies': 18876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2006, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7986, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2715}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:08.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:08.692 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:08.693 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28702/306872 objects degraded (9.353%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":406,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76718,"data_bytes":628426124,"bytes_used":2475540480,"bytes_avail":384071516160,"bytes_total":386547056640,"degraded_objects":28702,"degraded_total":306872,"degraded_ratio":0.093530853254777224,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":110549,"recovering_keys_per_sec":0,"num_objects_recovered":208,"num_bytes_recovered":1327104,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":291},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:08.713 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:08.714 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:09.231 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:09.231 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:09.251 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-18T11:50:07.965827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":628426124,"num_objects":76718,"num_object_clones":0,"num_object_copies":306872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76718,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130369,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4754,"num_bytes_recovered":36741120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130396,"ondisk_log_size":130396,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2417520,"kb_used_data":1205904,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375069840,"statfs":{"total":386547056640,"available":384071516160,"internally_reserved":0,"allocated":1234845696,"data_stored":1234143832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,1,11,11,13,4,1,0,0,2,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28631040,"num_objects":-3495,"num_object_clones":0,"num_object_copies":-13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004585"},"pg_stats":[{"pgid":"1.f","version":"35'8308","reported_seq":8395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.309959+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:06.309959+0000","last_peered":"2024-09-18T11:50:06.309959+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:06.309959+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40484864,"num_objects":4942,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2833}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8238","reported_seq":12528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:06.207300+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:06.207300+0000","last_peered":"2024-09-18T11:50:06.207300+0000","last_clean":"2024-09-18T11:50:06.207300+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:06.207300+0000","last_undegraded":"2024-09-18T11:50:06.207300+0000","last_fullsized":"2024-09-18T11:50:06.207300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8238,"log_dups_size":0,"ondisk_log_size":8238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39466230,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8236,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8059","reported_seq":8173,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.998758+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:06.998758+0000","last_peered":"2024-09-18T11:50:06.998758+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:06.998758+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38902419,"num_objects":4750,"num_object_clones":0,"num_object_copies":19000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8058,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8156","reported_seq":8238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.231405+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:07.231405+0000","last_peered":"2024-09-18T11:50:07.231405+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:07.231405+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39073014,"num_objects":4770,"num_object_clones":0,"num_object_copies":19080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8136","reported_seq":8237,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.230078+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:06.230078+0000","last_peered":"2024-09-18T11:50:06.230078+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:06.230078+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38982902,"num_objects":4759,"num_object_clones":0,"num_object_copies":19036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2063,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8134,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8126","reported_seq":8227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.191492+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:06.191492+0000","last_peered":"2024-09-18T11:50:06.191492+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:06.191492+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8126,"log_dups_size":0,"ondisk_log_size":8126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38174720,"num_objects":4660,"num_object_clones":0,"num_object_copies":18640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8229","reported_seq":8330,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.312335+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:07.312335+0000","last_peered":"2024-09-18T11:50:07.312335+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:07.312335+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8229,"log_dups_size":0,"ondisk_log_size":8229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39212278,"num_objects":4787,"num_object_clones":0,"num_object_copies":19148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8081","reported_seq":8182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.286494+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:07.286494+0000","last_peered":"2024-09-18T11:50:07.286494+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:07.286494+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8081,"log_dups_size":0,"ondisk_log_size":8081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39269622,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2777}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8325","reported_seq":9014,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.336430+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:06.336430+0000","last_peered":"2024-09-18T11:50:06.336430+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:06.336430+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39733740,"num_objects":4851,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1825,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8322,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":2392064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7910","reported_seq":8020,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.124009+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:07.124009+0000","last_peered":"2024-09-18T11:50:07.124009+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:07.124009+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38368502,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2721}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8163","reported_seq":8264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.355243+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:06.355243+0000","last_peered":"2024-09-18T11:50:06.355243+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:06.355243+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8163,"log_dups_size":0,"ondisk_log_size":8163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39890801,"num_objects":4870,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2827}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8194","reported_seq":8275,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.301912+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:06.301912+0000","last_peered":"2024-09-18T11:50:06.301912+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:06.301912+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8194,"log_dups_size":0,"ondisk_log_size":8194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39623291,"num_objects":4837,"num_object_clones":0,"num_object_copies":19348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8107","reported_seq":8197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.053390+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:07.053390+0000","last_peered":"2024-09-18T11:50:07.053390+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:07.053390+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8107,"log_dups_size":0,"ondisk_log_size":8107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39811707,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8106,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8241","reported_seq":8348,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.274217+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:06.274217+0000","last_peered":"2024-09-18T11:50:06.274217+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:06.274217+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39720182,"num_objects":4849,"num_object_clones":0,"num_object_copies":19396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8239,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2718}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8135","reported_seq":12422,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:06.288586+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:06.288586+0000","last_peered":"2024-09-18T11:50:06.288586+0000","last_clean":"2024-09-18T11:50:06.288586+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:06.288586+0000","last_undegraded":"2024-09-18T11:50:06.288586+0000","last_fullsized":"2024-09-18T11:50:06.288586+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8135,"log_dups_size":0,"ondisk_log_size":8135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39056630,"num_objects":4768,"num_object_clones":0,"num_object_copies":19072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7988","reported_seq":8072,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.280354+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:07.280354+0000","last_peered":"2024-09-18T11:50:07.280354+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:07.280354+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38655222,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":628426124,"num_objects":76718,"num_object_clones":0,"num_object_copies":306872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76718,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130369,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4754,"num_bytes_recovered":36741120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1232629760,"data_stored":1232629760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130396,"ondisk_log_size":130396,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618184,"kb_used_data":307840,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93753656,"statfs":{"total":96636764160,"available":96003743744,"internally_reserved":0,"allocated":315228160,"data_stored":315052694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,1,1,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.919},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":10.888}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":7.4089999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":19.866},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.60499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608672,"kb_used_data":307160,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93763168,"statfs":{"total":96636764160,"available":96013484032,"internally_reserved":0,"allocated":314531840,"data_stored":314356374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.5419999999999998},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.5709999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.0530000000000008},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.23899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.30199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586180,"kb_used_data":284220,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93785660,"statfs":{"total":96636764160,"available":96036515840,"internally_reserved":0,"allocated":291041280,"data_stored":290865814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.9770000000000003},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.042},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":6.0030000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":5.9160000000000004},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.3940000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604484,"kb_used_data":306684,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93767356,"statfs":{"total":96636764160,"available":96017772544,"internally_reserved":0,"allocated":314044416,"data_stored":313868950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,4,5,4,1,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.6150000000000002},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":10.754}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.593},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":11.007}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.2909999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":10.792999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313544704,"data_stored":313544704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":290742272,"data_stored":290742272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313827328,"data_stored":313827328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":314515456,"data_stored":314515456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:12.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:12.803 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:12.804 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:12.829 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-18T11:50:11.971673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622953868,"num_objects":76050,"num_object_clones":0,"num_object_copies":304200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28635,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76050,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131037,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4821,"num_bytes_recovered":37158912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131064,"ondisk_log_size":131064,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2404152,"kb_used_data":1200848,"kb_used_omap":14,"kb_used_meta":1203249,"kb_avail":375083208,"statfs":{"total":386547056640,"available":384085204992,"internally_reserved":0,"allocated":1229668352,"data_stored":1228966488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232127003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,6,4,11,20,6,1,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24731648,"num_objects":-3019,"num_object_clones":0,"num_object_copies":-12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3019,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008765"},"pg_stats":[{"pgid":"1.f","version":"35'8389","reported_seq":8478,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.217327+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:11.217327+0000","last_peered":"2024-09-18T11:50:11.217327+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:11.217327+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39821312,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8300","reported_seq":12590,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:11.431529+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:11.431529+0000","last_peered":"2024-09-18T11:50:11.431529+0000","last_clean":"2024-09-18T11:50:11.431529+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:11.431529+0000","last_undegraded":"2024-09-18T11:50:11.431529+0000","last_fullsized":"2024-09-18T11:50:11.431529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8300,"log_dups_size":0,"ondisk_log_size":8300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958326,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8298,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8059","reported_seq":8173,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.998758+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:06.998758+0000","last_peered":"2024-09-18T11:50:06.998758+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:06.998758+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38902419,"num_objects":4750,"num_object_clones":0,"num_object_copies":19000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8058,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8156","reported_seq":8238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.231405+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:07.231405+0000","last_peered":"2024-09-18T11:50:07.231405+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:07.231405+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39073014,"num_objects":4770,"num_object_clones":0,"num_object_copies":19080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8205","reported_seq":8310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.137286+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:11.137286+0000","last_peered":"2024-09-18T11:50:11.137286+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:11.137286+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8205,"log_dups_size":0,"ondisk_log_size":8205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38417654,"num_objects":4690,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8203,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2631}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8205","reported_seq":8314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.359556+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:11.359556+0000","last_peered":"2024-09-18T11:50:11.359556+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:11.359556+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8205,"log_dups_size":0,"ondisk_log_size":8205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8229","reported_seq":8330,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.312335+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:07.312335+0000","last_peered":"2024-09-18T11:50:07.312335+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:07.312335+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8229,"log_dups_size":0,"ondisk_log_size":8229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39212278,"num_objects":4787,"num_object_clones":0,"num_object_copies":19148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8081","reported_seq":8182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.286494+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:07.286494+0000","last_peered":"2024-09-18T11:50:07.286494+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:07.286494+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8081,"log_dups_size":0,"ondisk_log_size":8081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39269622,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2777}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8403","reported_seq":9196,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.370701+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:11.370701+0000","last_peered":"2024-09-18T11:50:11.370701+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:11.370701+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39094764,"num_objects":4773,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":2809856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7910","reported_seq":8020,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.124009+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:07.124009+0000","last_peered":"2024-09-18T11:50:07.124009+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:07.124009+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38368502,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2721}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8233","reported_seq":8340,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.348840+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:11.348840+0000","last_peered":"2024-09-18T11:50:11.348840+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:11.348840+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8233,"log_dups_size":0,"ondisk_log_size":8233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39317361,"num_objects":4800,"num_object_clones":0,"num_object_copies":19200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8274","reported_seq":8355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.266199+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:11.266199+0000","last_peered":"2024-09-18T11:50:11.266199+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:11.266199+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38967931,"num_objects":4757,"num_object_clones":0,"num_object_copies":19028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8273,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2668}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8107","reported_seq":8197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.053390+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:07.053390+0000","last_peered":"2024-09-18T11:50:07.053390+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:07.053390+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8107,"log_dups_size":0,"ondisk_log_size":8107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39811707,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8106,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8313","reported_seq":8430,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.361751+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:11.361751+0000","last_peered":"2024-09-18T11:50:11.361751+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:11.361751+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8313,"log_dups_size":0,"ondisk_log_size":8313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39130358,"num_objects":4777,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2651}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8212","reported_seq":12499,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:11.396098+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:11.396098+0000","last_peered":"2024-09-18T11:50:11.396098+0000","last_clean":"2024-09-18T11:50:11.396098+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:11.396098+0000","last_undegraded":"2024-09-18T11:50:11.396098+0000","last_fullsized":"2024-09-18T11:50:11.396098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8212,"log_dups_size":0,"ondisk_log_size":8212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38425846,"num_objects":4691,"num_object_clones":0,"num_object_copies":18764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8210,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7988","reported_seq":8072,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.280354+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:07.280354+0000","last_peered":"2024-09-18T11:50:07.280354+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:07.280354+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38655222,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622953868,"num_objects":76050,"num_object_clones":0,"num_object_copies":304200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28635,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76050,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131037,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4821,"num_bytes_recovered":37158912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1227333632,"data_stored":1227333632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131064,"ondisk_log_size":131064,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601160,"kb_used_data":303288,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93770680,"statfs":{"total":96636764160,"available":96021176320,"internally_reserved":0,"allocated":310566912,"data_stored":310391446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,3,10,3,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":11.644},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":11.676}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":26.899999999999999},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":9.0389999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":13.896000000000001},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":9.0069999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608672,"kb_used_data":307160,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93763168,"statfs":{"total":96636764160,"available":96013484032,"internally_reserved":0,"allocated":314531840,"data_stored":314356374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.5419999999999998},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.5709999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.0530000000000008},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.23899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.30199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589836,"kb_used_data":283716,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93782004,"statfs":{"total":96636764160,"available":96032772096,"internally_reserved":0,"allocated":290525184,"data_stored":290349718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.968},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.9930000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":3.8959999999999999},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.9390000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":1.2669999999999999},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.25}]}]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604484,"kb_used_data":306684,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93767356,"statfs":{"total":96636764160,"available":96017772544,"internally_reserved":0,"allocated":314044416,"data_stored":313868950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,4,5,4,1,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.6150000000000002},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":10.754}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.593},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":11.007}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.2909999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":10.792999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313544704,"data_stored":313544704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":290213888,"data_stored":290213888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313827328,"data_stored":313827328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":309747712,"data_stored":309747712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:12.830 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:13.452 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:13.452 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:13.477 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-18T11:50:11.971673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622953868,"num_objects":76050,"num_object_clones":0,"num_object_copies":304200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28635,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76050,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131037,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4821,"num_bytes_recovered":37158912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131064,"ondisk_log_size":131064,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2404152,"kb_used_data":1200848,"kb_used_omap":14,"kb_used_meta":1203249,"kb_avail":375083208,"statfs":{"total":386547056640,"available":384085204992,"internally_reserved":0,"allocated":1229668352,"data_stored":1228966488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232127003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,6,4,11,20,6,1,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24731648,"num_objects":-3019,"num_object_clones":0,"num_object_copies":-12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3019,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008765"},"pg_stats":[{"pgid":"1.f","version":"35'8389","reported_seq":8478,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.217327+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:11.217327+0000","last_peered":"2024-09-18T11:50:11.217327+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:11.217327+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39821312,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8300","reported_seq":12590,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:11.431529+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:11.431529+0000","last_peered":"2024-09-18T11:50:11.431529+0000","last_clean":"2024-09-18T11:50:11.431529+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:11.431529+0000","last_undegraded":"2024-09-18T11:50:11.431529+0000","last_fullsized":"2024-09-18T11:50:11.431529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8300,"log_dups_size":0,"ondisk_log_size":8300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958326,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8298,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8059","reported_seq":8173,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:06.998758+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:06.998758+0000","last_peered":"2024-09-18T11:50:06.998758+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:06.998758+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38902419,"num_objects":4750,"num_object_clones":0,"num_object_copies":19000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8058,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8156","reported_seq":8238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.231405+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:07.231405+0000","last_peered":"2024-09-18T11:50:07.231405+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:07.231405+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39073014,"num_objects":4770,"num_object_clones":0,"num_object_copies":19080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8205","reported_seq":8310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.137286+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:11.137286+0000","last_peered":"2024-09-18T11:50:11.137286+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:11.137286+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8205,"log_dups_size":0,"ondisk_log_size":8205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38417654,"num_objects":4690,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8203,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2631}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8205","reported_seq":8314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.359556+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:11.359556+0000","last_peered":"2024-09-18T11:50:11.359556+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:11.359556+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8205,"log_dups_size":0,"ondisk_log_size":8205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8229","reported_seq":8330,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.312335+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:07.312335+0000","last_peered":"2024-09-18T11:50:07.312335+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:07.312335+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8229,"log_dups_size":0,"ondisk_log_size":8229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39212278,"num_objects":4787,"num_object_clones":0,"num_object_copies":19148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8081","reported_seq":8182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.286494+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:07.286494+0000","last_peered":"2024-09-18T11:50:07.286494+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:07.286494+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8081,"log_dups_size":0,"ondisk_log_size":8081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39269622,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2777}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8403","reported_seq":9196,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.370701+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:11.370701+0000","last_peered":"2024-09-18T11:50:11.370701+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:11.370701+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39094764,"num_objects":4773,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":2809856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7910","reported_seq":8020,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.124009+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:07.124009+0000","last_peered":"2024-09-18T11:50:07.124009+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:07.124009+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38368502,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2721}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8233","reported_seq":8340,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.348840+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:11.348840+0000","last_peered":"2024-09-18T11:50:11.348840+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:11.348840+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8233,"log_dups_size":0,"ondisk_log_size":8233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39317361,"num_objects":4800,"num_object_clones":0,"num_object_copies":19200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8274","reported_seq":8355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.266199+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:11.266199+0000","last_peered":"2024-09-18T11:50:11.266199+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:11.266199+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38967931,"num_objects":4757,"num_object_clones":0,"num_object_copies":19028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8273,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2668}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8107","reported_seq":8197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.053390+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:07.053390+0000","last_peered":"2024-09-18T11:50:07.053390+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:07.053390+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8107,"log_dups_size":0,"ondisk_log_size":8107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39811707,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8106,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8313","reported_seq":8430,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.361751+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:11.361751+0000","last_peered":"2024-09-18T11:50:11.361751+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:11.361751+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8313,"log_dups_size":0,"ondisk_log_size":8313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39130358,"num_objects":4777,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2651}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8212","reported_seq":12499,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:11.396098+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:11.396098+0000","last_peered":"2024-09-18T11:50:11.396098+0000","last_clean":"2024-09-18T11:50:11.396098+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:11.396098+0000","last_undegraded":"2024-09-18T11:50:11.396098+0000","last_fullsized":"2024-09-18T11:50:11.396098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8212,"log_dups_size":0,"ondisk_log_size":8212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38425846,"num_objects":4691,"num_object_clones":0,"num_object_copies":18764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8210,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'7988","reported_seq":8072,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:07.280354+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:07.280354+0000","last_peered":"2024-09-18T11:50:07.280354+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:07.280354+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38655222,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622953868,"num_objects":76050,"num_object_clones":0,"num_object_copies":304200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28635,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76050,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131037,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4821,"num_bytes_recovered":37158912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1227333632,"data_stored":1227333632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131064,"ondisk_log_size":131064,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601160,"kb_used_data":303288,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93770680,"statfs":{"total":96636764160,"available":96021176320,"internally_reserved":0,"allocated":310566912,"data_stored":310391446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,3,10,3,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":11.644},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":11.676}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":26.899999999999999},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":9.0389999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":13.896000000000001},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":9.0069999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608672,"kb_used_data":307160,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93763168,"statfs":{"total":96636764160,"available":96013484032,"internally_reserved":0,"allocated":314531840,"data_stored":314356374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.5419999999999998},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":6.5709999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.0530000000000008},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.23899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.30199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589836,"kb_used_data":283716,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93782004,"statfs":{"total":96636764160,"available":96032772096,"internally_reserved":0,"allocated":290525184,"data_stored":290349718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.968},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.9930000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":3.8959999999999999},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.9390000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":1.2669999999999999},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.25}]}]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604484,"kb_used_data":306684,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93767356,"statfs":{"total":96636764160,"available":96017772544,"internally_reserved":0,"allocated":314044416,"data_stored":313868950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,4,5,4,1,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.6150000000000002},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":10.754}]},{"osd":2,"last update":"Wed Sep 18 11:49:06 2024","interfaces":[{"interface":"back","average":{"1min":5.6139999999999999,"5min":4.6260000000000003,"15min":2.1030000000000002},"min":{"1min":0.28499999999999998,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":29.437000000000001,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.593},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.6739999999999999,"15min":2.016},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":14.132999999999999,"5min":22.739999999999998,"15min":22.739999999999998},"last":11.007}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.2909999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":10.792999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313544704,"data_stored":313544704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":290213888,"data_stored":290213888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313827328,"data_stored":313827328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":309747712,"data_stored":309747712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:13.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:13.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8389", 'reported_seq': 8478, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:11.217327+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:11.217327+0000', 'last_peered': '2024-09-18T11:50:11.217327+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:11.217327+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8389, 'log_dups_size': 0, 'ondisk_log_size': 8389, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39821312, 'num_objects': 4861, 'num_object_clones': 0, 'num_object_copies': 19444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2108, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8389, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2753}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:13.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:13.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8059", 'reported_seq': 8173, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:06.998758+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:06.998758+0000', 'last_peered': '2024-09-18T11:50:06.998758+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:06.998758+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8059, 'log_dups_size': 0, 'ondisk_log_size': 8059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38902419, 'num_objects': 4750, 'num_object_clones': 0, 'num_object_copies': 19000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4750, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8058, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2710}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:13.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:13.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8156", 'reported_seq': 8238, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.231405+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:07.231405+0000', 'last_peered': '2024-09-18T11:50:07.231405+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:07.231405+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8156, 'log_dups_size': 0, 'ondisk_log_size': 8156, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39073014, 'num_objects': 4770, 'num_object_clones': 0, 'num_object_copies': 19080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2108, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8154, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2664}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:13.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:13.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8205", 'reported_seq': 8310, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:11.137286+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:11.137286+0000', 'last_peered': '2024-09-18T11:50:11.137286+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:11.137286+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8205, 'log_dups_size': 0, 'ondisk_log_size': 8205, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38417654, 'num_objects': 4690, 'num_object_clones': 0, 'num_object_copies': 18760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8203, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2631}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:13.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:13.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8205", 'reported_seq': 8314, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:11.359556+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:11.359556+0000', 'last_peered': '2024-09-18T11:50:11.359556+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:11.359556+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8205, 'log_dups_size': 0, 'ondisk_log_size': 8205, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37527552, 'num_objects': 4581, 'num_object_clones': 0, 'num_object_copies': 18324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8205, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2458}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:13.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:13.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8229", 'reported_seq': 8330, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.312335+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:07.312335+0000', 'last_peered': '2024-09-18T11:50:07.312335+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:07.312335+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8229, 'log_dups_size': 0, 'ondisk_log_size': 8229, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39212278, 'num_objects': 4787, 'num_object_clones': 0, 'num_object_copies': 19148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8227, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2669}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:13.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:13.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8081", 'reported_seq': 8182, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.286494+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:07.286494+0000', 'last_peered': '2024-09-18T11:50:07.286494+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:07.286494+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8081, 'log_dups_size': 0, 'ondisk_log_size': 8081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39269622, 'num_objects': 4794, 'num_object_clones': 0, 'num_object_copies': 19176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2019, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8079, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2777}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:13.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:13.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8403", 'reported_seq': 9196, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:11.370701+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:11.370701+0000', 'last_peered': '2024-09-18T11:50:11.370701+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:11.370701+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8403, 'log_dups_size': 0, 'ondisk_log_size': 8403, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39094764, 'num_objects': 4773, 'num_object_clones': 0, 'num_object_copies': 19092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8400, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 368, 'num_bytes_recovered': 2809856, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3000}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:13.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:13.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7910", 'reported_seq': 8020, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.124009+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:07.124009+0000', 'last_peered': '2024-09-18T11:50:07.124009+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:07.124009+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7910, 'log_dups_size': 0, 'ondisk_log_size': 7910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38368502, 'num_objects': 4684, 'num_object_clones': 0, 'num_object_copies': 18736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7908, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2721}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:13.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:13.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8233", 'reported_seq': 8340, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:11.348840+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:11.348840+0000', 'last_peered': '2024-09-18T11:50:11.348840+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:11.348840+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8233, 'log_dups_size': 0, 'ondisk_log_size': 8233, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39317361, 'num_objects': 4800, 'num_object_clones': 0, 'num_object_copies': 19200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8230, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2760}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:13.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:13.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8274", 'reported_seq': 8355, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:11.266199+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:11.266199+0000', 'last_peered': '2024-09-18T11:50:11.266199+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:11.266199+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8274, 'log_dups_size': 0, 'ondisk_log_size': 8274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38967931, 'num_objects': 4757, 'num_object_clones': 0, 'num_object_copies': 19028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2090, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8273, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2668}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:13.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:13.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8107", 'reported_seq': 8197, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.053390+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:07.053390+0000', 'last_peered': '2024-09-18T11:50:07.053390+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:07.053390+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8107, 'log_dups_size': 0, 'ondisk_log_size': 8107, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39811707, 'num_objects': 4860, 'num_object_clones': 0, 'num_object_copies': 19440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8106, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2811}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:13.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:13.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8313", 'reported_seq': 8430, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:11.361751+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:11.361751+0000', 'last_peered': '2024-09-18T11:50:11.361751+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:11.361751+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8313, 'log_dups_size': 0, 'ondisk_log_size': 8313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39130358, 'num_objects': 4777, 'num_object_clones': 0, 'num_object_copies': 19108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2128, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8311, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:13.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:13.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7988", 'reported_seq': 8072, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:07.280354+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:07.280354+0000', 'last_peered': '2024-09-18T11:50:07.280354+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:07.280354+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 7988, 'log_dups_size': 0, 'ondisk_log_size': 7988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38655222, 'num_objects': 4719, 'num_object_clones': 0, 'num_object_copies': 18876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2006, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7986, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2715}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:13.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:14.169 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:14.169 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28632/302180 objects degraded (9.475%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":412,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":75545,"data_bytes":618816908,"bytes_used":2465193984,"bytes_avail":384081862656,"bytes_total":386547056640,"degraded_objects":28632,"degraded_total":302180,"degraded_ratio":0.094751472632205971,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":72990,"recovering_keys_per_sec":0,"num_objects_recovered":151,"num_bytes_recovered":876544,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":236},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:14.199 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:14.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:14.768 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:14.768 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:14.791 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-18T11:50:13.972421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":618816908,"num_objects":75545,"num_object_clones":0,"num_object_copies":302180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131542,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4824,"num_bytes_recovered":37158912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131569,"ondisk_log_size":131569,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2407416,"kb_used_data":1191584,"kb_used_omap":14,"kb_used_meta":1215729,"kb_avail":375079944,"statfs":{"total":386547056640,"available":384081862656,"internally_reserved":0,"allocated":1220182016,"data_stored":1219480152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244906523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8,2,10,15,5,4,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23273472,"num_objects":-2841,"num_object_clones":0,"num_object_copies":-11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2841,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008950"},"pg_stats":[{"pgid":"1.f","version":"35'8389","reported_seq":8478,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.217327+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:11.217327+0000","last_peered":"2024-09-18T11:50:11.217327+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:11.217327+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39821312,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8300","reported_seq":12590,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:11.431529+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:11.431529+0000","last_peered":"2024-09-18T11:50:11.431529+0000","last_clean":"2024-09-18T11:50:11.431529+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:11.431529+0000","last_undegraded":"2024-09-18T11:50:11.431529+0000","last_fullsized":"2024-09-18T11:50:11.431529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8300,"log_dups_size":0,"ondisk_log_size":8300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958326,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8298,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8129","reported_seq":8244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:12.134215+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:12.134215+0000","last_peered":"2024-09-18T11:50:12.134215+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:12.134215+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8129,"log_dups_size":0,"ondisk_log_size":8129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38328979,"num_objects":4680,"num_object_clones":0,"num_object_copies":18720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4680,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8128,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8239","reported_seq":8325,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:12.413955+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:12.413955+0000","last_peered":"2024-09-18T11:50:12.413955+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:12.413955+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8239,"log_dups_size":0,"ondisk_log_size":8239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38393078,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8237,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2583}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8205","reported_seq":8310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.137286+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:11.137286+0000","last_peered":"2024-09-18T11:50:11.137286+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:11.137286+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8205,"log_dups_size":0,"ondisk_log_size":8205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38417654,"num_objects":4690,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8203,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2631}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8205","reported_seq":8314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.359556+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:11.359556+0000","last_peered":"2024-09-18T11:50:11.359556+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:11.359556+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8205,"log_dups_size":0,"ondisk_log_size":8205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8300","reported_seq":8400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:12.416805+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:12.416805+0000","last_peered":"2024-09-18T11:50:12.416805+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:12.416805+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8300,"log_dups_size":0,"ondisk_log_size":8300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38630646,"num_objects":4716,"num_object_clones":0,"num_object_copies":18864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8298,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2598}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8165","reported_seq":8266,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:12.307970+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:12.307970+0000","last_peered":"2024-09-18T11:50:12.307970+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:12.307970+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8165,"log_dups_size":0,"ondisk_log_size":8165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38581494,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8163,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8403","reported_seq":9196,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.370701+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:11.370701+0000","last_peered":"2024-09-18T11:50:11.370701+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:11.370701+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39094764,"num_objects":4773,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":2809856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7961","reported_seq":8073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:12.039559+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:12.039559+0000","last_peered":"2024-09-18T11:50:12.039559+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:12.039559+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37950710,"num_objects":4633,"num_object_clones":0,"num_object_copies":18532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7959,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8233","reported_seq":8340,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.348840+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:11.348840+0000","last_peered":"2024-09-18T11:50:11.348840+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:11.348840+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8233,"log_dups_size":0,"ondisk_log_size":8233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39317361,"num_objects":4800,"num_object_clones":0,"num_object_copies":19200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8274","reported_seq":8355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.266199+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:11.266199+0000","last_peered":"2024-09-18T11:50:11.266199+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:11.266199+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38967931,"num_objects":4757,"num_object_clones":0,"num_object_copies":19028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8273,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2668}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8184","reported_seq":8274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:12.119012+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:12.119012+0000","last_peered":"2024-09-18T11:50:12.119012+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:12.119012+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8184,"log_dups_size":0,"ondisk_log_size":8184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39180923,"num_objects":4783,"num_object_clones":0,"num_object_copies":19132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8183,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2734}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8313","reported_seq":8430,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:11.361751+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:11.361751+0000","last_peered":"2024-09-18T11:50:11.361751+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:11.361751+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8313,"log_dups_size":0,"ondisk_log_size":8313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39130358,"num_objects":4777,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2651}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8212","reported_seq":12499,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:11.396098+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:11.396098+0000","last_peered":"2024-09-18T11:50:11.396098+0000","last_clean":"2024-09-18T11:50:11.396098+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:11.396098+0000","last_undegraded":"2024-09-18T11:50:11.396098+0000","last_fullsized":"2024-09-18T11:50:11.396098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8212,"log_dups_size":0,"ondisk_log_size":8212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38425846,"num_objects":4691,"num_object_clones":0,"num_object_copies":18764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8210,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8057","reported_seq":8143,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:12.309290+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:12.309290+0000","last_peered":"2024-09-18T11:50:12.309290+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:12.309290+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8057,"log_dups_size":0,"ondisk_log_size":8057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38089974,"num_objects":4650,"num_object_clones":0,"num_object_copies":18600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8055,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2647}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":618816908,"num_objects":75545,"num_object_clones":0,"num_object_copies":302180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131542,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4824,"num_bytes_recovered":37158912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1217658880,"data_stored":1217658880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131569,"ondisk_log_size":131569,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601160,"kb_used_data":303288,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93770680,"statfs":{"total":96636764160,"available":96021176320,"internally_reserved":0,"allocated":310566912,"data_stored":310391446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,3,10,3,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":11.644},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":11.676}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":26.899999999999999},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":9.0389999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":13.896000000000001},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":9.0069999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608300,"kb_used_data":302604,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93763540,"statfs":{"total":96636764160,"available":96013864960,"internally_reserved":0,"allocated":309866496,"data_stored":309691030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.8689999999999998},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.28999999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":4.7779999999999996},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.9470000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589836,"kb_used_data":283716,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93782004,"statfs":{"total":96636764160,"available":96032772096,"internally_reserved":0,"allocated":290525184,"data_stored":290349718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.968},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.9930000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":3.8959999999999999},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.9390000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":1.2669999999999999},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.25}]}]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608120,"kb_used_data":301976,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93763720,"statfs":{"total":96636764160,"available":96014049280,"internally_reserved":0,"allocated":309223424,"data_stored":309047958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":5.0919999999999996},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":8.8710000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":8.9149999999999991}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":3.4889999999999999},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":14.023999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":308719616,"data_stored":308719616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":290213888,"data_stored":290213888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308977664,"data_stored":308977664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":309747712,"data_stored":309747712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:17.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:18.384 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:18.386 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:18.407 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-18T11:50:17.974089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":609273228,"num_objects":74380,"num_object_clones":0,"num_object_copies":297520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74380,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132707,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4888,"num_bytes_recovered":37593088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132734,"ondisk_log_size":132734,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2409832,"kb_used_data":1177388,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":375077528,"statfs":{"total":386547056640,"available":384079388672,"internally_reserved":0,"allocated":1205645312,"data_stored":1204943448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,6,7,2,12,9,3,2,0,0,0,4,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28524544,"num_objects":-3482,"num_object_clones":0,"num_object_copies":-13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009229"},"pg_stats":[{"pgid":"1.f","version":"35'8460","reported_seq":8553,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.372072+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:16.372072+0000","last_peered":"2024-09-18T11:50:16.372072+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:16.372072+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8460,"log_dups_size":0,"ondisk_log_size":8460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39239680,"num_objects":4790,"num_object_clones":0,"num_object_copies":19160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8460,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2684}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8369","reported_seq":12659,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:16.463923+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:16.463923+0000","last_peered":"2024-09-18T11:50:16.463923+0000","last_clean":"2024-09-18T11:50:16.463923+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:16.463923+0000","last_undegraded":"2024-09-18T11:50:16.463923+0000","last_fullsized":"2024-09-18T11:50:16.463923+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8369,"log_dups_size":0,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38393078,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8197","reported_seq":8312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.064648+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:17.064648+0000","last_peered":"2024-09-18T11:50:17.064648+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:17.064648+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8197,"log_dups_size":0,"ondisk_log_size":8197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37771923,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8196,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2572}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8325","reported_seq":8411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.348813+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:17.348813+0000","last_peered":"2024-09-18T11:50:17.348813+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:17.348813+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37688566,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8271","reported_seq":8378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.344377+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:16.344377+0000","last_peered":"2024-09-18T11:50:16.344377+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:16.344377+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8271,"log_dups_size":0,"ondisk_log_size":8271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876982,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8269,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8273","reported_seq":8388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.247515+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:16.247515+0000","last_peered":"2024-09-18T11:50:16.247515+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:16.247515+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36970496,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8273,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8381","reported_seq":8481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.386834+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:17.386834+0000","last_peered":"2024-09-18T11:50:17.386834+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:17.386834+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37967094,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2517}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8245","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.483717+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:17.483717+0000","last_peered":"2024-09-18T11:50:17.483717+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:17.483717+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926134,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8243,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2613}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8471","reported_seq":9370,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.400465+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:16.400465+0000","last_peered":"2024-09-18T11:50:16.400465+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:16.400465+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38537708,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8468,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":3244032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8029","reported_seq":8141,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.161697+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:17.161697+0000","last_peered":"2024-09-18T11:50:17.161697+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:17.161697+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8029,"log_dups_size":0,"ondisk_log_size":8029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393654,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2602}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8306","reported_seq":8419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.272977+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:16.272977+0000","last_peered":"2024-09-18T11:50:16.272977+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:16.272977+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38719345,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8345","reported_seq":8428,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.326007+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:16.326007+0000","last_peered":"2024-09-18T11:50:16.326007+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:16.326007+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8345,"log_dups_size":0,"ondisk_log_size":8345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38386299,"num_objects":4686,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2598}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8257","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.122070+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:17.122070+0000","last_peered":"2024-09-18T11:50:17.122070+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:17.122070+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8257,"log_dups_size":0,"ondisk_log_size":8257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38582907,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2661}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8406","reported_seq":8525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.392915+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:16.392915+0000","last_peered":"2024-09-18T11:50:16.392915+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:16.392915+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38368502,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8280","reported_seq":12567,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:16.342801+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:16.342801+0000","last_peered":"2024-09-18T11:50:16.342801+0000","last_clean":"2024-09-18T11:50:16.342801+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:16.342801+0000","last_undegraded":"2024-09-18T11:50:16.342801+0000","last_fullsized":"2024-09-18T11:50:16.342801+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37868790,"num_objects":4623,"num_object_clones":0,"num_object_copies":18492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8278,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8119","reported_seq":8205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.039632+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:17.039632+0000","last_peered":"2024-09-18T11:50:17.039632+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:17.039632+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8119,"log_dups_size":0,"ondisk_log_size":8119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37582070,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":609273228,"num_objects":74380,"num_object_clones":0,"num_object_copies":297520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74380,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132707,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4888,"num_bytes_recovered":37593088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1203032064,"data_stored":1203032064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132734,"ondisk_log_size":132734,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600748,"kb_used_data":298716,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93771092,"statfs":{"total":96636764160,"available":96021598208,"internally_reserved":0,"allocated":305885184,"data_stored":305709718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,3,2,2,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.1849999999999996},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.90100000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.9830000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.911}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":11.677},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":11.622}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607864,"kb_used_data":298020,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93763976,"statfs":{"total":96636764160,"available":96014311424,"internally_reserved":0,"allocated":305172480,"data_stored":304997014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.8689999999999998},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.28999999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":4.7779999999999996},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.9470000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593504,"kb_used_data":283220,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93778336,"statfs":{"total":96636764160,"available":96029016064,"internally_reserved":0,"allocated":290017280,"data_stored":289841814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.1400000000000006},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":5.8810000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":7.819},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.7619999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":4.4820000000000002},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.9690000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607716,"kb_used_data":297432,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93764124,"statfs":{"total":96636764160,"available":96014462976,"internally_reserved":0,"allocated":304570368,"data_stored":304394902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,0,3,1,1,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.4539999999999997},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":12.75}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.23499999999999999},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":12.853999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.25},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":12.789999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":304046080,"data_stored":304046080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289705984,"data_stored":289705984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304259072,"data_stored":304259072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305020928,"data_stored":305020928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:18.409 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:18.958 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:18.958 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:18.982 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-18T11:50:17.974089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":609273228,"num_objects":74380,"num_object_clones":0,"num_object_copies":297520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74380,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132707,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4888,"num_bytes_recovered":37593088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132734,"ondisk_log_size":132734,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2409832,"kb_used_data":1177388,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":375077528,"statfs":{"total":386547056640,"available":384079388672,"internally_reserved":0,"allocated":1205645312,"data_stored":1204943448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,6,7,2,12,9,3,2,0,0,0,4,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28524544,"num_objects":-3482,"num_object_clones":0,"num_object_copies":-13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009229"},"pg_stats":[{"pgid":"1.f","version":"35'8460","reported_seq":8553,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.372072+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:16.372072+0000","last_peered":"2024-09-18T11:50:16.372072+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:16.372072+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8460,"log_dups_size":0,"ondisk_log_size":8460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39239680,"num_objects":4790,"num_object_clones":0,"num_object_copies":19160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8460,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2684}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8369","reported_seq":12659,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:16.463923+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:16.463923+0000","last_peered":"2024-09-18T11:50:16.463923+0000","last_clean":"2024-09-18T11:50:16.463923+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:16.463923+0000","last_undegraded":"2024-09-18T11:50:16.463923+0000","last_fullsized":"2024-09-18T11:50:16.463923+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8369,"log_dups_size":0,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38393078,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8197","reported_seq":8312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.064648+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:17.064648+0000","last_peered":"2024-09-18T11:50:17.064648+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:17.064648+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8197,"log_dups_size":0,"ondisk_log_size":8197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37771923,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8196,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2572}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8325","reported_seq":8411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.348813+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:17.348813+0000","last_peered":"2024-09-18T11:50:17.348813+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:17.348813+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37688566,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8271","reported_seq":8378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.344377+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:16.344377+0000","last_peered":"2024-09-18T11:50:16.344377+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:16.344377+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8271,"log_dups_size":0,"ondisk_log_size":8271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876982,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8269,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8273","reported_seq":8388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.247515+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:16.247515+0000","last_peered":"2024-09-18T11:50:16.247515+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:16.247515+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36970496,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8273,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8381","reported_seq":8481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.386834+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:17.386834+0000","last_peered":"2024-09-18T11:50:17.386834+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:17.386834+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37967094,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2517}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8245","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.483717+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:17.483717+0000","last_peered":"2024-09-18T11:50:17.483717+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:17.483717+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926134,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8243,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2613}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8471","reported_seq":9370,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.400465+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:16.400465+0000","last_peered":"2024-09-18T11:50:16.400465+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:16.400465+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38537708,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8468,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":3244032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8029","reported_seq":8141,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.161697+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:17.161697+0000","last_peered":"2024-09-18T11:50:17.161697+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:17.161697+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8029,"log_dups_size":0,"ondisk_log_size":8029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393654,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2602}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8306","reported_seq":8419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.272977+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:16.272977+0000","last_peered":"2024-09-18T11:50:16.272977+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:16.272977+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38719345,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8345","reported_seq":8428,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.326007+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:16.326007+0000","last_peered":"2024-09-18T11:50:16.326007+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:16.326007+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8345,"log_dups_size":0,"ondisk_log_size":8345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38386299,"num_objects":4686,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2598}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8257","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.122070+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:17.122070+0000","last_peered":"2024-09-18T11:50:17.122070+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:17.122070+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8257,"log_dups_size":0,"ondisk_log_size":8257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38582907,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2661}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8406","reported_seq":8525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.392915+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:16.392915+0000","last_peered":"2024-09-18T11:50:16.392915+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:16.392915+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38368502,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8280","reported_seq":12567,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:16.342801+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:16.342801+0000","last_peered":"2024-09-18T11:50:16.342801+0000","last_clean":"2024-09-18T11:50:16.342801+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:16.342801+0000","last_undegraded":"2024-09-18T11:50:16.342801+0000","last_fullsized":"2024-09-18T11:50:16.342801+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37868790,"num_objects":4623,"num_object_clones":0,"num_object_copies":18492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8278,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8119","reported_seq":8205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.039632+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:17.039632+0000","last_peered":"2024-09-18T11:50:17.039632+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:17.039632+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8119,"log_dups_size":0,"ondisk_log_size":8119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37582070,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":609273228,"num_objects":74380,"num_object_clones":0,"num_object_copies":297520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74380,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132707,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4888,"num_bytes_recovered":37593088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1203032064,"data_stored":1203032064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132734,"ondisk_log_size":132734,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600748,"kb_used_data":298716,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93771092,"statfs":{"total":96636764160,"available":96021598208,"internally_reserved":0,"allocated":305885184,"data_stored":305709718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,3,2,2,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.1849999999999996},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.90100000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.9830000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.911}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":11.677},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":11.622}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607864,"kb_used_data":298020,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93763976,"statfs":{"total":96636764160,"available":96014311424,"internally_reserved":0,"allocated":305172480,"data_stored":304997014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.8689999999999998},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.28999999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":4.7779999999999996},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.9470000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593504,"kb_used_data":283220,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93778336,"statfs":{"total":96636764160,"available":96029016064,"internally_reserved":0,"allocated":290017280,"data_stored":289841814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.1400000000000006},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":5.8810000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":7.819},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.7619999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":4.4820000000000002},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.9690000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607716,"kb_used_data":297432,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93764124,"statfs":{"total":96636764160,"available":96014462976,"internally_reserved":0,"allocated":304570368,"data_stored":304394902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,0,3,1,1,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.4539999999999997},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":12.75}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.23499999999999999},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":12.853999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.25},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":12.789999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":304046080,"data_stored":304046080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289705984,"data_stored":289705984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304259072,"data_stored":304259072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305020928,"data_stored":305020928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:18.984 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:18.984 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8460", 'reported_seq': 8553, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:16.372072+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:16.372072+0000', 'last_peered': '2024-09-18T11:50:16.372072+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:16.372072+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8460, 'log_dups_size': 0, 'ondisk_log_size': 8460, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39239680, 'num_objects': 4790, 'num_object_clones': 0, 'num_object_copies': 19160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8460, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2684}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:18.984 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:18.984 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8197", 'reported_seq': 8312, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:17.064648+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:17.064648+0000', 'last_peered': '2024-09-18T11:50:17.064648+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:17.064648+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8197, 'log_dups_size': 0, 'ondisk_log_size': 8197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37771923, 'num_objects': 4612, 'num_object_clones': 0, 'num_object_copies': 18448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4612, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8196, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2572}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:18.984 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:18.984 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8325", 'reported_seq': 8411, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:17.348813+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:17.348813+0000', 'last_peered': '2024-09-18T11:50:17.348813+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:17.348813+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8325, 'log_dups_size': 0, 'ondisk_log_size': 8325, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37688566, 'num_objects': 4601, 'num_object_clones': 0, 'num_object_copies': 18404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8323, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2497}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:18.984 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:18.984 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8271", 'reported_seq': 8378, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:16.344377+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:16.344377+0000', 'last_peered': '2024-09-18T11:50:16.344377+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:16.344377+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8271, 'log_dups_size': 0, 'ondisk_log_size': 8271, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37876982, 'num_objects': 4624, 'num_object_clones': 0, 'num_object_copies': 18496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8269, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2566}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8273", 'reported_seq': 8388, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:16.247515+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:16.247515+0000', 'last_peered': '2024-09-18T11:50:16.247515+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:16.247515+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8273, 'log_dups_size': 0, 'ondisk_log_size': 8273, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36970496, 'num_objects': 4513, 'num_object_clones': 0, 'num_object_copies': 18052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8273, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2393}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8381", 'reported_seq': 8481, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:17.386834+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:17.386834+0000', 'last_peered': '2024-09-18T11:50:17.386834+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:17.386834+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8381, 'log_dups_size': 0, 'ondisk_log_size': 8381, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37967094, 'num_objects': 4635, 'num_object_clones': 0, 'num_object_copies': 18540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8379, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2517}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8245", 'reported_seq': 8346, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:17.483717+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:17.483717+0000', 'last_peered': '2024-09-18T11:50:17.483717+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:17.483717+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8245, 'log_dups_size': 0, 'ondisk_log_size': 8245, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37926134, 'num_objects': 4630, 'num_object_clones': 0, 'num_object_copies': 18520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2019, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8243, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2613}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8471", 'reported_seq': 9370, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:16.400465+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:16.400465+0000', 'last_peered': '2024-09-18T11:50:16.400465+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:16.400465+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8471, 'log_dups_size': 0, 'ondisk_log_size': 8471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38537708, 'num_objects': 4705, 'num_object_clones': 0, 'num_object_copies': 18820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1720, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8468, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 421, 'num_bytes_recovered': 3244032, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2985}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8029", 'reported_seq': 8141, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:17.161697+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:17.161697+0000', 'last_peered': '2024-09-18T11:50:17.161697+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:17.161697+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8029, 'log_dups_size': 0, 'ondisk_log_size': 8029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37393654, 'num_objects': 4565, 'num_object_clones': 0, 'num_object_copies': 18260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8027, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2602}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:18.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8306", 'reported_seq': 8419, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:16.272977+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:16.272977+0000', 'last_peered': '2024-09-18T11:50:16.272977+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:16.272977+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8306, 'log_dups_size': 0, 'ondisk_log_size': 8306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38719345, 'num_objects': 4727, 'num_object_clones': 0, 'num_object_copies': 18908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8303, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2690}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:18.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:18.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8345", 'reported_seq': 8428, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:16.326007+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:16.326007+0000', 'last_peered': '2024-09-18T11:50:16.326007+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:16.326007+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8345, 'log_dups_size': 0, 'ondisk_log_size': 8345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38386299, 'num_objects': 4686, 'num_object_clones': 0, 'num_object_copies': 18744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2089, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8344, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2598}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:18.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:18.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8257", 'reported_seq': 8346, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:17.122070+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:17.122070+0000', 'last_peered': '2024-09-18T11:50:17.122070+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:17.122070+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8257, 'log_dups_size': 0, 'ondisk_log_size': 8257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38582907, 'num_objects': 4710, 'num_object_clones': 0, 'num_object_copies': 18840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8256, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2661}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:18.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:18.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8406", 'reported_seq': 8525, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:16.392915+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:16.392915+0000', 'last_peered': '2024-09-18T11:50:16.392915+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:16.392915+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8406, 'log_dups_size': 0, 'ondisk_log_size': 8406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38368502, 'num_objects': 4684, 'num_object_clones': 0, 'num_object_copies': 18736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2127, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8404, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:18.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:18.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8119", 'reported_seq': 8205, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:17.039632+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:17.039632+0000', 'last_peered': '2024-09-18T11:50:17.039632+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:17.039632+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8119, 'log_dups_size': 0, 'ondisk_log_size': 8119, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37582070, 'num_objects': 4588, 'num_object_clones': 0, 'num_object_copies': 18352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8117, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2585}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:18.987 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:19.721 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:19.721 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28568/297520 objects degraded (9.602%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":417,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74380,"data_bytes":609273228,"bytes_used":2467667968,"bytes_avail":384079388672,"bytes_total":386547056640,"degraded_objects":28568,"degraded_total":297520,"degraded_ratio":0.096020435600967999,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":109142,"recovering_keys_per_sec":0,"num_objects_recovered":209,"num_bytes_recovered":1310720,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":289},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:19.745 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:19.746 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:20.280 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:20.281 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:20.312 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-18T11:50:19.974881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":609273228,"num_objects":74380,"num_object_clones":0,"num_object_copies":297520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74380,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132707,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4888,"num_bytes_recovered":37593088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132734,"ondisk_log_size":132734,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2409832,"kb_used_data":1177388,"kb_used_omap":14,"kb_used_meta":1232369,"kb_avail":375077528,"statfs":{"total":386547056640,"available":384079388672,"internally_reserved":0,"allocated":1205645312,"data_stored":1204943448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1261945883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,6,7,2,12,9,3,2,0,0,0,4,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19152896,"num_objects":-2338,"num_object_clones":0,"num_object_copies":-9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009054"},"pg_stats":[{"pgid":"1.f","version":"35'8460","reported_seq":8553,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.372072+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:16.372072+0000","last_peered":"2024-09-18T11:50:16.372072+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:16.372072+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8460,"log_dups_size":0,"ondisk_log_size":8460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39239680,"num_objects":4790,"num_object_clones":0,"num_object_copies":19160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8460,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2684}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8369","reported_seq":12659,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:16.463923+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:16.463923+0000","last_peered":"2024-09-18T11:50:16.463923+0000","last_clean":"2024-09-18T11:50:16.463923+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:16.463923+0000","last_undegraded":"2024-09-18T11:50:16.463923+0000","last_fullsized":"2024-09-18T11:50:16.463923+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8369,"log_dups_size":0,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38393078,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8197","reported_seq":8312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.064648+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:17.064648+0000","last_peered":"2024-09-18T11:50:17.064648+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:17.064648+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8197,"log_dups_size":0,"ondisk_log_size":8197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37771923,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8196,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2572}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8325","reported_seq":8411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.348813+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:17.348813+0000","last_peered":"2024-09-18T11:50:17.348813+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:17.348813+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37688566,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8271","reported_seq":8378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.344377+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:16.344377+0000","last_peered":"2024-09-18T11:50:16.344377+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:16.344377+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8271,"log_dups_size":0,"ondisk_log_size":8271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876982,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8269,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8273","reported_seq":8388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.247515+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:16.247515+0000","last_peered":"2024-09-18T11:50:16.247515+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:16.247515+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36970496,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8273,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8381","reported_seq":8481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.386834+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:17.386834+0000","last_peered":"2024-09-18T11:50:17.386834+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:17.386834+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37967094,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2517}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8245","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.483717+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:17.483717+0000","last_peered":"2024-09-18T11:50:17.483717+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:17.483717+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926134,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8243,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2613}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8471","reported_seq":9370,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.400465+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:16.400465+0000","last_peered":"2024-09-18T11:50:16.400465+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:16.400465+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38537708,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8468,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":3244032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8029","reported_seq":8141,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.161697+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:17.161697+0000","last_peered":"2024-09-18T11:50:17.161697+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:17.161697+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8029,"log_dups_size":0,"ondisk_log_size":8029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393654,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2602}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8306","reported_seq":8419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.272977+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:16.272977+0000","last_peered":"2024-09-18T11:50:16.272977+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:16.272977+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38719345,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8345","reported_seq":8428,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.326007+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:16.326007+0000","last_peered":"2024-09-18T11:50:16.326007+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:16.326007+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8345,"log_dups_size":0,"ondisk_log_size":8345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38386299,"num_objects":4686,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2598}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8257","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.122070+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:17.122070+0000","last_peered":"2024-09-18T11:50:17.122070+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:17.122070+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8257,"log_dups_size":0,"ondisk_log_size":8257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38582907,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2661}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8406","reported_seq":8525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:16.392915+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:16.392915+0000","last_peered":"2024-09-18T11:50:16.392915+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:16.392915+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38368502,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8280","reported_seq":12567,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:16.342801+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:16.342801+0000","last_peered":"2024-09-18T11:50:16.342801+0000","last_clean":"2024-09-18T11:50:16.342801+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:16.342801+0000","last_undegraded":"2024-09-18T11:50:16.342801+0000","last_fullsized":"2024-09-18T11:50:16.342801+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37868790,"num_objects":4623,"num_object_clones":0,"num_object_copies":18492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8278,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8119","reported_seq":8205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.039632+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:17.039632+0000","last_peered":"2024-09-18T11:50:17.039632+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:17.039632+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8119,"log_dups_size":0,"ondisk_log_size":8119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37582070,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":609273228,"num_objects":74380,"num_object_clones":0,"num_object_copies":297520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74380,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132707,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4888,"num_bytes_recovered":37593088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1203032064,"data_stored":1203032064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132734,"ondisk_log_size":132734,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600748,"kb_used_data":298716,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93771092,"statfs":{"total":96636764160,"available":96021598208,"internally_reserved":0,"allocated":305885184,"data_stored":305709718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,3,2,2,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":5.5119999999999996,"5min":4.4119999999999999,"15min":3.3380000000000001},"min":{"1min":0.32400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":18.751999999999999,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.1849999999999996},{"interface":"front","average":{"1min":5.5389999999999997,"5min":4.7930000000000001,"15min":3.3809999999999998},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":18.774000000000001,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.90100000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.9830000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.911}]},{"osd":2,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":3.9140000000000001,"15min":1.9630000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":19.850999999999999,"5min":29.087,"15min":29.087},"last":11.677},{"interface":"front","average":{"1min":6.1710000000000003,"5min":4.4130000000000003,"15min":3.198},"min":{"1min":0.29499999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.167999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":11.622}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607864,"kb_used_data":298020,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93763976,"statfs":{"total":96636764160,"available":96014311424,"internally_reserved":0,"allocated":305172480,"data_stored":304997014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.8689999999999998},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.28999999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":4.7779999999999996},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.9470000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593504,"kb_used_data":283220,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93778336,"statfs":{"total":96636764160,"available":96029016064,"internally_reserved":0,"allocated":290017280,"data_stored":289841814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.1400000000000006},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":5.8810000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":7.819},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.7619999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":4.4820000000000002},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.9690000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607716,"kb_used_data":297432,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93764124,"statfs":{"total":96636764160,"available":96014462976,"internally_reserved":0,"allocated":304570368,"data_stored":304394902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,0,3,1,1,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.4539999999999997},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":12.75}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.23499999999999999},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":12.853999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.25},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":12.789999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":304046080,"data_stored":304046080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289705984,"data_stored":289705984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304259072,"data_stored":304259072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305020928,"data_stored":305020928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:23.315 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:23.863 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:23.890 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-18T11:50:21.975459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":604276108,"num_objects":73770,"num_object_clones":0,"num_object_copies":295080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133317,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4966,"num_bytes_recovered":38191104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133344,"ondisk_log_size":133344,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2400772,"kb_used_data":1172500,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375086588,"statfs":{"total":386547056640,"available":384088666112,"internally_reserved":0,"allocated":1200640000,"data_stored":1199938136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,5,4,16,17,5,8,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24150016,"num_objects":-2948,"num_object_clones":0,"num_object_copies":-11792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2948,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004404"},"pg_stats":[{"pgid":"1.f","version":"35'8547","reported_seq":8640,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.405508+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:21.405508+0000","last_peered":"2024-09-18T11:50:21.405508+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:21.405508+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38526976,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8547,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8429","reported_seq":12719,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:21.035794+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:21.035794+0000","last_peered":"2024-09-18T11:50:21.035794+0000","last_clean":"2024-09-18T11:50:21.035794+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:21.035794+0000","last_undegraded":"2024-09-18T11:50:21.035794+0000","last_fullsized":"2024-09-18T11:50:21.035794+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8429,"log_dups_size":0,"ondisk_log_size":8429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37901558,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8427,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8197","reported_seq":8312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.064648+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:17.064648+0000","last_peered":"2024-09-18T11:50:17.064648+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:17.064648+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8197,"log_dups_size":0,"ondisk_log_size":8197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37771923,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8196,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2572}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8325","reported_seq":8411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.348813+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:17.348813+0000","last_peered":"2024-09-18T11:50:17.348813+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:17.348813+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37688566,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8330","reported_seq":8441,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.379215+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:21.379215+0000","last_peered":"2024-09-18T11:50:21.379215+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:21.379215+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8330,"log_dups_size":0,"ondisk_log_size":8330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393654,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8331","reported_seq":8446,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.276005+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:21.276005+0000","last_peered":"2024-09-18T11:50:21.276005+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:21.276005+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8331,"log_dups_size":0,"ondisk_log_size":8331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36495360,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8331,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8381","reported_seq":8481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.386834+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:17.386834+0000","last_peered":"2024-09-18T11:50:17.386834+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:17.386834+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37967094,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2517}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8245","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.483717+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:17.483717+0000","last_peered":"2024-09-18T11:50:17.483717+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:17.483717+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926134,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8243,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2613}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8544","reported_seq":9589,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.428170+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:21.428170+0000","last_peered":"2024-09-18T11:50:21.428170+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:21.428170+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37939692,"num_objects":4632,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8541,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":494,"num_bytes_recovered":3842048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8029","reported_seq":8141,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.161697+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:17.161697+0000","last_peered":"2024-09-18T11:50:17.161697+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:17.161697+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8029,"log_dups_size":0,"ondisk_log_size":8029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393654,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2602}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8366","reported_seq":8483,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.337718+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:21.337718+0000","last_peered":"2024-09-18T11:50:21.337718+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:21.337718+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8366,"log_dups_size":0,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38227825,"num_objects":4667,"num_object_clones":0,"num_object_copies":18668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8421","reported_seq":8506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.447892+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:21.447892+0000","last_peered":"2024-09-18T11:50:21.447892+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:21.447892+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37763707,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8420,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2523}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8257","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.122070+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:17.122070+0000","last_peered":"2024-09-18T11:50:17.122070+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:17.122070+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8257,"log_dups_size":0,"ondisk_log_size":8257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38582907,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2661}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8480","reported_seq":8599,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.375527+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:21.375527+0000","last_peered":"2024-09-18T11:50:21.375527+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:21.375527+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37762294,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2485}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8343","reported_seq":12630,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:21.338561+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:21.338561+0000","last_peered":"2024-09-18T11:50:21.338561+0000","last_clean":"2024-09-18T11:50:21.338561+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:21.338561+0000","last_undegraded":"2024-09-18T11:50:21.338561+0000","last_fullsized":"2024-09-18T11:50:21.338561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8343,"log_dups_size":0,"ondisk_log_size":8343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37352694,"num_objects":4560,"num_object_clones":0,"num_object_copies":18240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8341,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8119","reported_seq":8205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:17.039632+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:17.039632+0000","last_peered":"2024-09-18T11:50:17.039632+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:17.039632+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8119,"log_dups_size":0,"ondisk_log_size":8119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37582070,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":604276108,"num_objects":73770,"num_object_clones":0,"num_object_copies":295080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133317,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4966,"num_bytes_recovered":38191104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1198161920,"data_stored":1198161920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133344,"ondisk_log_size":133344,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600468,"kb_used_data":294284,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93771372,"statfs":{"total":96636764160,"available":96021884928,"internally_reserved":0,"allocated":301346816,"data_stored":301171350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,5,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.3799999999999999},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":5.8970000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":5.492},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":2.1869999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":6.1699999999999999},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.84299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607864,"kb_used_data":298020,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93763976,"statfs":{"total":96636764160,"available":96014311424,"internally_reserved":0,"allocated":305172480,"data_stored":304997014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":8.8689999999999998},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.28999999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:49:17 2024","interfaces":[{"interface":"back","average":{"1min":4.8550000000000004,"5min":3.7719999999999998,"15min":2.8540000000000001},"min":{"1min":0.48699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.394,"5min":14.394,"15min":14.394},"last":4.7779999999999996},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.9020000000000001,"15min":2.1000000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.151,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.9470000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584724,"kb_used_data":282764,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93787116,"statfs":{"total":96636764160,"available":96038006784,"internally_reserved":0,"allocated":289550336,"data_stored":289374870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.5070000000000001},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":11.741}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.17299999999999999},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.0149999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.20799999999999999},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.5499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607716,"kb_used_data":297432,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93764124,"statfs":{"total":96636764160,"available":96014462976,"internally_reserved":0,"allocated":304570368,"data_stored":304394902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,0,3,1,1,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.4539999999999997},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":12.75}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.23499999999999999},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":12.853999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:16 2024","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.3599999999999999,"15min":2.694},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.749000000000001,"5min":18.832000000000001,"15min":18.832000000000001},"last":1.25},{"interface":"front","average":{"1min":6.6449999999999996,"5min":4.3979999999999997,"15min":2.48},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":14.198,"5min":19.038,"15min":19.038},"last":12.789999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":304046080,"data_stored":304046080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289239040,"data_stored":289239040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304259072,"data_stored":304259072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300617728,"data_stored":300617728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:23.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:24.375 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:24.376 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:24.398 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-18T11:50:23.976248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600647052,"num_objects":73327,"num_object_clones":0,"num_object_copies":293308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133760,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4969,"num_bytes_recovered":38191104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133787,"ondisk_log_size":133787,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2383960,"kb_used_data":1164012,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375103400,"statfs":{"total":386547056640,"available":384105881600,"internally_reserved":0,"allocated":1191948288,"data_stored":1191246424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,4,18,15,5,8,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22306816,"num_objects":-2723,"num_object_clones":0,"num_object_copies":-10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2723,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004574"},"pg_stats":[{"pgid":"1.f","version":"35'8547","reported_seq":8640,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.405508+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:21.405508+0000","last_peered":"2024-09-18T11:50:21.405508+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:21.405508+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38526976,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8547,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8429","reported_seq":12719,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:21.035794+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:21.035794+0000","last_peered":"2024-09-18T11:50:21.035794+0000","last_clean":"2024-09-18T11:50:21.035794+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:21.035794+0000","last_undegraded":"2024-09-18T11:50:21.035794+0000","last_fullsized":"2024-09-18T11:50:21.035794+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8429,"log_dups_size":0,"ondisk_log_size":8429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37901558,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8427,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8263","reported_seq":8378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.163234+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:22.163234+0000","last_peered":"2024-09-18T11:50:22.163234+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:22.163234+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8263,"log_dups_size":0,"ondisk_log_size":8263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37231251,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8262,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2506}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8382","reported_seq":8472,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.377913+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:22.377913+0000","last_peered":"2024-09-18T11:50:22.377913+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:22.377913+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8382,"log_dups_size":0,"ondisk_log_size":8382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37221622,"num_objects":4544,"num_object_clones":0,"num_object_copies":18176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8380,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8330","reported_seq":8441,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.379215+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:21.379215+0000","last_peered":"2024-09-18T11:50:21.379215+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:21.379215+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8330,"log_dups_size":0,"ondisk_log_size":8330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393654,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8331","reported_seq":8446,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.276005+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:21.276005+0000","last_peered":"2024-09-18T11:50:21.276005+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:21.276005+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8331,"log_dups_size":0,"ondisk_log_size":8331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36495360,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8331,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8442","reported_seq":8543,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.219509+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:22.219509+0000","last_peered":"2024-09-18T11:50:22.219509+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:22.219509+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467382,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2456}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8309","reported_seq":8412,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.415512+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:22.415512+0000","last_peered":"2024-09-18T11:50:22.415512+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:22.415512+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8309,"log_dups_size":0,"ondisk_log_size":8309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37401846,"num_objects":4566,"num_object_clones":0,"num_object_copies":18264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2550}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8544","reported_seq":9589,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.428170+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:21.428170+0000","last_peered":"2024-09-18T11:50:21.428170+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:21.428170+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37939692,"num_objects":4632,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8541,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":494,"num_bytes_recovered":3842048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8086","reported_seq":8198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.983012+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:21.983012+0000","last_peered":"2024-09-18T11:50:21.983012+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:21.983012+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36926710,"num_objects":4508,"num_object_clones":0,"num_object_copies":18032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8084,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8366","reported_seq":8483,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.337718+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:21.337718+0000","last_peered":"2024-09-18T11:50:21.337718+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:21.337718+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8366,"log_dups_size":0,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38227825,"num_objects":4667,"num_object_clones":0,"num_object_copies":18668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8421","reported_seq":8506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.447892+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:21.447892+0000","last_peered":"2024-09-18T11:50:21.447892+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:21.447892+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37763707,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8420,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2523}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8325","reported_seq":8415,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.114284+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:22.114284+0000","last_peered":"2024-09-18T11:50:22.114284+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:22.114284+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38025851,"num_objects":4642,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8480","reported_seq":8599,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.375527+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:21.375527+0000","last_peered":"2024-09-18T11:50:21.375527+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:21.375527+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37762294,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2485}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8343","reported_seq":12630,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:21.338561+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:21.338561+0000","last_peered":"2024-09-18T11:50:21.338561+0000","last_clean":"2024-09-18T11:50:21.338561+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:21.338561+0000","last_undegraded":"2024-09-18T11:50:21.338561+0000","last_fullsized":"2024-09-18T11:50:21.338561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8343,"log_dups_size":0,"ondisk_log_size":8343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37352694,"num_objects":4560,"num_object_clones":0,"num_object_copies":18240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8341,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8189","reported_seq":8275,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.482619+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:22.482619+0000","last_peered":"2024-09-18T11:50:22.482619+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:22.482619+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8189,"log_dups_size":0,"ondisk_log_size":8189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37008630,"num_objects":4518,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8187,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2515}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":600647052,"num_objects":73327,"num_object_clones":0,"num_object_copies":293308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133760,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4969,"num_bytes_recovered":38191104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1189568512,"data_stored":1189568512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133787,"ondisk_log_size":133787,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600468,"kb_used_data":294284,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93771372,"statfs":{"total":96636764160,"available":96021884928,"internally_reserved":0,"allocated":301346816,"data_stored":301171350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,5,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.3799999999999999},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":5.8970000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":5.492},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":2.1869999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":6.1699999999999999},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.84299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591100,"kb_used_data":293736,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93780740,"statfs":{"total":96636764160,"available":96031477760,"internally_reserved":0,"allocated":300785664,"data_stored":300610198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,1,0,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":8.8699999999999992},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":8.9039999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":9.2200000000000006},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":8.9570000000000007}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.9339999999999993},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.1899999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584724,"kb_used_data":282764,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93787116,"statfs":{"total":96636764160,"available":96038006784,"internally_reserved":0,"allocated":289550336,"data_stored":289374870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.5070000000000001},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":11.741}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.17299999999999999},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.0149999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.20799999999999999},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.5499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607668,"kb_used_data":293228,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93764172,"statfs":{"total":96636764160,"available":96014512128,"internally_reserved":0,"allocated":300265472,"data_stored":300090006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,3,1,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.4589999999999996},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":2.0049999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":4.7320000000000002},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":4.4160000000000004},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":5.4450000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":299745280,"data_stored":299745280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289239040,"data_stored":289239040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299966464,"data_stored":299966464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300617728,"data_stored":300617728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:24.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:24.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8547", 'reported_seq': 8640, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:21.405508+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:21.405508+0000', 'last_peered': '2024-09-18T11:50:21.405508+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:21.405508+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8547, 'log_dups_size': 0, 'ondisk_log_size': 8547, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38526976, 'num_objects': 4703, 'num_object_clones': 0, 'num_object_copies': 18812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8547, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2597}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:24.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:24.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8263", 'reported_seq': 8378, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:22.163234+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:22.163234+0000', 'last_peered': '2024-09-18T11:50:22.163234+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:22.163234+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8263, 'log_dups_size': 0, 'ondisk_log_size': 8263, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37231251, 'num_objects': 4546, 'num_object_clones': 0, 'num_object_copies': 18184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4546, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8262, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2506}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:24.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:24.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8382", 'reported_seq': 8472, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:22.377913+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:22.377913+0000', 'last_peered': '2024-09-18T11:50:22.377913+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:22.377913+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8382, 'log_dups_size': 0, 'ondisk_log_size': 8382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37221622, 'num_objects': 4544, 'num_object_clones': 0, 'num_object_copies': 18176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4544, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8380, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2442}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:24.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:24.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8330", 'reported_seq': 8441, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:21.379215+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:21.379215+0000', 'last_peered': '2024-09-18T11:50:21.379215+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:21.379215+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8330, 'log_dups_size': 0, 'ondisk_log_size': 8330, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37393654, 'num_objects': 4565, 'num_object_clones': 0, 'num_object_copies': 18260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8328, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2509}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:24.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:24.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8331", 'reported_seq': 8446, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:21.276005+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:21.276005+0000', 'last_peered': '2024-09-18T11:50:21.276005+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:21.276005+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8331, 'log_dups_size': 0, 'ondisk_log_size': 8331, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36495360, 'num_objects': 4455, 'num_object_clones': 0, 'num_object_copies': 17820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8331, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:24.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:24.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8442", 'reported_seq': 8543, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:22.219509+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:22.219509+0000', 'last_peered': '2024-09-18T11:50:22.219509+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:22.219509+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8442, 'log_dups_size': 0, 'ondisk_log_size': 8442, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37467382, 'num_objects': 4574, 'num_object_clones': 0, 'num_object_copies': 18296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8440, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2456}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8309", 'reported_seq': 8412, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:22.415512+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:22.415512+0000', 'last_peered': '2024-09-18T11:50:22.415512+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:22.415512+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8309, 'log_dups_size': 0, 'ondisk_log_size': 8309, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37401846, 'num_objects': 4566, 'num_object_clones': 0, 'num_object_copies': 18264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2018, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8307, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2550}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8544", 'reported_seq': 9589, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:21.428170+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:21.428170+0000', 'last_peered': '2024-09-18T11:50:21.428170+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:21.428170+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8544, 'log_dups_size': 0, 'ondisk_log_size': 8544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37939692, 'num_objects': 4632, 'num_object_clones': 0, 'num_object_copies': 18528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1647, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4632, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8541, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 494, 'num_bytes_recovered': 3842048, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2985}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8086", 'reported_seq': 8198, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:21.983012+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:21.983012+0000', 'last_peered': '2024-09-18T11:50:21.983012+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:21.983012+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8086, 'log_dups_size': 0, 'ondisk_log_size': 8086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36926710, 'num_objects': 4508, 'num_object_clones': 0, 'num_object_copies': 18032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8084, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2545}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8366", 'reported_seq': 8483, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:21.337718+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:21.337718+0000', 'last_peered': '2024-09-18T11:50:21.337718+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:21.337718+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8366, 'log_dups_size': 0, 'ondisk_log_size': 8366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38227825, 'num_objects': 4667, 'num_object_clones': 0, 'num_object_copies': 18668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8363, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2632}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8421", 'reported_seq': 8506, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:21.447892+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:21.447892+0000', 'last_peered': '2024-09-18T11:50:21.447892+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:21.447892+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8421, 'log_dups_size': 0, 'ondisk_log_size': 8421, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37763707, 'num_objects': 4610, 'num_object_clones': 0, 'num_object_copies': 18440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2088, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8420, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2523}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:24.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8325", 'reported_seq': 8415, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:22.114284+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:22.114284+0000', 'last_peered': '2024-09-18T11:50:22.114284+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:22.114284+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8325, 'log_dups_size': 0, 'ondisk_log_size': 8325, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38025851, 'num_objects': 4642, 'num_object_clones': 0, 'num_object_copies': 18568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8324, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:24.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:24.402 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8480", 'reported_seq': 8599, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:21.375527+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:21.375527+0000', 'last_peered': '2024-09-18T11:50:21.375527+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:21.375527+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8480, 'log_dups_size': 0, 'ondisk_log_size': 8480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37762294, 'num_objects': 4610, 'num_object_clones': 0, 'num_object_copies': 18440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2127, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8478, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2485}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:24.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:24.402 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8189", 'reported_seq': 8275, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:22.482619+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:22.482619+0000', 'last_peered': '2024-09-18T11:50:22.482619+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:22.482619+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8189, 'log_dups_size': 0, 'ondisk_log_size': 8189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37008630, 'num_objects': 4518, 'num_object_clones': 0, 'num_object_copies': 18072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8187, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2515}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:24.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:25.038 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:25.038 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28487/293308 objects degraded (9.712%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":423,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":73327,"data_bytes":600647052,"bytes_used":2441175040,"bytes_avail":384105881600,"bytes_total":386547056640,"degraded_objects":28487,"degraded_total":293308,"degraded_ratio":0.097123160636600439,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":85983,"recovering_keys_per_sec":0,"num_objects_recovered":148,"num_bytes_recovered":1032192,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":226},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:25.064 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:25.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:25.577 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:25.578 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:25.601 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-18T11:50:23.976248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600647052,"num_objects":73327,"num_object_clones":0,"num_object_copies":293308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133760,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4969,"num_bytes_recovered":38191104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133787,"ondisk_log_size":133787,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2383960,"kb_used_data":1164012,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375103400,"statfs":{"total":386547056640,"available":384105881600,"internally_reserved":0,"allocated":1191948288,"data_stored":1191246424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,4,18,15,5,8,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22306816,"num_objects":-2723,"num_object_clones":0,"num_object_copies":-10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2723,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004574"},"pg_stats":[{"pgid":"1.f","version":"35'8547","reported_seq":8640,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.405508+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:21.405508+0000","last_peered":"2024-09-18T11:50:21.405508+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:21.405508+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38526976,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8547,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8429","reported_seq":12719,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:21.035794+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:21.035794+0000","last_peered":"2024-09-18T11:50:21.035794+0000","last_clean":"2024-09-18T11:50:21.035794+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:21.035794+0000","last_undegraded":"2024-09-18T11:50:21.035794+0000","last_fullsized":"2024-09-18T11:50:21.035794+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8429,"log_dups_size":0,"ondisk_log_size":8429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37901558,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8427,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8263","reported_seq":8378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.163234+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:22.163234+0000","last_peered":"2024-09-18T11:50:22.163234+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:22.163234+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8263,"log_dups_size":0,"ondisk_log_size":8263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37231251,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8262,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2506}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8382","reported_seq":8472,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.377913+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:22.377913+0000","last_peered":"2024-09-18T11:50:22.377913+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:22.377913+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8382,"log_dups_size":0,"ondisk_log_size":8382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37221622,"num_objects":4544,"num_object_clones":0,"num_object_copies":18176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8380,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8330","reported_seq":8441,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.379215+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:21.379215+0000","last_peered":"2024-09-18T11:50:21.379215+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:21.379215+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8330,"log_dups_size":0,"ondisk_log_size":8330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393654,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8331","reported_seq":8446,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.276005+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:21.276005+0000","last_peered":"2024-09-18T11:50:21.276005+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:21.276005+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8331,"log_dups_size":0,"ondisk_log_size":8331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36495360,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8331,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8442","reported_seq":8543,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.219509+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:22.219509+0000","last_peered":"2024-09-18T11:50:22.219509+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:22.219509+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467382,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2456}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8309","reported_seq":8412,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.415512+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:22.415512+0000","last_peered":"2024-09-18T11:50:22.415512+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:22.415512+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8309,"log_dups_size":0,"ondisk_log_size":8309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37401846,"num_objects":4566,"num_object_clones":0,"num_object_copies":18264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2550}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8544","reported_seq":9589,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.428170+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:21.428170+0000","last_peered":"2024-09-18T11:50:21.428170+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:21.428170+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37939692,"num_objects":4632,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8541,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":494,"num_bytes_recovered":3842048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8086","reported_seq":8198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.983012+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:21.983012+0000","last_peered":"2024-09-18T11:50:21.983012+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:21.983012+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36926710,"num_objects":4508,"num_object_clones":0,"num_object_copies":18032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8084,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8366","reported_seq":8483,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.337718+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:21.337718+0000","last_peered":"2024-09-18T11:50:21.337718+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:21.337718+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8366,"log_dups_size":0,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38227825,"num_objects":4667,"num_object_clones":0,"num_object_copies":18668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8421","reported_seq":8506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.447892+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:21.447892+0000","last_peered":"2024-09-18T11:50:21.447892+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:21.447892+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37763707,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8420,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2523}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8325","reported_seq":8415,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.114284+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:22.114284+0000","last_peered":"2024-09-18T11:50:22.114284+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:22.114284+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38025851,"num_objects":4642,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8480","reported_seq":8599,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:21.375527+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:21.375527+0000","last_peered":"2024-09-18T11:50:21.375527+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:21.375527+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37762294,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2485}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8343","reported_seq":12630,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:21.338561+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:21.338561+0000","last_peered":"2024-09-18T11:50:21.338561+0000","last_clean":"2024-09-18T11:50:21.338561+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:21.338561+0000","last_undegraded":"2024-09-18T11:50:21.338561+0000","last_fullsized":"2024-09-18T11:50:21.338561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8343,"log_dups_size":0,"ondisk_log_size":8343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37352694,"num_objects":4560,"num_object_clones":0,"num_object_copies":18240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8341,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8189","reported_seq":8275,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:22.482619+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:22.482619+0000","last_peered":"2024-09-18T11:50:22.482619+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:22.482619+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8189,"log_dups_size":0,"ondisk_log_size":8189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37008630,"num_objects":4518,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8187,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2515}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":600647052,"num_objects":73327,"num_object_clones":0,"num_object_copies":293308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133760,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4969,"num_bytes_recovered":38191104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1189568512,"data_stored":1189568512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133787,"ondisk_log_size":133787,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600468,"kb_used_data":294284,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93771372,"statfs":{"total":96636764160,"available":96021884928,"internally_reserved":0,"allocated":301346816,"data_stored":301171350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,5,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.3799999999999999},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":5.8970000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":5.492},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":2.1869999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":6.1699999999999999},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.84299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591100,"kb_used_data":293736,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93780740,"statfs":{"total":96636764160,"available":96031477760,"internally_reserved":0,"allocated":300785664,"data_stored":300610198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,1,0,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":8.8699999999999992},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":8.9039999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":9.2200000000000006},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":8.9570000000000007}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.9339999999999993},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.1899999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584724,"kb_used_data":282764,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93787116,"statfs":{"total":96636764160,"available":96038006784,"internally_reserved":0,"allocated":289550336,"data_stored":289374870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.5070000000000001},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":11.741}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.17299999999999999},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.0149999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.20799999999999999},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.5499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607668,"kb_used_data":293228,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93764172,"statfs":{"total":96636764160,"available":96014512128,"internally_reserved":0,"allocated":300265472,"data_stored":300090006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,3,1,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.4589999999999996},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":2.0049999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":4.7320000000000002},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":4.4160000000000004},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":5.4450000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":299745280,"data_stored":299745280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289239040,"data_stored":289239040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299966464,"data_stored":299966464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300617728,"data_stored":300617728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:28.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:29.111 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:29.112 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:29.148 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-09-18T11:50:27.977729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592594316,"num_objects":72344,"num_object_clones":0,"num_object_copies":289376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134743,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5099,"num_bytes_recovered":39215104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134770,"ondisk_log_size":134770,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2372012,"kb_used_data":1152048,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375115348,"statfs":{"total":386547056640,"available":384118116352,"internally_reserved":0,"allocated":1179697152,"data_stored":1178995288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,5,7,8,3,0,0,0,0,0,0,6,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26222592,"num_objects":-3201,"num_object_clones":0,"num_object_copies":-12804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3201,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":2056192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004472"},"pg_stats":[{"pgid":"1.f","version":"35'8616","reported_seq":8709,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.213539+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:26.213539+0000","last_peered":"2024-09-18T11:50:26.213539+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:26.213539+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8616,"log_dups_size":0,"ondisk_log_size":8616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37961728,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8616,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8492","reported_seq":12782,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:26.454443+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:26.454443+0000","last_peered":"2024-09-18T11:50:26.454443+0000","last_clean":"2024-09-18T11:50:26.454443+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:26.454443+0000","last_undegraded":"2024-09-18T11:50:26.454443+0000","last_fullsized":"2024-09-18T11:50:26.454443+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8492,"log_dups_size":0,"ondisk_log_size":8492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37385462,"num_objects":4564,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8324","reported_seq":8439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.054558+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:27.054558+0000","last_peered":"2024-09-18T11:50:27.054558+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:27.054558+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36731539,"num_objects":4485,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8323,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2445}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8448","reported_seq":8538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.451618+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:27.451618+0000","last_peered":"2024-09-18T11:50:27.451618+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:27.451618+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8448,"log_dups_size":0,"ondisk_log_size":8448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36680950,"num_objects":4478,"num_object_clones":0,"num_object_copies":17912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8446,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8398","reported_seq":8509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.276430+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:26.276430+0000","last_peered":"2024-09-18T11:50:26.276430+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:26.276430+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8398,"log_dups_size":0,"ondisk_log_size":8398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36836598,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8396,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2441}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8391","reported_seq":8506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.379549+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:26.379549+0000","last_peered":"2024-09-18T11:50:26.379549+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:26.379549+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36003840,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8504","reported_seq":8609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.492625+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:27.492625+0000","last_peered":"2024-09-18T11:50:27.492625+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:27.492625+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8504,"log_dups_size":0,"ondisk_log_size":8504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36959478,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8502,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8360","reported_seq":8465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.493167+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:27.493167+0000","last_peered":"2024-09-18T11:50:27.493167+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:27.493167+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8360,"log_dups_size":0,"ondisk_log_size":8360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36984054,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8358,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8591","reported_seq":9886,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.472489+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:26.472489+0000","last_peered":"2024-09-18T11:50:26.472489+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:26.472489+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37554668,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":4866048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3063}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8153","reported_seq":8265,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.189034+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:27.189034+0000","last_peered":"2024-09-18T11:50:27.189034+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:27.189034+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377846,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8425","reported_seq":8542,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.380606+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:26.380606+0000","last_peered":"2024-09-18T11:50:26.380606+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:26.380606+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37744497,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8422,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8474","reported_seq":8559,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.401680+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:26.401680+0000","last_peered":"2024-09-18T11:50:26.401680+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:26.401680+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37329531,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2470}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8391","reported_seq":8477,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.036131+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:27.036131+0000","last_peered":"2024-09-18T11:50:27.036131+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:27.036131+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37485179,"num_objects":4576,"num_object_clones":0,"num_object_copies":18304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8390,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2527}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8551","reported_seq":8670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.297428+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:26.297428+0000","last_peered":"2024-09-18T11:50:26.297428+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:26.297428+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37180662,"num_objects":4539,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8549,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2414}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8394","reported_seq":12681,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:26.482907+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:26.482907+0000","last_peered":"2024-09-18T11:50:26.482907+0000","last_clean":"2024-09-18T11:50:26.482907+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:26.482907+0000","last_undegraded":"2024-09-18T11:50:26.482907+0000","last_fullsized":"2024-09-18T11:50:26.482907+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934902,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8258","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.404386+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:27.404386+0000","last_peered":"2024-09-18T11:50:27.404386+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:27.404386+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36443382,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2447}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":592594316,"num_objects":72344,"num_object_clones":0,"num_object_copies":289376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134743,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5099,"num_bytes_recovered":39215104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1177264128,"data_stored":1177264128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134770,"ondisk_log_size":134770,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600748,"kb_used_data":290392,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93771092,"statfs":{"total":96636764160,"available":96021598208,"internally_reserved":0,"allocated":297361408,"data_stored":297185942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,2,2,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.5229999999999997},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.6060000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.73},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":6.4569999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":9.0619999999999994},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":9.0220000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591276,"kb_used_data":289764,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93780564,"statfs":{"total":96636764160,"available":96031297536,"internally_reserved":0,"allocated":296718336,"data_stored":296542870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,0,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.667},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.694}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.385},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.7190000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.5},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.35499999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588756,"kb_used_data":282636,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93783084,"statfs":{"total":96636764160,"available":96033878016,"internally_reserved":0,"allocated":289419264,"data_stored":289243798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.78900000000000003},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.1600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.82499999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.9509999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591232,"kb_used_data":289256,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93780608,"statfs":{"total":96636764160,"available":96031342592,"internally_reserved":0,"allocated":296198144,"data_stored":296022678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.3239999999999998},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.38200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.389},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.59299999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.435},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295653376,"data_stored":295653376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289083392,"data_stored":289083392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295899136,"data_stored":295899136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":296628224,"data_stored":296628224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:29.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:29.698 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:29.699 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:29.724 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-09-18T11:50:27.977729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592594316,"num_objects":72344,"num_object_clones":0,"num_object_copies":289376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134743,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5099,"num_bytes_recovered":39215104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134770,"ondisk_log_size":134770,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2372012,"kb_used_data":1152048,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375115348,"statfs":{"total":386547056640,"available":384118116352,"internally_reserved":0,"allocated":1179697152,"data_stored":1178995288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,5,7,8,3,0,0,0,0,0,0,6,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26222592,"num_objects":-3201,"num_object_clones":0,"num_object_copies":-12804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3201,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":2056192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004472"},"pg_stats":[{"pgid":"1.f","version":"35'8616","reported_seq":8709,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.213539+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:26.213539+0000","last_peered":"2024-09-18T11:50:26.213539+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:26.213539+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8616,"log_dups_size":0,"ondisk_log_size":8616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37961728,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8616,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8492","reported_seq":12782,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:26.454443+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:26.454443+0000","last_peered":"2024-09-18T11:50:26.454443+0000","last_clean":"2024-09-18T11:50:26.454443+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:26.454443+0000","last_undegraded":"2024-09-18T11:50:26.454443+0000","last_fullsized":"2024-09-18T11:50:26.454443+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8492,"log_dups_size":0,"ondisk_log_size":8492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37385462,"num_objects":4564,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8324","reported_seq":8439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.054558+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:27.054558+0000","last_peered":"2024-09-18T11:50:27.054558+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:27.054558+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36731539,"num_objects":4485,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8323,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2445}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8448","reported_seq":8538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.451618+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:27.451618+0000","last_peered":"2024-09-18T11:50:27.451618+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:27.451618+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8448,"log_dups_size":0,"ondisk_log_size":8448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36680950,"num_objects":4478,"num_object_clones":0,"num_object_copies":17912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8446,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8398","reported_seq":8509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.276430+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:26.276430+0000","last_peered":"2024-09-18T11:50:26.276430+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:26.276430+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8398,"log_dups_size":0,"ondisk_log_size":8398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36836598,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8396,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2441}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8391","reported_seq":8506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.379549+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:26.379549+0000","last_peered":"2024-09-18T11:50:26.379549+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:26.379549+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36003840,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8504","reported_seq":8609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.492625+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:27.492625+0000","last_peered":"2024-09-18T11:50:27.492625+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:27.492625+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8504,"log_dups_size":0,"ondisk_log_size":8504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36959478,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8502,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8360","reported_seq":8465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.493167+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:27.493167+0000","last_peered":"2024-09-18T11:50:27.493167+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:27.493167+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8360,"log_dups_size":0,"ondisk_log_size":8360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36984054,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8358,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8591","reported_seq":9886,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.472489+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:26.472489+0000","last_peered":"2024-09-18T11:50:26.472489+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:26.472489+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37554668,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":4866048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3063}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8153","reported_seq":8265,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.189034+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:27.189034+0000","last_peered":"2024-09-18T11:50:27.189034+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:27.189034+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377846,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8425","reported_seq":8542,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.380606+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:26.380606+0000","last_peered":"2024-09-18T11:50:26.380606+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:26.380606+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37744497,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8422,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8474","reported_seq":8559,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.401680+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:26.401680+0000","last_peered":"2024-09-18T11:50:26.401680+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:26.401680+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37329531,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2470}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8391","reported_seq":8477,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.036131+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:27.036131+0000","last_peered":"2024-09-18T11:50:27.036131+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:27.036131+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37485179,"num_objects":4576,"num_object_clones":0,"num_object_copies":18304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8390,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2527}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8551","reported_seq":8670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.297428+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:26.297428+0000","last_peered":"2024-09-18T11:50:26.297428+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:26.297428+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37180662,"num_objects":4539,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8549,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2414}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8394","reported_seq":12681,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:26.482907+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:26.482907+0000","last_peered":"2024-09-18T11:50:26.482907+0000","last_clean":"2024-09-18T11:50:26.482907+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:26.482907+0000","last_undegraded":"2024-09-18T11:50:26.482907+0000","last_fullsized":"2024-09-18T11:50:26.482907+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934902,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8258","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.404386+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:27.404386+0000","last_peered":"2024-09-18T11:50:27.404386+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:27.404386+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36443382,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2447}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":592594316,"num_objects":72344,"num_object_clones":0,"num_object_copies":289376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134743,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5099,"num_bytes_recovered":39215104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1177264128,"data_stored":1177264128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134770,"ondisk_log_size":134770,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600748,"kb_used_data":290392,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93771092,"statfs":{"total":96636764160,"available":96021598208,"internally_reserved":0,"allocated":297361408,"data_stored":297185942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,2,2,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.5229999999999997},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.6060000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.73},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":6.4569999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":9.0619999999999994},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":9.0220000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591276,"kb_used_data":289764,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93780564,"statfs":{"total":96636764160,"available":96031297536,"internally_reserved":0,"allocated":296718336,"data_stored":296542870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,0,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.667},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.694}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.385},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.7190000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.5},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.35499999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588756,"kb_used_data":282636,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93783084,"statfs":{"total":96636764160,"available":96033878016,"internally_reserved":0,"allocated":289419264,"data_stored":289243798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.78900000000000003},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.1600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.82499999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.9509999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591232,"kb_used_data":289256,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93780608,"statfs":{"total":96636764160,"available":96031342592,"internally_reserved":0,"allocated":296198144,"data_stored":296022678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.3239999999999998},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.38200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.389},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.59299999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.435},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295653376,"data_stored":295653376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289083392,"data_stored":289083392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295899136,"data_stored":295899136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":296628224,"data_stored":296628224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:29.725 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:29.725 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8616", 'reported_seq': 8709, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:26.213539+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:26.213539+0000', 'last_peered': '2024-09-18T11:50:26.213539+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:26.213539+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8616, 'log_dups_size': 0, 'ondisk_log_size': 8616, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37961728, 'num_objects': 4634, 'num_object_clones': 0, 'num_object_copies': 18536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8616, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2528}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:29.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:29.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8324", 'reported_seq': 8439, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:27.054558+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:27.054558+0000', 'last_peered': '2024-09-18T11:50:27.054558+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:27.054558+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8324, 'log_dups_size': 0, 'ondisk_log_size': 8324, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36731539, 'num_objects': 4485, 'num_object_clones': 0, 'num_object_copies': 17940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4485, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8323, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2445}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:29.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:29.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8448", 'reported_seq': 8538, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:27.451618+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:27.451618+0000', 'last_peered': '2024-09-18T11:50:27.451618+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:27.451618+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8448, 'log_dups_size': 0, 'ondisk_log_size': 8448, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36680950, 'num_objects': 4478, 'num_object_clones': 0, 'num_object_copies': 17912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8446, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2376}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:29.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:29.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8398", 'reported_seq': 8509, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:26.276430+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:26.276430+0000', 'last_peered': '2024-09-18T11:50:26.276430+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:26.276430+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8398, 'log_dups_size': 0, 'ondisk_log_size': 8398, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36836598, 'num_objects': 4497, 'num_object_clones': 0, 'num_object_copies': 17988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8396, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2441}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:29.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:29.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8391", 'reported_seq': 8506, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:26.379549+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:26.379549+0000', 'last_peered': '2024-09-18T11:50:26.379549+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:26.379549+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8391, 'log_dups_size': 0, 'ondisk_log_size': 8391, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36003840, 'num_objects': 4395, 'num_object_clones': 0, 'num_object_copies': 17580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8391, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2275}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:29.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:29.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8504", 'reported_seq': 8609, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:27.492625+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:27.492625+0000', 'last_peered': '2024-09-18T11:50:27.492625+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:27.492625+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8504, 'log_dups_size': 0, 'ondisk_log_size': 8504, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36959478, 'num_objects': 4512, 'num_object_clones': 0, 'num_object_copies': 18048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8502, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2396}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:29.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:29.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8360", 'reported_seq': 8465, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:27.493167+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:27.493167+0000', 'last_peered': '2024-09-18T11:50:27.493167+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:27.493167+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8360, 'log_dups_size': 0, 'ondisk_log_size': 8360, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36984054, 'num_objects': 4515, 'num_object_clones': 0, 'num_object_copies': 18060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8358, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:29.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:29.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8591", 'reported_seq': 9886, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:26.472489+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:26.472489+0000', 'last_peered': '2024-09-18T11:50:26.472489+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:26.472489+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8591, 'log_dups_size': 0, 'ondisk_log_size': 8591, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37554668, 'num_objects': 4585, 'num_object_clones': 0, 'num_object_copies': 18340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1522, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8588, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 619, 'num_bytes_recovered': 4866048, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3063}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:29.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:29.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8153", 'reported_seq': 8265, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:27.189034+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:27.189034+0000', 'last_peered': '2024-09-18T11:50:27.189034+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:27.189034+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8153, 'log_dups_size': 0, 'ondisk_log_size': 8153, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36377846, 'num_objects': 4441, 'num_object_clones': 0, 'num_object_copies': 17764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8151, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2478}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:29.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:29.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8425", 'reported_seq': 8542, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:26.380606+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:26.380606+0000', 'last_peered': '2024-09-18T11:50:26.380606+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:26.380606+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8425, 'log_dups_size': 0, 'ondisk_log_size': 8425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37744497, 'num_objects': 4608, 'num_object_clones': 0, 'num_object_copies': 18432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8422, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2573}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:29.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:29.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8474", 'reported_seq': 8559, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:26.401680+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:26.401680+0000', 'last_peered': '2024-09-18T11:50:26.401680+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:26.401680+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8474, 'log_dups_size': 0, 'ondisk_log_size': 8474, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37329531, 'num_objects': 4557, 'num_object_clones': 0, 'num_object_copies': 18228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2088, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8473, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2470}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:29.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:29.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8391", 'reported_seq': 8477, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:27.036131+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:27.036131+0000', 'last_peered': '2024-09-18T11:50:27.036131+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:27.036131+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8391, 'log_dups_size': 0, 'ondisk_log_size': 8391, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37485179, 'num_objects': 4576, 'num_object_clones': 0, 'num_object_copies': 18304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8390, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:29.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:29.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8551", 'reported_seq': 8670, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:26.297428+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:26.297428+0000', 'last_peered': '2024-09-18T11:50:26.297428+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:26.297428+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8551, 'log_dups_size': 0, 'ondisk_log_size': 8551, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37180662, 'num_objects': 4539, 'num_object_clones': 0, 'num_object_copies': 18156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2127, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8549, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2414}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:29.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:29.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8258", 'reported_seq': 8346, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:27.404386+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:27.404386+0000', 'last_peered': '2024-09-18T11:50:27.404386+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:27.404386+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8258, 'log_dups_size': 0, 'ondisk_log_size': 8258, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36443382, 'num_objects': 4449, 'num_object_clones': 0, 'num_object_copies': 17796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8256, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2447}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:29.728 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:30.403 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:30.404 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28357/289376 objects degraded (9.799%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":428,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72344,"data_bytes":592594316,"bytes_used":2428940288,"bytes_avail":384118116352,"bytes_total":386547056640,"degraded_objects":28357,"degraded_total":289376,"degraded_ratio":0.097993613844962954,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":135118,"recovering_keys_per_sec":0,"num_objects_recovered":211,"num_bytes_recovered":1622016,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":169},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:30.434 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:30.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:30.997 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:30.998 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:31.028 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-18T11:50:29.978467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592594316,"num_objects":72344,"num_object_clones":0,"num_object_copies":289376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134743,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5099,"num_bytes_recovered":39215104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134770,"ondisk_log_size":134770,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2372012,"kb_used_data":1152048,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375115348,"statfs":{"total":386547056640,"available":384118116352,"internally_reserved":0,"allocated":1179697152,"data_stored":1178995288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,5,7,8,3,0,0,0,0,0,0,6,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16678912,"num_objects":-2036,"num_object_clones":0,"num_object_copies":-8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2036,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004378"},"pg_stats":[{"pgid":"1.f","version":"35'8616","reported_seq":8709,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.213539+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:26.213539+0000","last_peered":"2024-09-18T11:50:26.213539+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:26.213539+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8616,"log_dups_size":0,"ondisk_log_size":8616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37961728,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8616,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8492","reported_seq":12782,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:26.454443+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:26.454443+0000","last_peered":"2024-09-18T11:50:26.454443+0000","last_clean":"2024-09-18T11:50:26.454443+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:26.454443+0000","last_undegraded":"2024-09-18T11:50:26.454443+0000","last_fullsized":"2024-09-18T11:50:26.454443+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8492,"log_dups_size":0,"ondisk_log_size":8492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37385462,"num_objects":4564,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8324","reported_seq":8439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.054558+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:27.054558+0000","last_peered":"2024-09-18T11:50:27.054558+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:27.054558+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36731539,"num_objects":4485,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8323,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2445}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8448","reported_seq":8538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.451618+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:27.451618+0000","last_peered":"2024-09-18T11:50:27.451618+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:27.451618+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8448,"log_dups_size":0,"ondisk_log_size":8448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36680950,"num_objects":4478,"num_object_clones":0,"num_object_copies":17912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8446,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8398","reported_seq":8509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.276430+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:26.276430+0000","last_peered":"2024-09-18T11:50:26.276430+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:26.276430+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8398,"log_dups_size":0,"ondisk_log_size":8398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36836598,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8396,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2441}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8391","reported_seq":8506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.379549+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:26.379549+0000","last_peered":"2024-09-18T11:50:26.379549+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:26.379549+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36003840,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8504","reported_seq":8609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.492625+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:27.492625+0000","last_peered":"2024-09-18T11:50:27.492625+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:27.492625+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8504,"log_dups_size":0,"ondisk_log_size":8504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36959478,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8502,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8360","reported_seq":8465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.493167+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:27.493167+0000","last_peered":"2024-09-18T11:50:27.493167+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:27.493167+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8360,"log_dups_size":0,"ondisk_log_size":8360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36984054,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8358,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8591","reported_seq":9886,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.472489+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:26.472489+0000","last_peered":"2024-09-18T11:50:26.472489+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:26.472489+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37554668,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":4866048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3063}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8153","reported_seq":8265,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.189034+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:27.189034+0000","last_peered":"2024-09-18T11:50:27.189034+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:27.189034+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377846,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8425","reported_seq":8542,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.380606+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:26.380606+0000","last_peered":"2024-09-18T11:50:26.380606+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:26.380606+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37744497,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8422,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8474","reported_seq":8559,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.401680+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:26.401680+0000","last_peered":"2024-09-18T11:50:26.401680+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:26.401680+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37329531,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2470}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8391","reported_seq":8477,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.036131+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:27.036131+0000","last_peered":"2024-09-18T11:50:27.036131+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:27.036131+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37485179,"num_objects":4576,"num_object_clones":0,"num_object_copies":18304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8390,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2527}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8551","reported_seq":8670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:26.297428+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:26.297428+0000","last_peered":"2024-09-18T11:50:26.297428+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:26.297428+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37180662,"num_objects":4539,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8549,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2414}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8394","reported_seq":12681,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:26.482907+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:26.482907+0000","last_peered":"2024-09-18T11:50:26.482907+0000","last_clean":"2024-09-18T11:50:26.482907+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:26.482907+0000","last_undegraded":"2024-09-18T11:50:26.482907+0000","last_fullsized":"2024-09-18T11:50:26.482907+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934902,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8258","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:27.404386+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:27.404386+0000","last_peered":"2024-09-18T11:50:27.404386+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:27.404386+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36443382,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2447}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":592594316,"num_objects":72344,"num_object_clones":0,"num_object_copies":289376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134743,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5099,"num_bytes_recovered":39215104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1177264128,"data_stored":1177264128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134770,"ondisk_log_size":134770,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600748,"kb_used_data":290392,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93771092,"statfs":{"total":96636764160,"available":96021598208,"internally_reserved":0,"allocated":297361408,"data_stored":297185942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,2,2,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.5229999999999997},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.6060000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":1.73},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":6.4569999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":9.0619999999999994},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":9.0220000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591276,"kb_used_data":289764,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93780564,"statfs":{"total":96636764160,"available":96031297536,"internally_reserved":0,"allocated":296718336,"data_stored":296542870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,0,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.667},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.694}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.385},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.7190000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.5},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.35499999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588756,"kb_used_data":282636,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93783084,"statfs":{"total":96636764160,"available":96033878016,"internally_reserved":0,"allocated":289419264,"data_stored":289243798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.78900000000000003},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.1600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.82499999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.9509999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591232,"kb_used_data":289256,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93780608,"statfs":{"total":96636764160,"available":96031342592,"internally_reserved":0,"allocated":296198144,"data_stored":296022678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":4.3239999999999998},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.38200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.389},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.59299999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.435},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295653376,"data_stored":295653376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":289083392,"data_stored":289083392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295899136,"data_stored":295899136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":296628224,"data_stored":296628224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:34.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:34.591 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-18T11:50:33.979440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583910796,"num_objects":71284,"num_object_clones":0,"num_object_copies":285136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135803,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5170,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135830,"ondisk_log_size":135830,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2362848,"kb_used_data":1138748,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375124512,"statfs":{"total":386547056640,"available":384127500288,"internally_reserved":0,"allocated":1166077952,"data_stored":1165376088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,6,6,10,6,0,0,0,0,0,1,0,9],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20365312,"num_objects":-2486,"num_object_clones":0,"num_object_copies":-9944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003981"},"pg_stats":[{"pgid":"1.f","version":"35'8680","reported_seq":8773,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.416651+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:31.416651+0000","last_peered":"2024-09-18T11:50:31.416651+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:31.416651+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37437440,"num_objects":4570,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8680,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8556","reported_seq":12846,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:31.468726+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:31.468726+0000","last_peered":"2024-09-18T11:50:31.468726+0000","last_clean":"2024-09-18T11:50:31.468726+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:31.468726+0000","last_undegraded":"2024-09-18T11:50:31.468726+0000","last_fullsized":"2024-09-18T11:50:31.468726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36861174,"num_objects":4500,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8395","reported_seq":8510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.117290+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:32.117290+0000","last_peered":"2024-09-18T11:50:32.117290+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:32.117290+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8395,"log_dups_size":0,"ondisk_log_size":8395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36149907,"num_objects":4414,"num_object_clones":0,"num_object_copies":17656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4414,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8394,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2374}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8508","reported_seq":8598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.472976+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:32.472976+0000","last_peered":"2024-09-18T11:50:32.472976+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:32.472976+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8508,"log_dups_size":0,"ondisk_log_size":8508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36189430,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8506,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2316}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8457","reported_seq":8568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.373068+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:31.373068+0000","last_peered":"2024-09-18T11:50:31.373068+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:31.373068+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36353270,"num_objects":4438,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2382}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8465","reported_seq":8580,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.302173+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:31.302173+0000","last_peered":"2024-09-18T11:50:31.302173+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:31.302173+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8465,"log_dups_size":0,"ondisk_log_size":8465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35397632,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8586","reported_seq":8691,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.488128+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:32.488128+0000","last_peered":"2024-09-18T11:50:32.488128+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:32.488128+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8586,"log_dups_size":0,"ondisk_log_size":8586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36287734,"num_objects":4430,"num_object_clones":0,"num_object_copies":17720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8584,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2314}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8425","reported_seq":8532,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.514118+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:32.514118+0000","last_peered":"2024-09-18T11:50:32.514118+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:32.514118+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36451574,"num_objects":4450,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8664","reported_seq":10101,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.439501+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:31.439501+0000","last_peered":"2024-09-18T11:50:31.439501+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:31.439501+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36956652,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8207","reported_seq":8319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.173740+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:32.173740+0000","last_peered":"2024-09-18T11:50:32.173740+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:32.173740+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8207,"log_dups_size":0,"ondisk_log_size":8207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935478,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2424}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8504","reported_seq":8621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.397230+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:31.397230+0000","last_peered":"2024-09-18T11:50:31.397230+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:31.397230+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8504,"log_dups_size":0,"ondisk_log_size":8504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37097329,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8501,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2494}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8536","reported_seq":8621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.245623+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:31.245623+0000","last_peered":"2024-09-18T11:50:31.245623+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:31.245623+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36821627,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8535,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2408}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8456","reported_seq":8546,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.195781+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:32.195781+0000","last_peered":"2024-09-18T11:50:32.195781+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:32.195781+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36952699,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8629","reported_seq":8748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.260034+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:31.260034+0000","last_peered":"2024-09-18T11:50:31.260034+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:31.260034+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8629,"log_dups_size":0,"ondisk_log_size":8629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541686,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8627,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2336}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8457","reported_seq":12744,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:31.122619+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:31.122619+0000","last_peered":"2024-09-18T11:50:31.122619+0000","last_clean":"2024-09-18T11:50:31.122619+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:31.122619+0000","last_undegraded":"2024-09-18T11:50:31.122619+0000","last_fullsized":"2024-09-18T11:50:31.122619+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36418806,"num_objects":4446,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8305","reported_seq":8393,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.379546+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:32.379546+0000","last_peered":"2024-09-18T11:50:32.379546+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:32.379546+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8305,"log_dups_size":0,"ondisk_log_size":8305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36058358,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2400}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583910796,"num_objects":71284,"num_object_clones":0,"num_object_copies":285136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135803,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5170,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1163616256,"data_stored":1163616256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135830,"ondisk_log_size":135830,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584028,"kb_used_data":286164,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93787812,"statfs":{"total":96636764160,"available":96038719488,"internally_reserved":0,"allocated":293031936,"data_stored":292856470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,1,4,1,0,0,0,0,0,0,0,6],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":4.5270000000000001},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.4649999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":4.4950000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.169}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":8.2219999999999995},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.1970000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595404,"kb_used_data":285572,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93776436,"statfs":{"total":96636764160,"available":96027070464,"internally_reserved":0,"allocated":292425728,"data_stored":292250262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,0,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.4430000000000001},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.466}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.415}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.307},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.1530000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592300,"kb_used_data":282020,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93779540,"statfs":{"total":96636764160,"available":96030248960,"internally_reserved":0,"allocated":288788480,"data_stored":288613014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.0990000000000002},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":3.7679999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":4.9749999999999996},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.0489999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":3.919},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.0919999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591116,"kb_used_data":284992,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93780724,"statfs":{"total":96636764160,"available":96031461376,"internally_reserved":0,"allocated":291831808,"data_stored":291656342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,0,2,5,4,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":1.1539999999999999},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.2830000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.23899999999999999},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.3239999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.577},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":4.3540000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291311616,"data_stored":291311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288456704,"data_stored":288456704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":291581952,"data_stored":291581952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292265984,"data_stored":292265984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:34.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:35.136 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:35.167 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-18T11:50:33.979440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583910796,"num_objects":71284,"num_object_clones":0,"num_object_copies":285136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135803,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5170,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135830,"ondisk_log_size":135830,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2362848,"kb_used_data":1138748,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375124512,"statfs":{"total":386547056640,"available":384127500288,"internally_reserved":0,"allocated":1166077952,"data_stored":1165376088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,6,6,10,6,0,0,0,0,0,1,0,9],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20365312,"num_objects":-2486,"num_object_clones":0,"num_object_copies":-9944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003981"},"pg_stats":[{"pgid":"1.f","version":"35'8680","reported_seq":8773,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.416651+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:31.416651+0000","last_peered":"2024-09-18T11:50:31.416651+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:31.416651+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37437440,"num_objects":4570,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8680,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8556","reported_seq":12846,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:31.468726+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:31.468726+0000","last_peered":"2024-09-18T11:50:31.468726+0000","last_clean":"2024-09-18T11:50:31.468726+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:31.468726+0000","last_undegraded":"2024-09-18T11:50:31.468726+0000","last_fullsized":"2024-09-18T11:50:31.468726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36861174,"num_objects":4500,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8395","reported_seq":8510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.117290+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:32.117290+0000","last_peered":"2024-09-18T11:50:32.117290+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:32.117290+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8395,"log_dups_size":0,"ondisk_log_size":8395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36149907,"num_objects":4414,"num_object_clones":0,"num_object_copies":17656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4414,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8394,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2374}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8508","reported_seq":8598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.472976+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:32.472976+0000","last_peered":"2024-09-18T11:50:32.472976+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:32.472976+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8508,"log_dups_size":0,"ondisk_log_size":8508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36189430,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8506,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2316}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8457","reported_seq":8568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.373068+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:31.373068+0000","last_peered":"2024-09-18T11:50:31.373068+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:31.373068+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36353270,"num_objects":4438,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2382}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8465","reported_seq":8580,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.302173+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:31.302173+0000","last_peered":"2024-09-18T11:50:31.302173+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:31.302173+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8465,"log_dups_size":0,"ondisk_log_size":8465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35397632,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8586","reported_seq":8691,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.488128+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:32.488128+0000","last_peered":"2024-09-18T11:50:32.488128+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:32.488128+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8586,"log_dups_size":0,"ondisk_log_size":8586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36287734,"num_objects":4430,"num_object_clones":0,"num_object_copies":17720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8584,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2314}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8425","reported_seq":8532,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.514118+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:32.514118+0000","last_peered":"2024-09-18T11:50:32.514118+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:32.514118+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36451574,"num_objects":4450,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8664","reported_seq":10101,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.439501+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:31.439501+0000","last_peered":"2024-09-18T11:50:31.439501+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:31.439501+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36956652,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8207","reported_seq":8319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.173740+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:32.173740+0000","last_peered":"2024-09-18T11:50:32.173740+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:32.173740+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8207,"log_dups_size":0,"ondisk_log_size":8207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935478,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2424}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8504","reported_seq":8621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.397230+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:31.397230+0000","last_peered":"2024-09-18T11:50:31.397230+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:31.397230+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8504,"log_dups_size":0,"ondisk_log_size":8504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37097329,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8501,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2494}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8536","reported_seq":8621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.245623+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:31.245623+0000","last_peered":"2024-09-18T11:50:31.245623+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:31.245623+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36821627,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8535,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2408}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8456","reported_seq":8546,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.195781+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:32.195781+0000","last_peered":"2024-09-18T11:50:32.195781+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:32.195781+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36952699,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8629","reported_seq":8748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.260034+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:31.260034+0000","last_peered":"2024-09-18T11:50:31.260034+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:31.260034+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8629,"log_dups_size":0,"ondisk_log_size":8629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541686,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8627,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2336}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8457","reported_seq":12744,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:31.122619+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:31.122619+0000","last_peered":"2024-09-18T11:50:31.122619+0000","last_clean":"2024-09-18T11:50:31.122619+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:31.122619+0000","last_undegraded":"2024-09-18T11:50:31.122619+0000","last_fullsized":"2024-09-18T11:50:31.122619+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36418806,"num_objects":4446,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8305","reported_seq":8393,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.379546+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:32.379546+0000","last_peered":"2024-09-18T11:50:32.379546+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:32.379546+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8305,"log_dups_size":0,"ondisk_log_size":8305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36058358,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2400}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583910796,"num_objects":71284,"num_object_clones":0,"num_object_copies":285136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135803,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5170,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1163616256,"data_stored":1163616256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135830,"ondisk_log_size":135830,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584028,"kb_used_data":286164,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93787812,"statfs":{"total":96636764160,"available":96038719488,"internally_reserved":0,"allocated":293031936,"data_stored":292856470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,1,4,1,0,0,0,0,0,0,0,6],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":4.5270000000000001},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.4649999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":4.4950000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.169}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":8.2219999999999995},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.1970000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595404,"kb_used_data":285572,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93776436,"statfs":{"total":96636764160,"available":96027070464,"internally_reserved":0,"allocated":292425728,"data_stored":292250262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,0,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.4430000000000001},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.466}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.415}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.307},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.1530000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592300,"kb_used_data":282020,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93779540,"statfs":{"total":96636764160,"available":96030248960,"internally_reserved":0,"allocated":288788480,"data_stored":288613014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.0990000000000002},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":3.7679999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":4.9749999999999996},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.0489999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":3.919},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.0919999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591116,"kb_used_data":284992,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93780724,"statfs":{"total":96636764160,"available":96031461376,"internally_reserved":0,"allocated":291831808,"data_stored":291656342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,0,2,5,4,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":1.1539999999999999},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.2830000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.23899999999999999},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.3239999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.577},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":4.3540000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291311616,"data_stored":291311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288456704,"data_stored":288456704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":291581952,"data_stored":291581952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292265984,"data_stored":292265984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:35.168 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:35.168 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8680", 'reported_seq': 8773, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:31.416651+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:31.416651+0000', 'last_peered': '2024-09-18T11:50:31.416651+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:31.416651+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8680, 'log_dups_size': 0, 'ondisk_log_size': 8680, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37437440, 'num_objects': 4570, 'num_object_clones': 0, 'num_object_copies': 18280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8680, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2464}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:35.168 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:35.168 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8395", 'reported_seq': 8510, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:32.117290+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:32.117290+0000', 'last_peered': '2024-09-18T11:50:32.117290+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:32.117290+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8395, 'log_dups_size': 0, 'ondisk_log_size': 8395, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36149907, 'num_objects': 4414, 'num_object_clones': 0, 'num_object_copies': 17656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4414, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8394, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2374}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8508", 'reported_seq': 8598, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:32.472976+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:32.472976+0000', 'last_peered': '2024-09-18T11:50:32.472976+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:32.472976+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8508, 'log_dups_size': 0, 'ondisk_log_size': 8508, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36189430, 'num_objects': 4418, 'num_object_clones': 0, 'num_object_copies': 17672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8506, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2316}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8457", 'reported_seq': 8568, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:31.373068+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:31.373068+0000', 'last_peered': '2024-09-18T11:50:31.373068+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:31.373068+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8457, 'log_dups_size': 0, 'ondisk_log_size': 8457, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36353270, 'num_objects': 4438, 'num_object_clones': 0, 'num_object_copies': 17752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8455, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2382}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8465", 'reported_seq': 8580, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:31.302173+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:31.302173+0000', 'last_peered': '2024-09-18T11:50:31.302173+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:31.302173+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8465, 'log_dups_size': 0, 'ondisk_log_size': 8465, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35397632, 'num_objects': 4321, 'num_object_clones': 0, 'num_object_copies': 17284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8465, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2201}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8586", 'reported_seq': 8691, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:32.488128+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:32.488128+0000', 'last_peered': '2024-09-18T11:50:32.488128+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:32.488128+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8586, 'log_dups_size': 0, 'ondisk_log_size': 8586, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36287734, 'num_objects': 4430, 'num_object_clones': 0, 'num_object_copies': 17720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4430, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8584, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2314}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8425", 'reported_seq': 8532, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:32.514118+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:32.514118+0000', 'last_peered': '2024-09-18T11:50:32.514118+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:32.514118+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8425, 'log_dups_size': 0, 'ondisk_log_size': 8425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36451574, 'num_objects': 4450, 'num_object_clones': 0, 'num_object_copies': 17800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8423, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2436}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:35.169 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8664", 'reported_seq': 10101, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:31.439501+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:31.439501+0000', 'last_peered': '2024-09-18T11:50:31.439501+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:31.439501+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8664, 'log_dups_size': 0, 'ondisk_log_size': 8664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36956652, 'num_objects': 4512, 'num_object_clones': 0, 'num_object_copies': 18048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1451, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8661, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 690, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3061}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8207", 'reported_seq': 8319, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:32.173740+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:32.173740+0000', 'last_peered': '2024-09-18T11:50:32.173740+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:32.173740+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8207, 'log_dups_size': 0, 'ondisk_log_size': 8207, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35935478, 'num_objects': 4387, 'num_object_clones': 0, 'num_object_copies': 17548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8205, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2424}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8504", 'reported_seq': 8621, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:31.397230+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:31.397230+0000', 'last_peered': '2024-09-18T11:50:31.397230+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:31.397230+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8504, 'log_dups_size': 0, 'ondisk_log_size': 8504, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37097329, 'num_objects': 4529, 'num_object_clones': 0, 'num_object_copies': 18116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8501, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2494}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8536", 'reported_seq': 8621, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:31.245623+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:31.245623+0000', 'last_peered': '2024-09-18T11:50:31.245623+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:31.245623+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8536, 'log_dups_size': 0, 'ondisk_log_size': 8536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36821627, 'num_objects': 4495, 'num_object_clones': 0, 'num_object_copies': 17980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2088, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8535, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2408}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8456", 'reported_seq': 8546, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:32.195781+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:32.195781+0000', 'last_peered': '2024-09-18T11:50:32.195781+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:32.195781+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8456, 'log_dups_size': 0, 'ondisk_log_size': 8456, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36952699, 'num_objects': 4511, 'num_object_clones': 0, 'num_object_copies': 18044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8455, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:35.170 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8629", 'reported_seq': 8748, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:31.260034+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:31.260034+0000', 'last_peered': '2024-09-18T11:50:31.260034+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:31.260034+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8629, 'log_dups_size': 0, 'ondisk_log_size': 8629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36541686, 'num_objects': 4461, 'num_object_clones': 0, 'num_object_copies': 17844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2127, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8627, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2336}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:35.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:35.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8305", 'reported_seq': 8393, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:32.379546+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:32.379546+0000', 'last_peered': '2024-09-18T11:50:32.379546+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:32.379546+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8305, 'log_dups_size': 0, 'ondisk_log_size': 8305, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36058358, 'num_objects': 4402, 'num_object_clones': 0, 'num_object_copies': 17608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8303, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2400}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:35.171 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:35.822 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:35.822 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28286/285136 objects degraded (9.920%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":433,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71284,"data_bytes":583910796,"bytes_used":2419556352,"bytes_avail":384127500288,"bytes_total":386547056640,"degraded_objects":28286,"degraded_total":285136,"degraded_ratio":0.099201784411649174,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":117379,"recovering_keys_per_sec":0,"num_objects_recovered":204,"num_bytes_recovered":1409024,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:35.857 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:35.857 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:36.401 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:36.433 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-18T11:50:35.979929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583910796,"num_objects":71284,"num_object_clones":0,"num_object_copies":285136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135803,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5170,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135830,"ondisk_log_size":135830,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2362848,"kb_used_data":1138748,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375124512,"statfs":{"total":386547056640,"available":384127500288,"internally_reserved":0,"allocated":1166077952,"data_stored":1165376088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,6,6,10,6,0,0,0,0,0,1,0,9],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16736256,"num_objects":-2043,"num_object_clones":0,"num_object_copies":-8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2043,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003681"},"pg_stats":[{"pgid":"1.f","version":"35'8680","reported_seq":8773,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.416651+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:31.416651+0000","last_peered":"2024-09-18T11:50:31.416651+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:31.416651+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37437440,"num_objects":4570,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8680,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8556","reported_seq":12846,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:31.468726+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:31.468726+0000","last_peered":"2024-09-18T11:50:31.468726+0000","last_clean":"2024-09-18T11:50:31.468726+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:31.468726+0000","last_undegraded":"2024-09-18T11:50:31.468726+0000","last_fullsized":"2024-09-18T11:50:31.468726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36861174,"num_objects":4500,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8395","reported_seq":8510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.117290+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:32.117290+0000","last_peered":"2024-09-18T11:50:32.117290+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:32.117290+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8395,"log_dups_size":0,"ondisk_log_size":8395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36149907,"num_objects":4414,"num_object_clones":0,"num_object_copies":17656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4414,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8394,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2374}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8508","reported_seq":8598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.472976+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:32.472976+0000","last_peered":"2024-09-18T11:50:32.472976+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:32.472976+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8508,"log_dups_size":0,"ondisk_log_size":8508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36189430,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8506,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2316}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8457","reported_seq":8568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.373068+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:31.373068+0000","last_peered":"2024-09-18T11:50:31.373068+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:31.373068+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36353270,"num_objects":4438,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2382}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8465","reported_seq":8580,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.302173+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:31.302173+0000","last_peered":"2024-09-18T11:50:31.302173+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:31.302173+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8465,"log_dups_size":0,"ondisk_log_size":8465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35397632,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8586","reported_seq":8691,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.488128+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:32.488128+0000","last_peered":"2024-09-18T11:50:32.488128+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:32.488128+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8586,"log_dups_size":0,"ondisk_log_size":8586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36287734,"num_objects":4430,"num_object_clones":0,"num_object_copies":17720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8584,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2314}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8425","reported_seq":8532,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.514118+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:32.514118+0000","last_peered":"2024-09-18T11:50:32.514118+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:32.514118+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36451574,"num_objects":4450,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8664","reported_seq":10101,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.439501+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:31.439501+0000","last_peered":"2024-09-18T11:50:31.439501+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:31.439501+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36956652,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8207","reported_seq":8319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.173740+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:32.173740+0000","last_peered":"2024-09-18T11:50:32.173740+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:32.173740+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8207,"log_dups_size":0,"ondisk_log_size":8207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935478,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2424}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8504","reported_seq":8621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.397230+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:31.397230+0000","last_peered":"2024-09-18T11:50:31.397230+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:31.397230+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8504,"log_dups_size":0,"ondisk_log_size":8504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37097329,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8501,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2494}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8536","reported_seq":8621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.245623+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:31.245623+0000","last_peered":"2024-09-18T11:50:31.245623+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:31.245623+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36821627,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8535,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2408}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8456","reported_seq":8546,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.195781+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:32.195781+0000","last_peered":"2024-09-18T11:50:32.195781+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:32.195781+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36952699,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8629","reported_seq":8748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:31.260034+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:31.260034+0000","last_peered":"2024-09-18T11:50:31.260034+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:31.260034+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8629,"log_dups_size":0,"ondisk_log_size":8629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541686,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8627,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2336}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8457","reported_seq":12744,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:31.122619+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:31.122619+0000","last_peered":"2024-09-18T11:50:31.122619+0000","last_clean":"2024-09-18T11:50:31.122619+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:31.122619+0000","last_undegraded":"2024-09-18T11:50:31.122619+0000","last_fullsized":"2024-09-18T11:50:31.122619+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36418806,"num_objects":4446,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8305","reported_seq":8393,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:32.379546+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:32.379546+0000","last_peered":"2024-09-18T11:50:32.379546+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:32.379546+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8305,"log_dups_size":0,"ondisk_log_size":8305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36058358,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2400}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583910796,"num_objects":71284,"num_object_clones":0,"num_object_copies":285136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135803,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5170,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1163616256,"data_stored":1163616256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135830,"ondisk_log_size":135830,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584028,"kb_used_data":286164,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93787812,"statfs":{"total":96636764160,"available":96038719488,"internally_reserved":0,"allocated":293031936,"data_stored":292856470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,1,4,1,0,0,0,0,0,0,0,6],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":4.5270000000000001},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.4649999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":4.4950000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.169}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":8.2219999999999995},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.1970000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595404,"kb_used_data":285572,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93776436,"statfs":{"total":96636764160,"available":96027070464,"internally_reserved":0,"allocated":292425728,"data_stored":292250262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,0,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.4430000000000001},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":1.466}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.415}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.307},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.1530000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592300,"kb_used_data":282020,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93779540,"statfs":{"total":96636764160,"available":96030248960,"internally_reserved":0,"allocated":288788480,"data_stored":288613014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.0990000000000002},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":3.7679999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":4.9749999999999996},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.0489999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":3.919},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.0919999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591116,"kb_used_data":284992,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93780724,"statfs":{"total":96636764160,"available":96031461376,"internally_reserved":0,"allocated":291831808,"data_stored":291656342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,0,2,5,4,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":1.1539999999999999},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.2830000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.23899999999999999},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.3239999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.577},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":4.3540000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291311616,"data_stored":291311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288456704,"data_stored":288456704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":291581952,"data_stored":291581952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292265984,"data_stored":292265984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:39.436 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:40.015 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:40.016 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:40.039 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-18T11:50:39.980929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":575137164,"num_objects":70213,"num_object_clones":0,"num_object_copies":280852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136874,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5267,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136901,"ondisk_log_size":136901,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2332696,"kb_used_data":1125216,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375154664,"statfs":{"total":386547056640,"available":384158375936,"internally_reserved":0,"allocated":1152221184,"data_stored":1151519320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,9,7,1,0,0,0,0,0,0,0,7,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17457152,"num_objects":-2131,"num_object_clones":0,"num_object_copies":-8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":385024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003199"},"pg_stats":[{"pgid":"1.f","version":"35'8740","reported_seq":8835,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.321145+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:36.321145+0000","last_peered":"2024-09-18T11:50:36.321145+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:36.321145+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36945920,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8740,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2405}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8620","reported_seq":12910,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:36.445883+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:36.445883+0000","last_peered":"2024-09-18T11:50:36.445883+0000","last_clean":"2024-09-18T11:50:36.445883+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:36.445883+0000","last_undegraded":"2024-09-18T11:50:36.445883+0000","last_fullsized":"2024-09-18T11:50:36.445883+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8620,"log_dups_size":0,"ondisk_log_size":8620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36336886,"num_objects":4436,"num_object_clones":0,"num_object_copies":17744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8460","reported_seq":8575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.124838+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:37.124838+0000","last_peered":"2024-09-18T11:50:37.124838+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:37.124838+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8460,"log_dups_size":0,"ondisk_log_size":8460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35617427,"num_objects":4349,"num_object_clones":0,"num_object_copies":17396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8459,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2309}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8583","reported_seq":8673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.536237+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:37.536237+0000","last_peered":"2024-09-18T11:50:37.536237+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:37.536237+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8583,"log_dups_size":0,"ondisk_log_size":8583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35575030,"num_objects":4343,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8516","reported_seq":8631,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.344642+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:36.344642+0000","last_peered":"2024-09-18T11:50:36.344642+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:36.344642+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35869942,"num_objects":4379,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8541","reported_seq":8658,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.356251+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:36.356251+0000","last_peered":"2024-09-18T11:50:36.356251+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:36.356251+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34775040,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8541,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2126}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8649","reported_seq":8754,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.444683+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:37.444683+0000","last_peered":"2024-09-18T11:50:37.444683+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:37.444683+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35771638,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8647,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2251}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8480","reported_seq":8587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.487274+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:37.487274+0000","last_peered":"2024-09-18T11:50:37.487274+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:37.487274+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36001014,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8718","reported_seq":10336,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.490943+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:36.490943+0000","last_peered":"2024-09-18T11:50:36.490943+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:36.490943+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8718,"log_dups_size":0,"ondisk_log_size":8718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36514284,"num_objects":4458,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":781,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8273","reported_seq":8385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.124256+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:37.124256+0000","last_peered":"2024-09-18T11:50:37.124256+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:37.124256+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394806,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8271,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2358}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8573","reported_seq":8690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.443753+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:36.443753+0000","last_peered":"2024-09-18T11:50:36.443753+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:36.443753+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36532081,"num_objects":4460,"num_object_clones":0,"num_object_copies":17840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8608","reported_seq":8694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.432993+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:36.432993+0000","last_peered":"2024-09-18T11:50:36.432993+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:36.432993+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8608,"log_dups_size":0,"ondisk_log_size":8608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36231803,"num_objects":4423,"num_object_clones":0,"num_object_copies":17692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2337}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8539","reported_seq":8628,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.246954+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:37.246954+0000","last_peered":"2024-09-18T11:50:37.246954+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:37.246954+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8539,"log_dups_size":0,"ondisk_log_size":8539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36272763,"num_objects":4428,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8538,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8697","reported_seq":8818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.402891+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:36.402891+0000","last_peered":"2024-09-18T11:50:36.402891+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:36.402891+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35984630,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2269}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8530","reported_seq":12817,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:36.538891+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:36.538891+0000","last_peered":"2024-09-18T11:50:36.538891+0000","last_clean":"2024-09-18T11:50:36.538891+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:36.538891+0000","last_undegraded":"2024-09-18T11:50:36.538891+0000","last_fullsized":"2024-09-18T11:50:36.538891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35820790,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8374","reported_seq":8462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.449918+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:37.449918+0000","last_peered":"2024-09-18T11:50:37.449918+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:37.449918+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8374,"log_dups_size":0,"ondisk_log_size":8374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35493110,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8372,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":575137164,"num_objects":70213,"num_object_clones":0,"num_object_copies":280852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136874,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5267,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1149722624,"data_stored":1149722624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136901,"ondisk_log_size":136901,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583968,"kb_used_data":281936,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93787872,"statfs":{"total":96636764160,"available":96038780928,"internally_reserved":0,"allocated":288702464,"data_stored":288526998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,1,0,0,0,0,0,0,0,6],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.0609999999999999},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.47699999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.1160000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.218}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":9.9000000000000004},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.4409999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578644,"kb_used_data":281276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93793196,"statfs":{"total":96636764160,"available":96044232704,"internally_reserved":0,"allocated":288026624,"data_stored":287851158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,1,0,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":7.3289999999999997},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":7.3540000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.9020000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":6.5090000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":4.9370000000000003},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.51800000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579068,"kb_used_data":281268,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93792772,"statfs":{"total":96636764160,"available":96043798528,"internally_reserved":0,"allocated":288018432,"data_stored":287842966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.5830000000000002},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.9970000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":6.4320000000000004},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.3439999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":12.747},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.7389999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591016,"kb_used_data":280736,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93780824,"statfs":{"total":96636764160,"available":96031563776,"internally_reserved":0,"allocated":287473664,"data_stored":287298198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,2,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":2.0619999999999998},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.9009999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.794},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":1.9590000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":2.0390000000000001},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":1.9910000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286883840,"data_stored":286883840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":287674368,"data_stored":287674368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287182848,"data_stored":287182848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":287981568,"data_stored":287981568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:40.041 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:40.550 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:40.551 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:40.589 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-18T11:50:39.980929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":575137164,"num_objects":70213,"num_object_clones":0,"num_object_copies":280852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136874,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5267,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136901,"ondisk_log_size":136901,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2332696,"kb_used_data":1125216,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375154664,"statfs":{"total":386547056640,"available":384158375936,"internally_reserved":0,"allocated":1152221184,"data_stored":1151519320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,9,7,1,0,0,0,0,0,0,0,7,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17457152,"num_objects":-2131,"num_object_clones":0,"num_object_copies":-8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":385024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003199"},"pg_stats":[{"pgid":"1.f","version":"35'8740","reported_seq":8835,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.321145+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:36.321145+0000","last_peered":"2024-09-18T11:50:36.321145+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:36.321145+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36945920,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8740,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2405}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8620","reported_seq":12910,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:36.445883+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:36.445883+0000","last_peered":"2024-09-18T11:50:36.445883+0000","last_clean":"2024-09-18T11:50:36.445883+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:36.445883+0000","last_undegraded":"2024-09-18T11:50:36.445883+0000","last_fullsized":"2024-09-18T11:50:36.445883+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8620,"log_dups_size":0,"ondisk_log_size":8620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36336886,"num_objects":4436,"num_object_clones":0,"num_object_copies":17744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8460","reported_seq":8575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.124838+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:37.124838+0000","last_peered":"2024-09-18T11:50:37.124838+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:37.124838+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8460,"log_dups_size":0,"ondisk_log_size":8460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35617427,"num_objects":4349,"num_object_clones":0,"num_object_copies":17396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8459,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2309}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8583","reported_seq":8673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.536237+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:37.536237+0000","last_peered":"2024-09-18T11:50:37.536237+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:37.536237+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8583,"log_dups_size":0,"ondisk_log_size":8583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35575030,"num_objects":4343,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8516","reported_seq":8631,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.344642+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:36.344642+0000","last_peered":"2024-09-18T11:50:36.344642+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:36.344642+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35869942,"num_objects":4379,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8541","reported_seq":8658,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.356251+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:36.356251+0000","last_peered":"2024-09-18T11:50:36.356251+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:36.356251+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34775040,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8541,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2126}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8649","reported_seq":8754,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.444683+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:37.444683+0000","last_peered":"2024-09-18T11:50:37.444683+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:37.444683+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35771638,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8647,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2251}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8480","reported_seq":8587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.487274+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:37.487274+0000","last_peered":"2024-09-18T11:50:37.487274+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:37.487274+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36001014,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8718","reported_seq":10336,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.490943+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:36.490943+0000","last_peered":"2024-09-18T11:50:36.490943+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:36.490943+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8718,"log_dups_size":0,"ondisk_log_size":8718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36514284,"num_objects":4458,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":781,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8273","reported_seq":8385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.124256+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:37.124256+0000","last_peered":"2024-09-18T11:50:37.124256+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:37.124256+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394806,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8271,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2358}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8573","reported_seq":8690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.443753+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:36.443753+0000","last_peered":"2024-09-18T11:50:36.443753+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:36.443753+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36532081,"num_objects":4460,"num_object_clones":0,"num_object_copies":17840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8608","reported_seq":8694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.432993+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:36.432993+0000","last_peered":"2024-09-18T11:50:36.432993+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:36.432993+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8608,"log_dups_size":0,"ondisk_log_size":8608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36231803,"num_objects":4423,"num_object_clones":0,"num_object_copies":17692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2337}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8539","reported_seq":8628,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.246954+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:37.246954+0000","last_peered":"2024-09-18T11:50:37.246954+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:37.246954+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8539,"log_dups_size":0,"ondisk_log_size":8539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36272763,"num_objects":4428,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8538,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8697","reported_seq":8818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.402891+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:36.402891+0000","last_peered":"2024-09-18T11:50:36.402891+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:36.402891+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35984630,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2269}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8530","reported_seq":12817,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:36.538891+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:36.538891+0000","last_peered":"2024-09-18T11:50:36.538891+0000","last_clean":"2024-09-18T11:50:36.538891+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:36.538891+0000","last_undegraded":"2024-09-18T11:50:36.538891+0000","last_fullsized":"2024-09-18T11:50:36.538891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35820790,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8374","reported_seq":8462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.449918+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:37.449918+0000","last_peered":"2024-09-18T11:50:37.449918+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:37.449918+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8374,"log_dups_size":0,"ondisk_log_size":8374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35493110,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8372,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":575137164,"num_objects":70213,"num_object_clones":0,"num_object_copies":280852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136874,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5267,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1149722624,"data_stored":1149722624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136901,"ondisk_log_size":136901,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583968,"kb_used_data":281936,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93787872,"statfs":{"total":96636764160,"available":96038780928,"internally_reserved":0,"allocated":288702464,"data_stored":288526998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,1,0,0,0,0,0,0,0,6],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.0609999999999999},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.47699999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.1160000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.218}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":9.9000000000000004},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.4409999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578644,"kb_used_data":281276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93793196,"statfs":{"total":96636764160,"available":96044232704,"internally_reserved":0,"allocated":288026624,"data_stored":287851158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,1,0,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":7.3289999999999997},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":7.3540000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.9020000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":6.5090000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":4.9370000000000003},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.51800000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579068,"kb_used_data":281268,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93792772,"statfs":{"total":96636764160,"available":96043798528,"internally_reserved":0,"allocated":288018432,"data_stored":287842966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.5830000000000002},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.9970000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":6.4320000000000004},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.3439999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":12.747},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.7389999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591016,"kb_used_data":280736,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93780824,"statfs":{"total":96636764160,"available":96031563776,"internally_reserved":0,"allocated":287473664,"data_stored":287298198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,2,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":2.0619999999999998},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.9009999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.794},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":1.9590000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":2.0390000000000001},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":1.9910000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286883840,"data_stored":286883840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":287674368,"data_stored":287674368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287182848,"data_stored":287182848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":287981568,"data_stored":287981568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:40.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:40.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8740", 'reported_seq': 8835, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:36.321145+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:36.321145+0000', 'last_peered': '2024-09-18T11:50:36.321145+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:36.321145+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8740, 'log_dups_size': 0, 'ondisk_log_size': 8740, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36945920, 'num_objects': 4510, 'num_object_clones': 0, 'num_object_copies': 18040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2105, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8740, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2405}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:40.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:40.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8460", 'reported_seq': 8575, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:37.124838+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:37.124838+0000', 'last_peered': '2024-09-18T11:50:37.124838+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:37.124838+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8460, 'log_dups_size': 0, 'ondisk_log_size': 8460, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35617427, 'num_objects': 4349, 'num_object_clones': 0, 'num_object_copies': 17396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4349, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8459, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2309}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8583", 'reported_seq': 8673, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:37.536237+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:37.536237+0000', 'last_peered': '2024-09-18T11:50:37.536237+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:37.536237+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8583, 'log_dups_size': 0, 'ondisk_log_size': 8583, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35575030, 'num_objects': 4343, 'num_object_clones': 0, 'num_object_copies': 17372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8581, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2241}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8516", 'reported_seq': 8631, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:36.344642+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:36.344642+0000', 'last_peered': '2024-09-18T11:50:36.344642+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:36.344642+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8516, 'log_dups_size': 0, 'ondisk_log_size': 8516, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35869942, 'num_objects': 4379, 'num_object_clones': 0, 'num_object_copies': 17516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8514, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2325}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8541", 'reported_seq': 8658, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:36.356251+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:36.356251+0000', 'last_peered': '2024-09-18T11:50:36.356251+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:36.356251+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8541, 'log_dups_size': 0, 'ondisk_log_size': 8541, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34775040, 'num_objects': 4245, 'num_object_clones': 0, 'num_object_copies': 16980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8541, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2126}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8649", 'reported_seq': 8754, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:37.444683+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:37.444683+0000', 'last_peered': '2024-09-18T11:50:37.444683+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:37.444683+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8649, 'log_dups_size': 0, 'ondisk_log_size': 8649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35771638, 'num_objects': 4367, 'num_object_clones': 0, 'num_object_copies': 17468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8647, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2251}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8480", 'reported_seq': 8587, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:37.487274+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:37.487274+0000', 'last_peered': '2024-09-18T11:50:37.487274+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:37.487274+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8480, 'log_dups_size': 0, 'ondisk_log_size': 8480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36001014, 'num_objects': 4395, 'num_object_clones': 0, 'num_object_copies': 17580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8478, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2381}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:40.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8718", 'reported_seq': 10336, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:36.490943+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:36.490943+0000', 'last_peered': '2024-09-18T11:50:36.490943+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:36.490943+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8718, 'log_dups_size': 0, 'ondisk_log_size': 8718, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36514284, 'num_objects': 4458, 'num_object_clones': 0, 'num_object_copies': 17832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1360, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8715, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 781, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3098}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8273", 'reported_seq': 8385, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:37.124256+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:37.124256+0000', 'last_peered': '2024-09-18T11:50:37.124256+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:37.124256+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8273, 'log_dups_size': 0, 'ondisk_log_size': 8273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35394806, 'num_objects': 4321, 'num_object_clones': 0, 'num_object_copies': 17284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8271, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2358}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8573", 'reported_seq': 8690, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:36.443753+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:36.443753+0000', 'last_peered': '2024-09-18T11:50:36.443753+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:36.443753+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8573, 'log_dups_size': 0, 'ondisk_log_size': 8573, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36532081, 'num_objects': 4460, 'num_object_clones': 0, 'num_object_copies': 17840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8570, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2425}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8608", 'reported_seq': 8694, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:36.432993+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:36.432993+0000', 'last_peered': '2024-09-18T11:50:36.432993+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:36.432993+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8608, 'log_dups_size': 0, 'ondisk_log_size': 8608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36231803, 'num_objects': 4423, 'num_object_clones': 0, 'num_object_copies': 17692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2087, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4423, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8607, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8539", 'reported_seq': 8628, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:37.246954+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:37.246954+0000', 'last_peered': '2024-09-18T11:50:37.246954+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:37.246954+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8539, 'log_dups_size': 0, 'ondisk_log_size': 8539, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36272763, 'num_objects': 4428, 'num_object_clones': 0, 'num_object_copies': 17712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8538, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:40.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8697", 'reported_seq': 8818, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:36.402891+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:36.402891+0000', 'last_peered': '2024-09-18T11:50:36.402891+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:36.402891+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8697, 'log_dups_size': 0, 'ondisk_log_size': 8697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35984630, 'num_objects': 4393, 'num_object_clones': 0, 'num_object_copies': 17572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2126, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8695, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2269}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:40.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:40.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8374", 'reported_seq': 8462, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:37.449918+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:37.449918+0000', 'last_peered': '2024-09-18T11:50:37.449918+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:37.449918+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8374, 'log_dups_size': 0, 'ondisk_log_size': 8374, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35493110, 'num_objects': 4333, 'num_object_clones': 0, 'num_object_copies': 17332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4333, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8372, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2331}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:40.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:41.245 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:41.245 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28189/280852 objects degraded (10.037%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":439,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":70213,"data_bytes":575137164,"bytes_used":2388680704,"bytes_avail":384158375936,"bytes_total":386547056640,"degraded_objects":28189,"degraded_total":280852,"degraded_ratio":0.10036958967712532,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":32076,"recovering_keys_per_sec":0,"num_objects_recovered":168,"num_bytes_recovered":385024,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":177},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:41.276 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:41.276 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:41.822 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:41.823 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:41.854 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-18T11:50:39.980929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":575137164,"num_objects":70213,"num_object_clones":0,"num_object_copies":280852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136874,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5267,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136901,"ondisk_log_size":136901,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2332696,"kb_used_data":1125216,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375154664,"statfs":{"total":386547056640,"available":384158375936,"internally_reserved":0,"allocated":1152221184,"data_stored":1151519320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,9,7,1,0,0,0,0,0,0,0,7,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17457152,"num_objects":-2131,"num_object_clones":0,"num_object_copies":-8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":385024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003199"},"pg_stats":[{"pgid":"1.f","version":"35'8740","reported_seq":8835,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.321145+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:36.321145+0000","last_peered":"2024-09-18T11:50:36.321145+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:36.321145+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36945920,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8740,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2405}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8620","reported_seq":12910,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:36.445883+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:36.445883+0000","last_peered":"2024-09-18T11:50:36.445883+0000","last_clean":"2024-09-18T11:50:36.445883+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:36.445883+0000","last_undegraded":"2024-09-18T11:50:36.445883+0000","last_fullsized":"2024-09-18T11:50:36.445883+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8620,"log_dups_size":0,"ondisk_log_size":8620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36336886,"num_objects":4436,"num_object_clones":0,"num_object_copies":17744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8460","reported_seq":8575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.124838+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:37.124838+0000","last_peered":"2024-09-18T11:50:37.124838+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:37.124838+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8460,"log_dups_size":0,"ondisk_log_size":8460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35617427,"num_objects":4349,"num_object_clones":0,"num_object_copies":17396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8459,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2309}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8583","reported_seq":8673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.536237+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:37.536237+0000","last_peered":"2024-09-18T11:50:37.536237+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:37.536237+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8583,"log_dups_size":0,"ondisk_log_size":8583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35575030,"num_objects":4343,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8516","reported_seq":8631,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.344642+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:36.344642+0000","last_peered":"2024-09-18T11:50:36.344642+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:36.344642+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35869942,"num_objects":4379,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8541","reported_seq":8658,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.356251+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:36.356251+0000","last_peered":"2024-09-18T11:50:36.356251+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:36.356251+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34775040,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8541,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2126}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8649","reported_seq":8754,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.444683+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:37.444683+0000","last_peered":"2024-09-18T11:50:37.444683+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:37.444683+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35771638,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8647,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2251}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8480","reported_seq":8587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.487274+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:37.487274+0000","last_peered":"2024-09-18T11:50:37.487274+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:37.487274+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36001014,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8718","reported_seq":10336,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.490943+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:36.490943+0000","last_peered":"2024-09-18T11:50:36.490943+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:36.490943+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8718,"log_dups_size":0,"ondisk_log_size":8718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36514284,"num_objects":4458,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":781,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8273","reported_seq":8385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.124256+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:37.124256+0000","last_peered":"2024-09-18T11:50:37.124256+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:37.124256+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394806,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8271,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2358}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8573","reported_seq":8690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.443753+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:36.443753+0000","last_peered":"2024-09-18T11:50:36.443753+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:36.443753+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36532081,"num_objects":4460,"num_object_clones":0,"num_object_copies":17840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8608","reported_seq":8694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.432993+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:36.432993+0000","last_peered":"2024-09-18T11:50:36.432993+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:36.432993+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8608,"log_dups_size":0,"ondisk_log_size":8608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36231803,"num_objects":4423,"num_object_clones":0,"num_object_copies":17692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2337}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8539","reported_seq":8628,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.246954+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:37.246954+0000","last_peered":"2024-09-18T11:50:37.246954+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:37.246954+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8539,"log_dups_size":0,"ondisk_log_size":8539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36272763,"num_objects":4428,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8538,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8697","reported_seq":8818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:36.402891+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:36.402891+0000","last_peered":"2024-09-18T11:50:36.402891+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:36.402891+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35984630,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2269}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8530","reported_seq":12817,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:36.538891+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:36.538891+0000","last_peered":"2024-09-18T11:50:36.538891+0000","last_clean":"2024-09-18T11:50:36.538891+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:36.538891+0000","last_undegraded":"2024-09-18T11:50:36.538891+0000","last_fullsized":"2024-09-18T11:50:36.538891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35820790,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8374","reported_seq":8462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:37.449918+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:37.449918+0000","last_peered":"2024-09-18T11:50:37.449918+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:37.449918+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8374,"log_dups_size":0,"ondisk_log_size":8374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35493110,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8372,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":575137164,"num_objects":70213,"num_object_clones":0,"num_object_copies":280852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136874,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5267,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1149722624,"data_stored":1149722624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136901,"ondisk_log_size":136901,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583968,"kb_used_data":281936,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93787872,"statfs":{"total":96636764160,"available":96038780928,"internally_reserved":0,"allocated":288702464,"data_stored":288526998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,1,0,0,0,0,0,0,0,6],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.0609999999999999},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.47699999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.1160000000000001},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.218}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":9.9000000000000004},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":2.4409999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578644,"kb_used_data":281276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93793196,"statfs":{"total":96636764160,"available":96044232704,"internally_reserved":0,"allocated":288026624,"data_stored":287851158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,1,0,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":7.3289999999999997},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":7.3540000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.9020000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":6.5090000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":4.9370000000000003},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":0.51800000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579068,"kb_used_data":281268,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93792772,"statfs":{"total":96636764160,"available":96043798528,"internally_reserved":0,"allocated":288018432,"data_stored":287842966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.5830000000000002},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.9970000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":6.4320000000000004},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.3439999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":12.747},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.7389999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591016,"kb_used_data":280736,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93780824,"statfs":{"total":96636764160,"available":96031563776,"internally_reserved":0,"allocated":287473664,"data_stored":287298198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,2,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":2.0619999999999998},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.9009999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.794},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":1.9590000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":2.0390000000000001},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":1.9910000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286883840,"data_stored":286883840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":287674368,"data_stored":287674368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287182848,"data_stored":287182848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":287981568,"data_stored":287981568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:44.856 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:45.401 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:45.402 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:45.419 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-18T11:50:43.981885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":566584716,"num_objects":69169,"num_object_clones":0,"num_object_copies":276676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137918,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5347,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137945,"ondisk_log_size":137945,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2316548,"kb_used_data":1109056,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375170812,"statfs":{"total":386547056640,"available":384174911488,"internally_reserved":0,"allocated":1135673344,"data_stored":1134971480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,11,6,3,6,5,0,0,0,0,0,0,1,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20963328,"num_objects":-2559,"num_object_clones":0,"num_object_copies":-10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002957"},"pg_stats":[{"pgid":"1.f","version":"35'8807","reported_seq":8906,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.527806+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:41.527806+0000","last_peered":"2024-09-18T11:50:41.527806+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:41.527806+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8807,"log_dups_size":0,"ondisk_log_size":8807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36397056,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8807,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8679","reported_seq":12969,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:41.515684+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:41.515684+0000","last_peered":"2024-09-18T11:50:41.515684+0000","last_clean":"2024-09-18T11:50:41.515684+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:41.515684+0000","last_undegraded":"2024-09-18T11:50:41.515684+0000","last_fullsized":"2024-09-18T11:50:41.515684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8679,"log_dups_size":0,"ondisk_log_size":8679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853558,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8677,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8520","reported_seq":8635,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.932174+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:41.932174+0000","last_peered":"2024-09-18T11:50:41.932174+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:41.932174+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8520,"log_dups_size":0,"ondisk_log_size":8520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35125907,"num_objects":4289,"num_object_clones":0,"num_object_copies":17156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4289,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8519,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2249}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8662","reported_seq":8752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.553130+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:42.553130+0000","last_peered":"2024-09-18T11:50:42.553130+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:42.553130+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34927862,"num_objects":4264,"num_object_clones":0,"num_object_copies":17056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8587","reported_seq":8706,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.459227+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:41.459227+0000","last_peered":"2024-09-18T11:50:41.459227+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:41.459227+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8587,"log_dups_size":0,"ondisk_log_size":8587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35288310,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8585,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8600","reported_seq":8717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.533470+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:41.533470+0000","last_peered":"2024-09-18T11:50:41.533470+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:41.533470+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34291712,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8707","reported_seq":8812,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.344021+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:42.344021+0000","last_peered":"2024-09-18T11:50:42.344021+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:42.344021+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8707,"log_dups_size":0,"ondisk_log_size":8707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35296502,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2193}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8548","reported_seq":8655,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.540709+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:42.540709+0000","last_peered":"2024-09-18T11:50:42.540709+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:42.540709+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35443958,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2313}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8798","reported_seq":10561,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.535203+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:41.535203+0000","last_peered":"2024-09-18T11:50:41.535203+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:41.535203+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8798,"log_dups_size":0,"ondisk_log_size":8798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35858924,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8795,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8326","reported_seq":8438,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.808722+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:41.808722+0000","last_peered":"2024-09-18T11:50:41.808722+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:41.808722+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34960630,"num_objects":4268,"num_object_clones":0,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2305}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8639","reported_seq":8758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.253866+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:41.253866+0000","last_peered":"2024-09-18T11:50:41.253866+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:41.253866+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8639,"log_dups_size":0,"ondisk_log_size":8639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35991409,"num_objects":4394,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8636,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8674","reported_seq":8765,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.471073+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:41.471073+0000","last_peered":"2024-09-18T11:50:41.471073+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:41.471073+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8674,"log_dups_size":0,"ondisk_log_size":8674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35691131,"num_objects":4357,"num_object_clones":0,"num_object_copies":17428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8673,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8589","reported_seq":8679,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.093917+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:42.093917+0000","last_peered":"2024-09-18T11:50:42.093917+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:42.093917+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35863163,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2329}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8769","reported_seq":8892,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.353717+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:41.353717+0000","last_peered":"2024-09-18T11:50:41.353717+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:41.353717+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8769,"log_dups_size":0,"ondisk_log_size":8769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394806,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8767,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8606","reported_seq":12892,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:41.504639+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:41.504639+0000","last_peered":"2024-09-18T11:50:41.504639+0000","last_clean":"2024-09-18T11:50:41.504639+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:41.504639+0000","last_undegraded":"2024-09-18T11:50:41.504639+0000","last_fullsized":"2024-09-18T11:50:41.504639+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8606,"log_dups_size":0,"ondisk_log_size":8606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35198198,"num_objects":4297,"num_object_clones":0,"num_object_copies":17188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8604,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8434","reported_seq":8522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.476341+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:42.476341+0000","last_peered":"2024-09-18T11:50:42.476341+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:42.476341+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8434,"log_dups_size":0,"ondisk_log_size":8434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35001590,"num_objects":4273,"num_object_clones":0,"num_object_copies":17092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8432,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2271}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":566584716,"num_objects":69169,"num_object_clones":0,"num_object_copies":276676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137918,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5347,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1132892160,"data_stored":1132892160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137945,"ondisk_log_size":137945,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583856,"kb_used_data":277664,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93787984,"statfs":{"total":96636764160,"available":96038895616,"internally_reserved":0,"allocated":284327936,"data_stored":284152470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":17.077999999999999},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.2679999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":17.042000000000002},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":0.32300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":7.0940000000000003},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.9630000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578508,"kb_used_data":276996,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93793332,"statfs":{"total":96636764160,"available":96044371968,"internally_reserved":0,"allocated":283643904,"data_stored":283468438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,0,2,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":2.9049999999999998},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.944}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":13.114000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.23999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.0549999999999997},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.5780000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579756,"kb_used_data":277768,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93792084,"statfs":{"total":96636764160,"available":96043094016,"internally_reserved":0,"allocated":284434432,"data_stored":284258966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":10.523999999999999},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.9249999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":4.7699999999999996},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":10.486000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.1159999999999997},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.6289999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574428,"kb_used_data":276628,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93797412,"statfs":{"total":96636764160,"available":96048549888,"internally_reserved":0,"allocated":283267072,"data_stored":283091606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,2,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":7.7519999999999998},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.5060000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":17.706},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.5350000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.7880000000000003},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":18.57}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282595328,"data_stored":282595328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":283860992,"data_stored":283860992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":282890240,"data_stored":282890240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283545600,"data_stored":283545600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:45.420 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:45.956 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:45.957 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:45.983 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-18T11:50:43.981885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":566584716,"num_objects":69169,"num_object_clones":0,"num_object_copies":276676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137918,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5347,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137945,"ondisk_log_size":137945,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2316548,"kb_used_data":1109056,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375170812,"statfs":{"total":386547056640,"available":384174911488,"internally_reserved":0,"allocated":1135673344,"data_stored":1134971480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,11,6,3,6,5,0,0,0,0,0,0,1,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20963328,"num_objects":-2559,"num_object_clones":0,"num_object_copies":-10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002957"},"pg_stats":[{"pgid":"1.f","version":"35'8807","reported_seq":8906,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.527806+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:41.527806+0000","last_peered":"2024-09-18T11:50:41.527806+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:41.527806+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8807,"log_dups_size":0,"ondisk_log_size":8807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36397056,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8807,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8679","reported_seq":12969,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:41.515684+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:41.515684+0000","last_peered":"2024-09-18T11:50:41.515684+0000","last_clean":"2024-09-18T11:50:41.515684+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:41.515684+0000","last_undegraded":"2024-09-18T11:50:41.515684+0000","last_fullsized":"2024-09-18T11:50:41.515684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8679,"log_dups_size":0,"ondisk_log_size":8679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853558,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8677,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8520","reported_seq":8635,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.932174+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:41.932174+0000","last_peered":"2024-09-18T11:50:41.932174+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:41.932174+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8520,"log_dups_size":0,"ondisk_log_size":8520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35125907,"num_objects":4289,"num_object_clones":0,"num_object_copies":17156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4289,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8519,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2249}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8662","reported_seq":8752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.553130+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:42.553130+0000","last_peered":"2024-09-18T11:50:42.553130+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:42.553130+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34927862,"num_objects":4264,"num_object_clones":0,"num_object_copies":17056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8587","reported_seq":8706,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.459227+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:41.459227+0000","last_peered":"2024-09-18T11:50:41.459227+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:41.459227+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8587,"log_dups_size":0,"ondisk_log_size":8587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35288310,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8585,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8600","reported_seq":8717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.533470+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:41.533470+0000","last_peered":"2024-09-18T11:50:41.533470+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:41.533470+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34291712,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8707","reported_seq":8812,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.344021+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:42.344021+0000","last_peered":"2024-09-18T11:50:42.344021+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:42.344021+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8707,"log_dups_size":0,"ondisk_log_size":8707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35296502,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2193}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8548","reported_seq":8655,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.540709+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:42.540709+0000","last_peered":"2024-09-18T11:50:42.540709+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:42.540709+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35443958,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2313}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8798","reported_seq":10561,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.535203+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:41.535203+0000","last_peered":"2024-09-18T11:50:41.535203+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:41.535203+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8798,"log_dups_size":0,"ondisk_log_size":8798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35858924,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8795,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8326","reported_seq":8438,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.808722+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:41.808722+0000","last_peered":"2024-09-18T11:50:41.808722+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:41.808722+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34960630,"num_objects":4268,"num_object_clones":0,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2305}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8639","reported_seq":8758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.253866+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:41.253866+0000","last_peered":"2024-09-18T11:50:41.253866+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:41.253866+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8639,"log_dups_size":0,"ondisk_log_size":8639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35991409,"num_objects":4394,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8636,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8674","reported_seq":8765,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.471073+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:41.471073+0000","last_peered":"2024-09-18T11:50:41.471073+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:41.471073+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8674,"log_dups_size":0,"ondisk_log_size":8674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35691131,"num_objects":4357,"num_object_clones":0,"num_object_copies":17428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8673,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8589","reported_seq":8679,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.093917+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:42.093917+0000","last_peered":"2024-09-18T11:50:42.093917+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:42.093917+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35863163,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2329}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8769","reported_seq":8892,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.353717+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:41.353717+0000","last_peered":"2024-09-18T11:50:41.353717+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:41.353717+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8769,"log_dups_size":0,"ondisk_log_size":8769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394806,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8767,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8606","reported_seq":12892,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:41.504639+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:41.504639+0000","last_peered":"2024-09-18T11:50:41.504639+0000","last_clean":"2024-09-18T11:50:41.504639+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:41.504639+0000","last_undegraded":"2024-09-18T11:50:41.504639+0000","last_fullsized":"2024-09-18T11:50:41.504639+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8606,"log_dups_size":0,"ondisk_log_size":8606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35198198,"num_objects":4297,"num_object_clones":0,"num_object_copies":17188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8604,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8434","reported_seq":8522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.476341+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:42.476341+0000","last_peered":"2024-09-18T11:50:42.476341+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:42.476341+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8434,"log_dups_size":0,"ondisk_log_size":8434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35001590,"num_objects":4273,"num_object_clones":0,"num_object_copies":17092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8432,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2271}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":566584716,"num_objects":69169,"num_object_clones":0,"num_object_copies":276676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137918,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5347,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1132892160,"data_stored":1132892160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137945,"ondisk_log_size":137945,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583856,"kb_used_data":277664,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93787984,"statfs":{"total":96636764160,"available":96038895616,"internally_reserved":0,"allocated":284327936,"data_stored":284152470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":17.077999999999999},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.2679999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":17.042000000000002},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":0.32300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":7.0940000000000003},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.9630000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578508,"kb_used_data":276996,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93793332,"statfs":{"total":96636764160,"available":96044371968,"internally_reserved":0,"allocated":283643904,"data_stored":283468438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,0,2,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":2.9049999999999998},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.944}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":13.114000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.23999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.0549999999999997},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.5780000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579756,"kb_used_data":277768,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93792084,"statfs":{"total":96636764160,"available":96043094016,"internally_reserved":0,"allocated":284434432,"data_stored":284258966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":10.523999999999999},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.9249999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":4.7699999999999996},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":10.486000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.1159999999999997},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.6289999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574428,"kb_used_data":276628,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93797412,"statfs":{"total":96636764160,"available":96048549888,"internally_reserved":0,"allocated":283267072,"data_stored":283091606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,2,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":7.7519999999999998},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.5060000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":17.706},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.5350000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.7880000000000003},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":18.57}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282595328,"data_stored":282595328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":283860992,"data_stored":283860992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":282890240,"data_stored":282890240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283545600,"data_stored":283545600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:45.984 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:45.984 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8807", 'reported_seq': 8906, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:41.527806+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:41.527806+0000', 'last_peered': '2024-09-18T11:50:41.527806+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:41.527806+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8807, 'log_dups_size': 0, 'ondisk_log_size': 8807, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36397056, 'num_objects': 4443, 'num_object_clones': 0, 'num_object_copies': 17772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8807, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:45.984 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:45.984 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8520", 'reported_seq': 8635, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:41.932174+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:41.932174+0000', 'last_peered': '2024-09-18T11:50:41.932174+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:41.932174+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8520, 'log_dups_size': 0, 'ondisk_log_size': 8520, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35125907, 'num_objects': 4289, 'num_object_clones': 0, 'num_object_copies': 17156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4289, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8519, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2249}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8662", 'reported_seq': 8752, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:42.553130+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:42.553130+0000', 'last_peered': '2024-09-18T11:50:42.553130+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:42.553130+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8662, 'log_dups_size': 0, 'ondisk_log_size': 8662, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34927862, 'num_objects': 4264, 'num_object_clones': 0, 'num_object_copies': 17056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8660, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8587", 'reported_seq': 8706, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:41.459227+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:41.459227+0000', 'last_peered': '2024-09-18T11:50:41.459227+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:41.459227+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8587, 'log_dups_size': 0, 'ondisk_log_size': 8587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35288310, 'num_objects': 4308, 'num_object_clones': 0, 'num_object_copies': 17232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8585, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2256}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8600", 'reported_seq': 8717, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:41.533470+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:41.533470+0000', 'last_peered': '2024-09-18T11:50:41.533470+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:41.533470+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8600, 'log_dups_size': 0, 'ondisk_log_size': 8600, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34291712, 'num_objects': 4186, 'num_object_clones': 0, 'num_object_copies': 16744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8600, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2067}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8707", 'reported_seq': 8812, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:42.344021+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:42.344021+0000', 'last_peered': '2024-09-18T11:50:42.344021+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:42.344021+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8707, 'log_dups_size': 0, 'ondisk_log_size': 8707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35296502, 'num_objects': 4309, 'num_object_clones': 0, 'num_object_copies': 17236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8705, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2193}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8548", 'reported_seq': 8655, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:42.540709+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:42.540709+0000', 'last_peered': '2024-09-18T11:50:42.540709+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:42.540709+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8548, 'log_dups_size': 0, 'ondisk_log_size': 8548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35443958, 'num_objects': 4327, 'num_object_clones': 0, 'num_object_copies': 17308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8546, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2313}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:45.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8798", 'reported_seq': 10561, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:41.535203+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:41.535203+0000', 'last_peered': '2024-09-18T11:50:41.535203+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:41.535203+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8798, 'log_dups_size': 0, 'ondisk_log_size': 8798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35858924, 'num_objects': 4378, 'num_object_clones': 0, 'num_object_copies': 17512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1288, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8795, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 853, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3090}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8326", 'reported_seq': 8438, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:41.808722+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:41.808722+0000', 'last_peered': '2024-09-18T11:50:41.808722+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:41.808722+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8326, 'log_dups_size': 0, 'ondisk_log_size': 8326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34960630, 'num_objects': 4268, 'num_object_clones': 0, 'num_object_copies': 17072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4268, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8324, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2305}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8639", 'reported_seq': 8758, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:41.253866+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:41.253866+0000', 'last_peered': '2024-09-18T11:50:41.253866+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:41.253866+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8639, 'log_dups_size': 0, 'ondisk_log_size': 8639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35991409, 'num_objects': 4394, 'num_object_clones': 0, 'num_object_copies': 17576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8636, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2360}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8674", 'reported_seq': 8765, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:41.471073+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:41.471073+0000', 'last_peered': '2024-09-18T11:50:41.471073+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:41.471073+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8674, 'log_dups_size': 0, 'ondisk_log_size': 8674, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35691131, 'num_objects': 4357, 'num_object_clones': 0, 'num_object_copies': 17428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8673, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8589", 'reported_seq': 8679, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:42.093917+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:42.093917+0000', 'last_peered': '2024-09-18T11:50:42.093917+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:42.093917+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8589, 'log_dups_size': 0, 'ondisk_log_size': 8589, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35863163, 'num_objects': 4378, 'num_object_clones': 0, 'num_object_copies': 17512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8588, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2329}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:45.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8769", 'reported_seq': 8892, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:41.353717+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:41.353717+0000', 'last_peered': '2024-09-18T11:50:41.353717+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:41.353717+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8769, 'log_dups_size': 0, 'ondisk_log_size': 8769, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35394806, 'num_objects': 4321, 'num_object_clones': 0, 'num_object_copies': 17284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8767, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:45.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:45.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8434", 'reported_seq': 8522, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:42.476341+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:42.476341+0000', 'last_peered': '2024-09-18T11:50:42.476341+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:42.476341+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8434, 'log_dups_size': 0, 'ondisk_log_size': 8434, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35001590, 'num_objects': 4273, 'num_object_clones': 0, 'num_object_copies': 17092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8432, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2271}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:45.987 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:46.635 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:46.635 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28109/276676 objects degraded (10.160%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":444,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":69169,"data_bytes":566584716,"bytes_used":2372145152,"bytes_avail":384174911488,"bytes_total":386547056640,"degraded_objects":28109,"degraded_total":276676,"degraded_ratio":0.1015953678671081,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":176},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:46.658 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:46.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:47.175 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:47.175 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:47.191 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-18T11:50:45.982546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":566584716,"num_objects":69169,"num_object_clones":0,"num_object_copies":276676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137918,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5347,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137945,"ondisk_log_size":137945,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2316548,"kb_used_data":1109056,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375170812,"statfs":{"total":386547056640,"available":384174911488,"internally_reserved":0,"allocated":1135673344,"data_stored":1134971480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,11,6,3,6,5,0,0,0,0,0,0,1,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17326080,"num_objects":-2115,"num_object_clones":0,"num_object_copies":-8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2115,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003105"},"pg_stats":[{"pgid":"1.f","version":"35'8807","reported_seq":8906,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.527806+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:41.527806+0000","last_peered":"2024-09-18T11:50:41.527806+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:41.527806+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8807,"log_dups_size":0,"ondisk_log_size":8807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36397056,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8807,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8679","reported_seq":12969,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:41.515684+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:41.515684+0000","last_peered":"2024-09-18T11:50:41.515684+0000","last_clean":"2024-09-18T11:50:41.515684+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:41.515684+0000","last_undegraded":"2024-09-18T11:50:41.515684+0000","last_fullsized":"2024-09-18T11:50:41.515684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8679,"log_dups_size":0,"ondisk_log_size":8679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853558,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8677,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8520","reported_seq":8635,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.932174+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:41.932174+0000","last_peered":"2024-09-18T11:50:41.932174+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:41.932174+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8520,"log_dups_size":0,"ondisk_log_size":8520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35125907,"num_objects":4289,"num_object_clones":0,"num_object_copies":17156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4289,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8519,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2249}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8662","reported_seq":8752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.553130+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:42.553130+0000","last_peered":"2024-09-18T11:50:42.553130+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:42.553130+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34927862,"num_objects":4264,"num_object_clones":0,"num_object_copies":17056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8587","reported_seq":8706,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.459227+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:41.459227+0000","last_peered":"2024-09-18T11:50:41.459227+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:41.459227+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8587,"log_dups_size":0,"ondisk_log_size":8587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35288310,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8585,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8600","reported_seq":8717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.533470+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:41.533470+0000","last_peered":"2024-09-18T11:50:41.533470+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:41.533470+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34291712,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8707","reported_seq":8812,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.344021+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:42.344021+0000","last_peered":"2024-09-18T11:50:42.344021+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:42.344021+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8707,"log_dups_size":0,"ondisk_log_size":8707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35296502,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2193}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8548","reported_seq":8655,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.540709+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:42.540709+0000","last_peered":"2024-09-18T11:50:42.540709+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:42.540709+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35443958,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2313}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8798","reported_seq":10561,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.535203+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:41.535203+0000","last_peered":"2024-09-18T11:50:41.535203+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:41.535203+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8798,"log_dups_size":0,"ondisk_log_size":8798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35858924,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8795,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8326","reported_seq":8438,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.808722+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:41.808722+0000","last_peered":"2024-09-18T11:50:41.808722+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:41.808722+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34960630,"num_objects":4268,"num_object_clones":0,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2305}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8639","reported_seq":8758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.253866+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:41.253866+0000","last_peered":"2024-09-18T11:50:41.253866+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:41.253866+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8639,"log_dups_size":0,"ondisk_log_size":8639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35991409,"num_objects":4394,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8636,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8674","reported_seq":8765,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.471073+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:41.471073+0000","last_peered":"2024-09-18T11:50:41.471073+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:41.471073+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8674,"log_dups_size":0,"ondisk_log_size":8674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35691131,"num_objects":4357,"num_object_clones":0,"num_object_copies":17428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8673,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8589","reported_seq":8679,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.093917+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:42.093917+0000","last_peered":"2024-09-18T11:50:42.093917+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:42.093917+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35863163,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2329}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8769","reported_seq":8892,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:41.353717+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:41.353717+0000","last_peered":"2024-09-18T11:50:41.353717+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:41.353717+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8769,"log_dups_size":0,"ondisk_log_size":8769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394806,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8767,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8606","reported_seq":12892,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:41.504639+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:41.504639+0000","last_peered":"2024-09-18T11:50:41.504639+0000","last_clean":"2024-09-18T11:50:41.504639+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:41.504639+0000","last_undegraded":"2024-09-18T11:50:41.504639+0000","last_fullsized":"2024-09-18T11:50:41.504639+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8606,"log_dups_size":0,"ondisk_log_size":8606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35198198,"num_objects":4297,"num_object_clones":0,"num_object_copies":17188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8604,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8434","reported_seq":8522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:42.476341+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:42.476341+0000","last_peered":"2024-09-18T11:50:42.476341+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:42.476341+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8434,"log_dups_size":0,"ondisk_log_size":8434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35001590,"num_objects":4273,"num_object_clones":0,"num_object_copies":17092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8432,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2271}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":566584716,"num_objects":69169,"num_object_clones":0,"num_object_copies":276676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137918,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5347,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1132892160,"data_stored":1132892160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137945,"ondisk_log_size":137945,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583856,"kb_used_data":277664,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93787984,"statfs":{"total":96636764160,"available":96038895616,"internally_reserved":0,"allocated":284327936,"data_stored":284152470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":17.077999999999999},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":7.2679999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":17.042000000000002},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":0.32300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":7.0940000000000003},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.9630000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578508,"kb_used_data":276996,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93793332,"statfs":{"total":96636764160,"available":96044371968,"internally_reserved":0,"allocated":283643904,"data_stored":283468438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,0,2,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":2.9049999999999998},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":2.944}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":13.114000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.23999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.0549999999999997},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.5780000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579756,"kb_used_data":277768,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93792084,"statfs":{"total":96636764160,"available":96043094016,"internally_reserved":0,"allocated":284434432,"data_stored":284258966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":10.523999999999999},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.9249999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":4.7699999999999996},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":10.486000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.1159999999999997},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.6289999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574428,"kb_used_data":276628,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93797412,"statfs":{"total":96636764160,"available":96048549888,"internally_reserved":0,"allocated":283267072,"data_stored":283091606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,2,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":7.7519999999999998},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.5060000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":17.706},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.5350000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.7880000000000003},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":18.57}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282595328,"data_stored":282595328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":283860992,"data_stored":283860992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":282890240,"data_stored":282890240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283545600,"data_stored":283545600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:50.194 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:50.767 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:50.800 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-18T11:50:49.984198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":557540748,"num_objects":68065,"num_object_clones":0,"num_object_copies":272260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139022,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5418,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139049,"ondisk_log_size":139049,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2315940,"kb_used_data":1091832,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375171420,"statfs":{"total":386547056640,"available":384175534080,"internally_reserved":0,"allocated":1118035968,"data_stored":1117334104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,5,7,1,10,3,0,0,0,0,0,0,0,4],"upper_bound":65536},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17596416,"num_objects":-2148,"num_object_clones":0,"num_object_copies":-8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2148,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003811"},"pg_stats":[{"pgid":"1.f","version":"35'8881","reported_seq":8980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.411050+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:46.411050+0000","last_peered":"2024-09-18T11:50:46.411050+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:46.411050+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8881,"log_dups_size":0,"ondisk_log_size":8881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35790848,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8881,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8746","reported_seq":13036,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:46.642779+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:46.642779+0000","last_peered":"2024-09-18T11:50:46.642779+0000","last_clean":"2024-09-18T11:50:46.642779+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:46.642779+0000","last_undegraded":"2024-09-18T11:50:46.642779+0000","last_fullsized":"2024-09-18T11:50:46.642779+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8746,"log_dups_size":0,"ondisk_log_size":8746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304694,"num_objects":4310,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8744,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8578","reported_seq":8693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.057356+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:47.057356+0000","last_peered":"2024-09-18T11:50:47.057356+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:47.057356+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8578,"log_dups_size":0,"ondisk_log_size":8578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34650771,"num_objects":4231,"num_object_clones":0,"num_object_copies":16924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8577,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2191}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8724","reported_seq":8814,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.444410+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:47.444410+0000","last_peered":"2024-09-18T11:50:47.444410+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:47.444410+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8724,"log_dups_size":0,"ondisk_log_size":8724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34419958,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8722,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8657","reported_seq":8776,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.456706+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:46.456706+0000","last_peered":"2024-09-18T11:50:46.456706+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:46.456706+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34714870,"num_objects":4238,"num_object_clones":0,"num_object_copies":16952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8655,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8667","reported_seq":8783,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.532728+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:46.532728+0000","last_peered":"2024-09-18T11:50:46.532728+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:46.532728+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8667,"log_dups_size":0,"ondisk_log_size":8667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33742848,"num_objects":4119,"num_object_clones":0,"num_object_copies":16476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8667,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8770","reported_seq":8875,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.576554+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:47.576554+0000","last_peered":"2024-09-18T11:50:47.576554+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:47.576554+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8770,"log_dups_size":0,"ondisk_log_size":8770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34780406,"num_objects":4246,"num_object_clones":0,"num_object_copies":16984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8768,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2130}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8611","reported_seq":8718,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.481700+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:47.481700+0000","last_peered":"2024-09-18T11:50:47.481700+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:47.481700+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34927862,"num_objects":4264,"num_object_clones":0,"num_object_copies":17056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8609,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8852","reported_seq":10757,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.552486+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:46.552486+0000","last_peered":"2024-09-18T11:50:46.552486+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:46.552486+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8852,"log_dups_size":0,"ondisk_log_size":8852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35416556,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8849,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":924,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8400","reported_seq":8511,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.226169+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:47.226169+0000","last_peered":"2024-09-18T11:50:47.226169+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:47.226169+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8400,"log_dups_size":0,"ondisk_log_size":8400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34354422,"num_objects":4194,"num_object_clones":0,"num_object_copies":16776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8398,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8709","reported_seq":8827,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.479698+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:46.479698+0000","last_peered":"2024-09-18T11:50:46.479698+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:46.479698+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35417969,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8706,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8751","reported_seq":8842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.537047+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:46.537047+0000","last_peered":"2024-09-18T11:50:46.537047+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:46.537047+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8751,"log_dups_size":0,"ondisk_log_size":8751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35060347,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8750,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8676","reported_seq":8766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.125584+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:47.125584+0000","last_peered":"2024-09-18T11:50:47.125584+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:47.125584+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8676,"log_dups_size":0,"ondisk_log_size":8676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35150459,"num_objects":4291,"num_object_clones":0,"num_object_copies":17164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8675,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8854","reported_seq":8976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.365801+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:46.365801+0000","last_peered":"2024-09-18T11:50:46.365801+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:46.365801+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34698486,"num_objects":4236,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8671","reported_seq":12958,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:46.457552+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:46.457552+0000","last_peered":"2024-09-18T11:50:46.457552+0000","last_clean":"2024-09-18T11:50:46.457552+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:46.457552+0000","last_undegraded":"2024-09-18T11:50:46.457552+0000","last_fullsized":"2024-09-18T11:50:46.457552+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665718,"num_objects":4232,"num_object_clones":0,"num_object_copies":16928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8502","reported_seq":8590,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.489273+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:47.489273+0000","last_peered":"2024-09-18T11:50:47.489273+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:47.489273+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8502,"log_dups_size":0,"ondisk_log_size":8502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34444534,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8500,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2203}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":557540748,"num_objects":68065,"num_object_clones":0,"num_object_copies":272260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139022,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5418,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1115148288,"data_stored":1115148288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139049,"ondisk_log_size":139049,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583604,"kb_used_data":273260,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93788236,"statfs":{"total":96636764160,"available":96039153664,"internally_reserved":0,"allocated":279818240,"data_stored":279642774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,1,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.3490000000000002},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.3140000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.3839999999999999},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":7.343}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":42.752000000000002},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.5910000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578440,"kb_used_data":272760,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93793400,"statfs":{"total":96636764160,"available":96044441600,"internally_reserved":0,"allocated":279306240,"data_stored":279130774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,0,0,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.372}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.8650000000000002},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.9079999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.264}]}]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579612,"kb_used_data":273488,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93792228,"statfs":{"total":96636764160,"available":96043241472,"internally_reserved":0,"allocated":280051712,"data_stored":279876246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":11.728},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.7529999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":5.7530000000000001},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.26200000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":12.593},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":10.433999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574284,"kb_used_data":272324,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93797556,"statfs":{"total":96636764160,"available":96048697344,"internally_reserved":0,"allocated":278859776,"data_stored":278684310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,0,0,2,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":0.34300000000000003},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":8.4860000000000007},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.4820000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":8.8109999999999999},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.30199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278183936,"data_stored":278183936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279379968,"data_stored":279379968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278491136,"data_stored":278491136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":279093248,"data_stored":279093248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:50.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:51.336 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:51.356 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-18T11:50:49.984198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":557540748,"num_objects":68065,"num_object_clones":0,"num_object_copies":272260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139022,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5418,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139049,"ondisk_log_size":139049,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2315940,"kb_used_data":1091832,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375171420,"statfs":{"total":386547056640,"available":384175534080,"internally_reserved":0,"allocated":1118035968,"data_stored":1117334104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,5,7,1,10,3,0,0,0,0,0,0,0,4],"upper_bound":65536},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17596416,"num_objects":-2148,"num_object_clones":0,"num_object_copies":-8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2148,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003811"},"pg_stats":[{"pgid":"1.f","version":"35'8881","reported_seq":8980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.411050+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:46.411050+0000","last_peered":"2024-09-18T11:50:46.411050+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:46.411050+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8881,"log_dups_size":0,"ondisk_log_size":8881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35790848,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8881,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8746","reported_seq":13036,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:46.642779+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:46.642779+0000","last_peered":"2024-09-18T11:50:46.642779+0000","last_clean":"2024-09-18T11:50:46.642779+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:46.642779+0000","last_undegraded":"2024-09-18T11:50:46.642779+0000","last_fullsized":"2024-09-18T11:50:46.642779+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8746,"log_dups_size":0,"ondisk_log_size":8746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304694,"num_objects":4310,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8744,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8578","reported_seq":8693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.057356+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:47.057356+0000","last_peered":"2024-09-18T11:50:47.057356+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:47.057356+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8578,"log_dups_size":0,"ondisk_log_size":8578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34650771,"num_objects":4231,"num_object_clones":0,"num_object_copies":16924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8577,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2191}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8724","reported_seq":8814,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.444410+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:47.444410+0000","last_peered":"2024-09-18T11:50:47.444410+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:47.444410+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8724,"log_dups_size":0,"ondisk_log_size":8724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34419958,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8722,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8657","reported_seq":8776,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.456706+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:46.456706+0000","last_peered":"2024-09-18T11:50:46.456706+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:46.456706+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34714870,"num_objects":4238,"num_object_clones":0,"num_object_copies":16952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8655,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8667","reported_seq":8783,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.532728+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:46.532728+0000","last_peered":"2024-09-18T11:50:46.532728+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:46.532728+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8667,"log_dups_size":0,"ondisk_log_size":8667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33742848,"num_objects":4119,"num_object_clones":0,"num_object_copies":16476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8667,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8770","reported_seq":8875,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.576554+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:47.576554+0000","last_peered":"2024-09-18T11:50:47.576554+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:47.576554+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8770,"log_dups_size":0,"ondisk_log_size":8770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34780406,"num_objects":4246,"num_object_clones":0,"num_object_copies":16984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8768,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2130}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8611","reported_seq":8718,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.481700+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:47.481700+0000","last_peered":"2024-09-18T11:50:47.481700+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:47.481700+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34927862,"num_objects":4264,"num_object_clones":0,"num_object_copies":17056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8609,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8852","reported_seq":10757,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.552486+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:46.552486+0000","last_peered":"2024-09-18T11:50:46.552486+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:46.552486+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8852,"log_dups_size":0,"ondisk_log_size":8852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35416556,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8849,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":924,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8400","reported_seq":8511,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.226169+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:47.226169+0000","last_peered":"2024-09-18T11:50:47.226169+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:47.226169+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8400,"log_dups_size":0,"ondisk_log_size":8400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34354422,"num_objects":4194,"num_object_clones":0,"num_object_copies":16776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8398,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8709","reported_seq":8827,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.479698+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:46.479698+0000","last_peered":"2024-09-18T11:50:46.479698+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:46.479698+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35417969,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8706,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8751","reported_seq":8842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.537047+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:46.537047+0000","last_peered":"2024-09-18T11:50:46.537047+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:46.537047+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8751,"log_dups_size":0,"ondisk_log_size":8751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35060347,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8750,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8676","reported_seq":8766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.125584+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:47.125584+0000","last_peered":"2024-09-18T11:50:47.125584+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:47.125584+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8676,"log_dups_size":0,"ondisk_log_size":8676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35150459,"num_objects":4291,"num_object_clones":0,"num_object_copies":17164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8675,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8854","reported_seq":8976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:46.365801+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:46.365801+0000","last_peered":"2024-09-18T11:50:46.365801+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:46.365801+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34698486,"num_objects":4236,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8671","reported_seq":12958,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:46.457552+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:46.457552+0000","last_peered":"2024-09-18T11:50:46.457552+0000","last_clean":"2024-09-18T11:50:46.457552+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:46.457552+0000","last_undegraded":"2024-09-18T11:50:46.457552+0000","last_fullsized":"2024-09-18T11:50:46.457552+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665718,"num_objects":4232,"num_object_clones":0,"num_object_copies":16928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8502","reported_seq":8590,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.489273+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:47.489273+0000","last_peered":"2024-09-18T11:50:47.489273+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:47.489273+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8502,"log_dups_size":0,"ondisk_log_size":8502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34444534,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8500,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2203}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":557540748,"num_objects":68065,"num_object_clones":0,"num_object_copies":272260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139022,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5418,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1115148288,"data_stored":1115148288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139049,"ondisk_log_size":139049,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583604,"kb_used_data":273260,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93788236,"statfs":{"total":96636764160,"available":96039153664,"internally_reserved":0,"allocated":279818240,"data_stored":279642774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,1,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.3490000000000002},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":3.3140000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":3.3839999999999999},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":7.343}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":42.752000000000002},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.5910000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578440,"kb_used_data":272760,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93793400,"statfs":{"total":96636764160,"available":96044441600,"internally_reserved":0,"allocated":279306240,"data_stored":279130774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,0,0,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.372}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.8650000000000002},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.9079999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.264}]}]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579612,"kb_used_data":273488,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93792228,"statfs":{"total":96636764160,"available":96043241472,"internally_reserved":0,"allocated":280051712,"data_stored":279876246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":11.728},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.7529999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":5.7530000000000001},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.26200000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":12.593},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":10.433999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574284,"kb_used_data":272324,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93797556,"statfs":{"total":96636764160,"available":96048697344,"internally_reserved":0,"allocated":278859776,"data_stored":278684310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,0,0,2,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":0.34300000000000003},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":8.4860000000000007},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.4820000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":8.8109999999999999},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.30199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278183936,"data_stored":278183936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279379968,"data_stored":279379968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278491136,"data_stored":278491136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":279093248,"data_stored":279093248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:51.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:51.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8881", 'reported_seq': 8980, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:46.411050+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:46.411050+0000', 'last_peered': '2024-09-18T11:50:46.411050+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:46.411050+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8881, 'log_dups_size': 0, 'ondisk_log_size': 8881, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35790848, 'num_objects': 4369, 'num_object_clones': 0, 'num_object_copies': 17476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8881, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2266}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:51.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:51.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8578", 'reported_seq': 8693, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:47.057356+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:47.057356+0000', 'last_peered': '2024-09-18T11:50:47.057356+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:47.057356+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8578, 'log_dups_size': 0, 'ondisk_log_size': 8578, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34650771, 'num_objects': 4231, 'num_object_clones': 0, 'num_object_copies': 16924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4231, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8577, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2191}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:51.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:51.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8724", 'reported_seq': 8814, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:47.444410+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:47.444410+0000', 'last_peered': '2024-09-18T11:50:47.444410+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:47.444410+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8724, 'log_dups_size': 0, 'ondisk_log_size': 8724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34419958, 'num_objects': 4202, 'num_object_clones': 0, 'num_object_copies': 16808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8722, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2100}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:51.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:51.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8657", 'reported_seq': 8776, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:46.456706+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:46.456706+0000', 'last_peered': '2024-09-18T11:50:46.456706+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:46.456706+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8657, 'log_dups_size': 0, 'ondisk_log_size': 8657, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34714870, 'num_objects': 4238, 'num_object_clones': 0, 'num_object_copies': 16952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8655, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:51.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:51.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8667", 'reported_seq': 8783, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:46.532728+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:46.532728+0000', 'last_peered': '2024-09-18T11:50:46.532728+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:46.532728+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8667, 'log_dups_size': 0, 'ondisk_log_size': 8667, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33742848, 'num_objects': 4119, 'num_object_clones': 0, 'num_object_copies': 16476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8667, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2000}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:51.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:51.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8770", 'reported_seq': 8875, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:47.576554+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:47.576554+0000', 'last_peered': '2024-09-18T11:50:47.576554+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:47.576554+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8770, 'log_dups_size': 0, 'ondisk_log_size': 8770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34780406, 'num_objects': 4246, 'num_object_clones': 0, 'num_object_copies': 16984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4246, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8768, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2130}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:51.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:51.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8611", 'reported_seq': 8718, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:47.481700+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:47.481700+0000', 'last_peered': '2024-09-18T11:50:47.481700+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:47.481700+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8611, 'log_dups_size': 0, 'ondisk_log_size': 8611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34927862, 'num_objects': 4264, 'num_object_clones': 0, 'num_object_copies': 17056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8609, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:51.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8852", 'reported_seq': 10757, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:46.552486+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:46.552486+0000', 'last_peered': '2024-09-18T11:50:46.552486+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:46.552486+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8852, 'log_dups_size': 0, 'ondisk_log_size': 8852, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35416556, 'num_objects': 4324, 'num_object_clones': 0, 'num_object_copies': 17296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1217, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8849, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 924, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3107}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8400", 'reported_seq': 8511, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:47.226169+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:47.226169+0000', 'last_peered': '2024-09-18T11:50:47.226169+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:47.226169+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8400, 'log_dups_size': 0, 'ondisk_log_size': 8400, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34354422, 'num_objects': 4194, 'num_object_clones': 0, 'num_object_copies': 16776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8398, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8709", 'reported_seq': 8827, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:46.479698+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:46.479698+0000', 'last_peered': '2024-09-18T11:50:46.479698+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:46.479698+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8709, 'log_dups_size': 0, 'ondisk_log_size': 8709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35417969, 'num_objects': 4324, 'num_object_clones': 0, 'num_object_copies': 17296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8706, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2290}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8751", 'reported_seq': 8842, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:46.537047+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:46.537047+0000', 'last_peered': '2024-09-18T11:50:46.537047+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:46.537047+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8751, 'log_dups_size': 0, 'ondisk_log_size': 8751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35060347, 'num_objects': 4280, 'num_object_clones': 0, 'num_object_copies': 17120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8750, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8676", 'reported_seq': 8766, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:47.125584+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:47.125584+0000', 'last_peered': '2024-09-18T11:50:47.125584+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:47.125584+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8676, 'log_dups_size': 0, 'ondisk_log_size': 8676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35150459, 'num_objects': 4291, 'num_object_clones': 0, 'num_object_copies': 17164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8675, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2242}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:51.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8854", 'reported_seq': 8976, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:46.365801+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:46.365801+0000', 'last_peered': '2024-09-18T11:50:46.365801+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:46.365801+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8854, 'log_dups_size': 0, 'ondisk_log_size': 8854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34698486, 'num_objects': 4236, 'num_object_clones': 0, 'num_object_copies': 16944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8852, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2113}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:51.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:51.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8502", 'reported_seq': 8590, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:47.489273+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:47.489273+0000', 'last_peered': '2024-09-18T11:50:47.489273+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:47.489273+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8502, 'log_dups_size': 0, 'ondisk_log_size': 8502, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34444534, 'num_objects': 4205, 'num_object_clones': 0, 'num_object_copies': 16820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8500, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2203}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:51.360 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:52.055 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:52.056 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28038/272260 objects degraded (10.298%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"0 slow ops, oldest one blocked for 30 sec, osd.2 has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":450,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68065,"data_bytes":557540748,"bytes_used":2371522560,"bytes_avail":384175534080,"bytes_total":386547056640,"degraded_objects":28038,"degraded_total":272260,"degraded_ratio":0.1029824432527731,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:45:45.849315+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:52.075 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:52.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:52.611 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:52.612 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:52.641 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-18T11:50:51.984847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":552347020,"num_objects":67431,"num_object_clones":0,"num_object_copies":269724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139656,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5474,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139683,"ondisk_log_size":139683,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2303032,"kb_used_data":1083088,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375184328,"statfs":{"total":386547056640,"available":384188751872,"internally_reserved":0,"allocated":1109082112,"data_stored":1108380248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,8,6,10,9,0,0,0,0,0,0,0,0,4],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22790144,"num_objects":-2782,"num_object_clones":0,"num_object_copies":-11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2782,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003918"},"pg_stats":[{"pgid":"1.f","version":"35'8955","reported_seq":9054,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.470306+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:51.470306+0000","last_peered":"2024-09-18T11:50:51.470306+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:51.470306+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35184640,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8955,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8825","reported_seq":13115,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:51.409310+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:51.409310+0000","last_peered":"2024-09-18T11:50:51.409310+0000","last_clean":"2024-09-18T11:50:51.409310+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:51.409310+0000","last_undegraded":"2024-09-18T11:50:51.409310+0000","last_fullsized":"2024-09-18T11:50:51.409310+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8825,"log_dups_size":0,"ondisk_log_size":8825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34657526,"num_objects":4231,"num_object_clones":0,"num_object_copies":16924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8823,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8578","reported_seq":8693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.057356+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:47.057356+0000","last_peered":"2024-09-18T11:50:47.057356+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:47.057356+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8578,"log_dups_size":0,"ondisk_log_size":8578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34650771,"num_objects":4231,"num_object_clones":0,"num_object_copies":16924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8577,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2191}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8724","reported_seq":8814,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.444410+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:47.444410+0000","last_peered":"2024-09-18T11:50:47.444410+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:47.444410+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8724,"log_dups_size":0,"ondisk_log_size":8724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34419958,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8722,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8715","reported_seq":8833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.564075+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:51.564075+0000","last_peered":"2024-09-18T11:50:51.564075+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:51.564075+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34239734,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8732","reported_seq":8849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.622949+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:51.622949+0000","last_peered":"2024-09-18T11:50:51.622949+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:51.622949+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8732,"log_dups_size":0,"ondisk_log_size":8732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33210368,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8732,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8770","reported_seq":8875,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.576554+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:47.576554+0000","last_peered":"2024-09-18T11:50:47.576554+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:47.576554+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8770,"log_dups_size":0,"ondisk_log_size":8770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34780406,"num_objects":4246,"num_object_clones":0,"num_object_copies":16984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8768,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2130}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8611","reported_seq":8718,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.481700+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:47.481700+0000","last_peered":"2024-09-18T11:50:47.481700+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:47.481700+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34927862,"num_objects":4264,"num_object_clones":0,"num_object_copies":17056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8609,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8941","reported_seq":10957,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.589850+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:51.589850+0000","last_peered":"2024-09-18T11:50:51.589850+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:51.589850+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8941,"log_dups_size":0,"ondisk_log_size":8941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34687468,"num_objects":4235,"num_object_clones":0,"num_object_copies":16940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8938,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8400","reported_seq":8511,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.226169+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:47.226169+0000","last_peered":"2024-09-18T11:50:47.226169+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:47.226169+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8400,"log_dups_size":0,"ondisk_log_size":8400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34354422,"num_objects":4194,"num_object_clones":0,"num_object_copies":16776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8398,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8780","reported_seq":8899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.425229+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:51.425229+0000","last_peered":"2024-09-18T11:50:51.425229+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:51.425229+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8780,"log_dups_size":0,"ondisk_log_size":8780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34836337,"num_objects":4253,"num_object_clones":0,"num_object_copies":17012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8821","reported_seq":8912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.395494+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:51.395494+0000","last_peered":"2024-09-18T11:50:51.395494+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:51.395494+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8821,"log_dups_size":0,"ondisk_log_size":8821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34486907,"num_objects":4210,"num_object_clones":0,"num_object_copies":16840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8820,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8676","reported_seq":8766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.125584+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:47.125584+0000","last_peered":"2024-09-18T11:50:47.125584+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:47.125584+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8676,"log_dups_size":0,"ondisk_log_size":8676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35150459,"num_objects":4291,"num_object_clones":0,"num_object_copies":17164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8675,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8912","reported_seq":9035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.490607+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:51.490607+0000","last_peered":"2024-09-18T11:50:51.490607+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:51.490607+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34223350,"num_objects":4178,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2055}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8741","reported_seq":13028,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:51.570419+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:51.570419+0000","last_peered":"2024-09-18T11:50:51.570419+0000","last_clean":"2024-09-18T11:50:51.570419+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:51.570419+0000","last_undegraded":"2024-09-18T11:50:51.570419+0000","last_fullsized":"2024-09-18T11:50:51.570419+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8741,"log_dups_size":0,"ondisk_log_size":8741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34092278,"num_objects":4162,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8739,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8502","reported_seq":8590,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:47.489273+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:47.489273+0000","last_peered":"2024-09-18T11:50:47.489273+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:47.489273+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8502,"log_dups_size":0,"ondisk_log_size":8502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34444534,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8500,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2203}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":552347020,"num_objects":67431,"num_object_clones":0,"num_object_copies":269724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139656,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5474,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1106153472,"data_stored":1106153472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139683,"ondisk_log_size":139683,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570912,"kb_used_data":268888,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93800928,"statfs":{"total":96636764160,"available":96052150272,"internally_reserved":0,"allocated":275341312,"data_stored":275165846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.75}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":15.206},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.119}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":0.31900000000000001},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.6660000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578440,"kb_used_data":272760,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93793400,"statfs":{"total":96636764160,"available":96044441600,"internally_reserved":0,"allocated":279306240,"data_stored":279130774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,0,0,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.372}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":6.8650000000000002},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.9079999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.264}]}]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579396,"kb_used_data":269116,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93792444,"statfs":{"total":96636764160,"available":96043462656,"internally_reserved":0,"allocated":275574784,"data_stored":275399318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":6.4619999999999997},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.2690000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.71},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":6.5}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.2350000000000003},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":8.8510000000000009}]}]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574284,"kb_used_data":272324,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93797556,"statfs":{"total":96636764160,"available":96048697344,"internally_reserved":0,"allocated":278859776,"data_stored":278684310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,0,0,2,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":0.34300000000000003},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":8.4860000000000007},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":4.4820000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":8.8109999999999999},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.30199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278183936,"data_stored":278183936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274894848,"data_stored":274894848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278491136,"data_stored":278491136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274583552,"data_stored":274583552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:55.644 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:56.194 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:56.195 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:56.213 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-18T11:50:55.986121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":548586892,"num_objects":66972,"num_object_clones":0,"num_object_copies":267888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140115,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5475,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140142,"ondisk_log_size":140142,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2302584,"kb_used_data":1074320,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375184776,"statfs":{"total":386547056640,"available":384189210624,"internally_reserved":0,"allocated":1100103680,"data_stored":1099401816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,12,10,15,11,1,0,0,0,0,0,0,0,0,3],"upper_bound":131072},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17997824,"num_objects":-2197,"num_object_clones":0,"num_object_copies":-8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004236"},"pg_stats":[{"pgid":"1.f","version":"35'8955","reported_seq":9054,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.470306+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:51.470306+0000","last_peered":"2024-09-18T11:50:51.470306+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:51.470306+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35184640,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8955,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8825","reported_seq":13115,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:51.409310+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:51.409310+0000","last_peered":"2024-09-18T11:50:51.409310+0000","last_clean":"2024-09-18T11:50:51.409310+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:51.409310+0000","last_undegraded":"2024-09-18T11:50:51.409310+0000","last_fullsized":"2024-09-18T11:50:51.409310+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8825,"log_dups_size":0,"ondisk_log_size":8825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34657526,"num_objects":4231,"num_object_clones":0,"num_object_copies":16924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8823,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8645","reported_seq":8760,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.295769+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:52.295769+0000","last_peered":"2024-09-18T11:50:52.295769+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:52.295769+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8645,"log_dups_size":0,"ondisk_log_size":8645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34101907,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8644,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2124}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8787","reported_seq":8879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.666587+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:52.666587+0000","last_peered":"2024-09-18T11:50:52.666587+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:52.666587+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8787,"log_dups_size":0,"ondisk_log_size":8787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33903862,"num_objects":4139,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8785,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8715","reported_seq":8833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.564075+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:51.564075+0000","last_peered":"2024-09-18T11:50:51.564075+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:51.564075+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34239734,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8732","reported_seq":8849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.622949+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:51.622949+0000","last_peered":"2024-09-18T11:50:51.622949+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:51.622949+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8732,"log_dups_size":0,"ondisk_log_size":8732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33210368,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8732,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8845","reported_seq":8950,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.368445+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:52.368445+0000","last_peered":"2024-09-18T11:50:52.368445+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:52.368445+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34166006,"num_objects":4171,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8843,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2055}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8671","reported_seq":8777,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.631444+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:52.631444+0000","last_peered":"2024-09-18T11:50:52.631444+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:52.631444+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34436342,"num_objects":4204,"num_object_clones":0,"num_object_copies":16816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2190}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8941","reported_seq":10957,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.589850+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:51.589850+0000","last_peered":"2024-09-18T11:50:51.589850+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:51.589850+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8941,"log_dups_size":0,"ondisk_log_size":8941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34687468,"num_objects":4235,"num_object_clones":0,"num_object_copies":16940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8938,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8469","reported_seq":8581,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.216690+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:52.216690+0000","last_peered":"2024-09-18T11:50:52.216690+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:52.216690+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8469,"log_dups_size":0,"ondisk_log_size":8469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33789174,"num_objects":4125,"num_object_clones":0,"num_object_copies":16500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8467,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2162}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8780","reported_seq":8899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.425229+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:51.425229+0000","last_peered":"2024-09-18T11:50:51.425229+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:51.425229+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8780,"log_dups_size":0,"ondisk_log_size":8780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34836337,"num_objects":4253,"num_object_clones":0,"num_object_copies":17012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8821","reported_seq":8912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.395494+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:51.395494+0000","last_peered":"2024-09-18T11:50:51.395494+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:51.395494+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8821,"log_dups_size":0,"ondisk_log_size":8821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34486907,"num_objects":4210,"num_object_clones":0,"num_object_copies":16840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8820,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8743","reported_seq":8833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.151222+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:52.151222+0000","last_peered":"2024-09-18T11:50:52.151222+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:52.151222+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8743,"log_dups_size":0,"ondisk_log_size":8743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34601595,"num_objects":4224,"num_object_clones":0,"num_object_copies":16896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8912","reported_seq":9035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.490607+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:51.490607+0000","last_peered":"2024-09-18T11:50:51.490607+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:51.490607+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34223350,"num_objects":4178,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2055}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8741","reported_seq":13028,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:51.570419+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:51.570419+0000","last_peered":"2024-09-18T11:50:51.570419+0000","last_clean":"2024-09-18T11:50:51.570419+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:51.570419+0000","last_undegraded":"2024-09-18T11:50:51.570419+0000","last_fullsized":"2024-09-18T11:50:51.570419+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8741,"log_dups_size":0,"ondisk_log_size":8741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34092278,"num_objects":4162,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8739,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8560","reported_seq":8648,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.630254+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:52.630254+0000","last_peered":"2024-09-18T11:50:52.630254+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:52.630254+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8560,"log_dups_size":0,"ondisk_log_size":8560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33969398,"num_objects":4147,"num_object_clones":0,"num_object_copies":16588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8558,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":548586892,"num_objects":66972,"num_object_clones":0,"num_object_copies":267888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140115,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5475,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097199616,"data_stored":1097199616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140142,"ondisk_log_size":140142,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570912,"kb_used_data":268888,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93800928,"statfs":{"total":96636764160,"available":96052150272,"internally_reserved":0,"allocated":275341312,"data_stored":275165846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.75}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":15.206},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.119}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":0.31900000000000001},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.6660000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578200,"kb_used_data":268360,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93793640,"statfs":{"total":96636764160,"available":96044687360,"internally_reserved":0,"allocated":274800640,"data_stored":274625174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,4,2,1,0,0,0,0,0,0,0,0,3],"upper_bound":131072},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":8.0120000000000005},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":8.0449999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":10.377000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.1579999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.3949999999999996},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":6.8499999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579396,"kb_used_data":269116,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93792444,"statfs":{"total":96636764160,"available":96043462656,"internally_reserved":0,"allocated":275574784,"data_stored":275399318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":6.4619999999999997},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.2690000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.71},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":6.5}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.2350000000000003},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":8.8510000000000009}]}]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574076,"kb_used_data":267956,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93797764,"statfs":{"total":96636764160,"available":96048910336,"internally_reserved":0,"allocated":274386944,"data_stored":274211478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":7.3330000000000002},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.6210000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.075},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":21.885000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":6.9889999999999999},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":6.593}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":273694720,"data_stored":273694720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274894848,"data_stored":274894848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274026496,"data_stored":274026496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274583552,"data_stored":274583552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:56.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:56.724 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:56.724 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:56.755 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-18T11:50:55.986121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":548586892,"num_objects":66972,"num_object_clones":0,"num_object_copies":267888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140115,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5475,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140142,"ondisk_log_size":140142,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2302584,"kb_used_data":1074320,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375184776,"statfs":{"total":386547056640,"available":384189210624,"internally_reserved":0,"allocated":1100103680,"data_stored":1099401816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,12,10,15,11,1,0,0,0,0,0,0,0,0,3],"upper_bound":131072},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17997824,"num_objects":-2197,"num_object_clones":0,"num_object_copies":-8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004236"},"pg_stats":[{"pgid":"1.f","version":"35'8955","reported_seq":9054,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.470306+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:51.470306+0000","last_peered":"2024-09-18T11:50:51.470306+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:51.470306+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35184640,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8955,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8825","reported_seq":13115,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:51.409310+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:51.409310+0000","last_peered":"2024-09-18T11:50:51.409310+0000","last_clean":"2024-09-18T11:50:51.409310+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:51.409310+0000","last_undegraded":"2024-09-18T11:50:51.409310+0000","last_fullsized":"2024-09-18T11:50:51.409310+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8825,"log_dups_size":0,"ondisk_log_size":8825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34657526,"num_objects":4231,"num_object_clones":0,"num_object_copies":16924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8823,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8645","reported_seq":8760,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.295769+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:52.295769+0000","last_peered":"2024-09-18T11:50:52.295769+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:52.295769+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8645,"log_dups_size":0,"ondisk_log_size":8645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34101907,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8644,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2124}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8787","reported_seq":8879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.666587+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:52.666587+0000","last_peered":"2024-09-18T11:50:52.666587+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:52.666587+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8787,"log_dups_size":0,"ondisk_log_size":8787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33903862,"num_objects":4139,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8785,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8715","reported_seq":8833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.564075+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:51.564075+0000","last_peered":"2024-09-18T11:50:51.564075+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:51.564075+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34239734,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8732","reported_seq":8849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.622949+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:51.622949+0000","last_peered":"2024-09-18T11:50:51.622949+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:51.622949+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8732,"log_dups_size":0,"ondisk_log_size":8732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33210368,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8732,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8845","reported_seq":8950,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.368445+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:52.368445+0000","last_peered":"2024-09-18T11:50:52.368445+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:52.368445+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34166006,"num_objects":4171,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8843,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2055}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8671","reported_seq":8777,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.631444+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:52.631444+0000","last_peered":"2024-09-18T11:50:52.631444+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:52.631444+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34436342,"num_objects":4204,"num_object_clones":0,"num_object_copies":16816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2190}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8941","reported_seq":10957,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.589850+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:51.589850+0000","last_peered":"2024-09-18T11:50:51.589850+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:51.589850+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8941,"log_dups_size":0,"ondisk_log_size":8941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34687468,"num_objects":4235,"num_object_clones":0,"num_object_copies":16940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8938,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8469","reported_seq":8581,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.216690+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:52.216690+0000","last_peered":"2024-09-18T11:50:52.216690+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:52.216690+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8469,"log_dups_size":0,"ondisk_log_size":8469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33789174,"num_objects":4125,"num_object_clones":0,"num_object_copies":16500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8467,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2162}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8780","reported_seq":8899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.425229+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:51.425229+0000","last_peered":"2024-09-18T11:50:51.425229+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:51.425229+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8780,"log_dups_size":0,"ondisk_log_size":8780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34836337,"num_objects":4253,"num_object_clones":0,"num_object_copies":17012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8821","reported_seq":8912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.395494+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:51.395494+0000","last_peered":"2024-09-18T11:50:51.395494+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:51.395494+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8821,"log_dups_size":0,"ondisk_log_size":8821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34486907,"num_objects":4210,"num_object_clones":0,"num_object_copies":16840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8820,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8743","reported_seq":8833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.151222+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:52.151222+0000","last_peered":"2024-09-18T11:50:52.151222+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:52.151222+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8743,"log_dups_size":0,"ondisk_log_size":8743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34601595,"num_objects":4224,"num_object_clones":0,"num_object_copies":16896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8912","reported_seq":9035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:51.490607+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:51.490607+0000","last_peered":"2024-09-18T11:50:51.490607+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:51.490607+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34223350,"num_objects":4178,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2055}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8741","reported_seq":13028,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:51.570419+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:51.570419+0000","last_peered":"2024-09-18T11:50:51.570419+0000","last_clean":"2024-09-18T11:50:51.570419+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:51.570419+0000","last_undegraded":"2024-09-18T11:50:51.570419+0000","last_fullsized":"2024-09-18T11:50:51.570419+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8741,"log_dups_size":0,"ondisk_log_size":8741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34092278,"num_objects":4162,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8739,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8560","reported_seq":8648,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:52.630254+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:52.630254+0000","last_peered":"2024-09-18T11:50:52.630254+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:52.630254+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8560,"log_dups_size":0,"ondisk_log_size":8560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33969398,"num_objects":4147,"num_object_clones":0,"num_object_copies":16588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8558,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":548586892,"num_objects":66972,"num_object_clones":0,"num_object_copies":267888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140115,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5475,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097199616,"data_stored":1097199616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140142,"ondisk_log_size":140142,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570912,"kb_used_data":268888,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93800928,"statfs":{"total":96636764160,"available":96052150272,"internally_reserved":0,"allocated":275341312,"data_stored":275165846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.75}]},{"osd":1,"last update":"Wed Sep 18 11:49:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4329999999999998,"5min":5.6859999999999999,"15min":5.5499999999999998},"min":{"1min":0.45500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":18.718,"5min":21.501000000000001,"15min":21.501000000000001},"last":15.206},{"interface":"front","average":{"1min":5.6459999999999999,"5min":4.6970000000000001,"15min":4.3650000000000002},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":25.573,"5min":25.573,"15min":25.573},"last":1.119}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":0.31900000000000001},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.6660000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578200,"kb_used_data":268360,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93793640,"statfs":{"total":96636764160,"available":96044687360,"internally_reserved":0,"allocated":274800640,"data_stored":274625174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,4,2,1,0,0,0,0,0,0,0,0,3],"upper_bound":131072},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":8.0120000000000005},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":8.0449999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:49:54 2024","interfaces":[{"interface":"back","average":{"1min":6.6580000000000004,"5min":5.9909999999999997,"15min":5.9500000000000002},"min":{"1min":0.48899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":21.073,"5min":21.626999999999999,"15min":21.626999999999999},"last":10.377000000000001},{"interface":"front","average":{"1min":5.7889999999999997,"5min":4.4729999999999999,"15min":3.9279999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":19.789999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.1579999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.3949999999999996},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":6.8499999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579396,"kb_used_data":269116,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93792444,"statfs":{"total":96636764160,"available":96043462656,"internally_reserved":0,"allocated":275574784,"data_stored":275399318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":6.4619999999999997},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.2690000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.71},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":6.5}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.2350000000000003},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":8.8510000000000009}]}]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574076,"kb_used_data":267956,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93797764,"statfs":{"total":96636764160,"available":96048910336,"internally_reserved":0,"allocated":274386944,"data_stored":274211478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:49:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9249999999999998,"5min":4.569,"15min":3.9769999999999999},"min":{"1min":0.20499999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":26.189,"5min":26.189,"15min":26.189},"last":7.3330000000000002},{"interface":"front","average":{"1min":6.0949999999999998,"5min":4.6310000000000002,"15min":4.0510000000000002},"min":{"1min":0.435,"5min":0.185,"15min":0.185},"max":{"1min":16.082000000000001,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.6210000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.075},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":21.885000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":6.9889999999999999},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":6.593}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":273694720,"data_stored":273694720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274894848,"data_stored":274894848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274026496,"data_stored":274026496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274583552,"data_stored":274583552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:50:56.757 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:50:56.757 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8955", 'reported_seq': 9054, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:51.470306+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:50:51.470306+0000', 'last_peered': '2024-09-18T11:50:51.470306+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:50:51.470306+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8955, 'log_dups_size': 0, 'ondisk_log_size': 8955, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35184640, 'num_objects': 4295, 'num_object_clones': 0, 'num_object_copies': 17180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8955, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2192}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:56.757 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:50:56.757 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8645", 'reported_seq': 8760, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:52.295769+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:52.295769+0000', 'last_peered': '2024-09-18T11:50:52.295769+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:52.295769+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8645, 'log_dups_size': 0, 'ondisk_log_size': 8645, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34101907, 'num_objects': 4164, 'num_object_clones': 0, 'num_object_copies': 16656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4164, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8644, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2124}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:56.757 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:50:56.757 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8787", 'reported_seq': 8879, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:52.666587+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:52.666587+0000', 'last_peered': '2024-09-18T11:50:52.666587+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:52.666587+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8787, 'log_dups_size': 0, 'ondisk_log_size': 8787, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33903862, 'num_objects': 4139, 'num_object_clones': 0, 'num_object_copies': 16556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8785, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:56.757 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8715", 'reported_seq': 8833, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:51.564075+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:50:51.564075+0000', 'last_peered': '2024-09-18T11:50:51.564075+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:50:51.564075+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8715, 'log_dups_size': 0, 'ondisk_log_size': 8715, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34239734, 'num_objects': 4180, 'num_object_clones': 0, 'num_object_copies': 16720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8713, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2128}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8732", 'reported_seq': 8849, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:51.622949+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:50:51.622949+0000', 'last_peered': '2024-09-18T11:50:51.622949+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:50:51.622949+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8732, 'log_dups_size': 0, 'ondisk_log_size': 8732, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33210368, 'num_objects': 4054, 'num_object_clones': 0, 'num_object_copies': 16216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8732, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1935}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8845", 'reported_seq': 8950, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:52.368445+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:52.368445+0000', 'last_peered': '2024-09-18T11:50:52.368445+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:52.368445+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8845, 'log_dups_size': 0, 'ondisk_log_size': 8845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34166006, 'num_objects': 4171, 'num_object_clones': 0, 'num_object_copies': 16684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8843, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2055}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8671", 'reported_seq': 8777, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:52.631444+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:52.631444+0000', 'last_peered': '2024-09-18T11:50:52.631444+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:52.631444+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8671, 'log_dups_size': 0, 'ondisk_log_size': 8671, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34436342, 'num_objects': 4204, 'num_object_clones': 0, 'num_object_copies': 16816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8669, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2190}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8941", 'reported_seq': 10957, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:51.589850+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:50:51.589850+0000', 'last_peered': '2024-09-18T11:50:51.589850+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:50:51.589850+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8941, 'log_dups_size': 0, 'ondisk_log_size': 8941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34687468, 'num_objects': 4235, 'num_object_clones': 0, 'num_object_copies': 16940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1161, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8938, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 980, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3074}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:50:56.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8469", 'reported_seq': 8581, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:52.216690+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:52.216690+0000', 'last_peered': '2024-09-18T11:50:52.216690+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:52.216690+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8469, 'log_dups_size': 0, 'ondisk_log_size': 8469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33789174, 'num_objects': 4125, 'num_object_clones': 0, 'num_object_copies': 16500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8467, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2162}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:56.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:50:56.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8780", 'reported_seq': 8899, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:51.425229+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:50:51.425229+0000', 'last_peered': '2024-09-18T11:50:51.425229+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:50:51.425229+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8780, 'log_dups_size': 0, 'ondisk_log_size': 8780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34836337, 'num_objects': 4253, 'num_object_clones': 0, 'num_object_copies': 17012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8777, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:56.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:50:56.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8821", 'reported_seq': 8912, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:51.395494+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:50:51.395494+0000', 'last_peered': '2024-09-18T11:50:51.395494+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:50:51.395494+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8821, 'log_dups_size': 0, 'ondisk_log_size': 8821, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34486907, 'num_objects': 4210, 'num_object_clones': 0, 'num_object_copies': 16840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8820, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:56.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:50:56.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8743", 'reported_seq': 8833, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:52.151222+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:52.151222+0000', 'last_peered': '2024-09-18T11:50:52.151222+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:52.151222+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8743, 'log_dups_size': 0, 'ondisk_log_size': 8743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34601595, 'num_objects': 4224, 'num_object_clones': 0, 'num_object_copies': 16896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8742, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2175}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:50:56.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:50:56.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8912", 'reported_seq': 9035, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:51.490607+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:50:51.490607+0000', 'last_peered': '2024-09-18T11:50:51.490607+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:50:51.490607+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8912, 'log_dups_size': 0, 'ondisk_log_size': 8912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34223350, 'num_objects': 4178, 'num_object_clones': 0, 'num_object_copies': 16712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8910, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2055}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:50:56.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:50:56.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8560", 'reported_seq': 8648, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:52.630254+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:52.630254+0000', 'last_peered': '2024-09-18T11:50:52.630254+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:52.630254+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8560, 'log_dups_size': 0, 'ondisk_log_size': 8560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33969398, 'num_objects': 4147, 'num_object_clones': 0, 'num_object_copies': 16588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8558, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2145}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:50:56.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:50:57.458 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:57.458 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27981/267888 objects degraded (10.445%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"0 slow ops, oldest one blocked for 35 sec, osd.2 has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":455,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66972,"data_bytes":548586892,"bytes_used":2357846016,"bytes_avail":384189210624,"bytes_total":386547056640,"degraded_objects":27981,"degraded_total":267888,"degraded_ratio":0.10445036731768501,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":183},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:50:57.474 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:50:57.475 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:50:58.046 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:50:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:50:58.081 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-18T11:50:57.986667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539239820,"num_objects":65831,"num_object_clones":0,"num_object_copies":263324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141256,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5535,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141283,"ondisk_log_size":141283,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2263868,"kb_used_data":1056412,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375223492,"statfs":{"total":386547056640,"available":384228855808,"internally_reserved":0,"allocated":1081765888,"data_stored":1081064024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,2,3,15,12,10,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27344896,"num_objects":-3338,"num_object_clones":0,"num_object_copies":-13352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3338,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004120"},"pg_stats":[{"pgid":"1.f","version":"35'9024","reported_seq":9123,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.442476+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:56.442476+0000","last_peered":"2024-09-18T11:50:56.442476+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:56.442476+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34619392,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9024,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8924","reported_seq":13214,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:56.839529+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:56.839529+0000","last_peered":"2024-09-18T11:50:56.839529+0000","last_clean":"2024-09-18T11:50:56.839529+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:56.839529+0000","last_undegraded":"2024-09-18T11:50:56.839529+0000","last_fullsized":"2024-09-18T11:50:56.839529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8924,"log_dups_size":0,"ondisk_log_size":8924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33846518,"num_objects":4132,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8922,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8715","reported_seq":8830,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.244065+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:57.244065+0000","last_peered":"2024-09-18T11:50:57.244065+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:57.244065+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528467,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8714,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8865","reported_seq":8956,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.749255+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:57.749255+0000","last_peered":"2024-09-18T11:50:57.749255+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:57.749255+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264886,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1960}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8779","reported_seq":8898,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.556540+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:56.556540+0000","last_peered":"2024-09-18T11:50:56.556540+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:56.556540+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8779,"log_dups_size":0,"ondisk_log_size":8779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33715446,"num_objects":4116,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8800","reported_seq":8917,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.560584+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:56.560584+0000","last_peered":"2024-09-18T11:50:56.560584+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:56.560584+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8800,"log_dups_size":0,"ondisk_log_size":8800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32653312,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8800,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1867}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8920","reported_seq":9025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.528244+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:57.528244+0000","last_peered":"2024-09-18T11:50:57.528244+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:57.528244+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8920,"log_dups_size":0,"ondisk_log_size":8920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551606,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1980}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8729","reported_seq":8835,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.615707+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:57.615707+0000","last_peered":"2024-09-18T11:50:57.615707+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:57.615707+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8729,"log_dups_size":0,"ondisk_log_size":8729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33961206,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8727,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9006","reported_seq":11136,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.618848+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:56.618848+0000","last_peered":"2024-09-18T11:50:56.618848+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:56.618848+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34154988,"num_objects":4170,"num_object_clones":0,"num_object_copies":16680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1037,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8541","reported_seq":8653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.186848+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:57.186848+0000","last_peered":"2024-09-18T11:50:57.186848+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:57.186848+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33199350,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8863","reported_seq":8982,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.350525+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:56.350525+0000","last_peered":"2024-09-18T11:50:56.350525+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:56.350525+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8863,"log_dups_size":0,"ondisk_log_size":8863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34156401,"num_objects":4170,"num_object_clones":0,"num_object_copies":16680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8860,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8893","reported_seq":8983,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.582241+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:56.582241+0000","last_peered":"2024-09-18T11:50:56.582241+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:56.582241+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8893,"log_dups_size":0,"ondisk_log_size":8893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33897083,"num_objects":4138,"num_object_clones":0,"num_object_copies":16552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8892,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2054}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8816","reported_seq":8912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.267436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:57.267436+0000","last_peered":"2024-09-18T11:50:57.267436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:57.267436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8816,"log_dups_size":0,"ondisk_log_size":8816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34003579,"num_objects":4151,"num_object_clones":0,"num_object_copies":16604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8963","reported_seq":9086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.351616+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:56.351616+0000","last_peered":"2024-09-18T11:50:56.351616+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:56.351616+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8963,"log_dups_size":0,"ondisk_log_size":8963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33805558,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8961,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2004}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8817","reported_seq":13104,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:56.920548+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:56.920548+0000","last_peered":"2024-09-18T11:50:56.920548+0000","last_clean":"2024-09-18T11:50:56.920548+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:56.920548+0000","last_undegraded":"2024-09-18T11:50:56.920548+0000","last_fullsized":"2024-09-18T11:50:56.920548+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8817,"log_dups_size":0,"ondisk_log_size":8817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33469686,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8628","reported_seq":8716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.278969+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:57.278969+0000","last_peered":"2024-09-18T11:50:57.278969+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:57.278969+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8628,"log_dups_size":0,"ondisk_log_size":8628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33412342,"num_objects":4079,"num_object_clones":0,"num_object_copies":16316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8626,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":539239820,"num_objects":65831,"num_object_clones":0,"num_object_copies":263324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141256,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5535,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1078562816,"data_stored":1078562816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141283,"ondisk_log_size":141283,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566540,"kb_used_data":264516,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93805300,"statfs":{"total":96636764160,"available":96056627200,"internally_reserved":0,"allocated":270864384,"data_stored":270688918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,1,7,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":9.8179999999999996},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":9.782}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.425000000000001},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":21.395}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":11.705},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":21.431999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561300,"kb_used_data":263948,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93810540,"statfs":{"total":96636764160,"available":96061992960,"internally_reserved":0,"allocated":270282752,"data_stored":270107286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.5880000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.8769999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.9280000000000008},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":8.8290000000000006}]}]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562388,"kb_used_data":264588,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93809452,"statfs":{"total":96636764160,"available":96060878848,"internally_reserved":0,"allocated":270938112,"data_stored":270762646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,10,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":7.3600000000000003},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":8.8640000000000008},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.101}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.8559999999999999},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.0069999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573640,"kb_used_data":263360,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93798200,"statfs":{"total":96636764160,"available":96049356800,"internally_reserved":0,"allocated":269680640,"data_stored":269505174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,6,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":6.6600000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.7060000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":5.6420000000000003},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.98599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.9020000000000001},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":269029376,"data_stored":269029376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270143488,"data_stored":270143488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269406208,"data_stored":269406208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":269983744,"data_stored":269983744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:01.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:01.611 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:01.611 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:01.630 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-18T11:50:59.987291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539239820,"num_objects":65831,"num_object_clones":0,"num_object_copies":263324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141256,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5535,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141283,"ondisk_log_size":141283,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2263868,"kb_used_data":1056412,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375223492,"statfs":{"total":386547056640,"available":384228855808,"internally_reserved":0,"allocated":1081765888,"data_stored":1081064024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,2,3,15,12,10,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18300928,"num_objects":-2234,"num_object_clones":0,"num_object_copies":-8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2234,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004015"},"pg_stats":[{"pgid":"1.f","version":"35'9024","reported_seq":9123,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.442476+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:50:56.442476+0000","last_peered":"2024-09-18T11:50:56.442476+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:50:56.442476+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34619392,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9024,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8924","reported_seq":13214,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:56.839529+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:56.839529+0000","last_peered":"2024-09-18T11:50:56.839529+0000","last_clean":"2024-09-18T11:50:56.839529+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:56.839529+0000","last_undegraded":"2024-09-18T11:50:56.839529+0000","last_fullsized":"2024-09-18T11:50:56.839529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8924,"log_dups_size":0,"ondisk_log_size":8924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33846518,"num_objects":4132,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8922,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8715","reported_seq":8830,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.244065+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:57.244065+0000","last_peered":"2024-09-18T11:50:57.244065+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:57.244065+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528467,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8714,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8865","reported_seq":8956,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.749255+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:57.749255+0000","last_peered":"2024-09-18T11:50:57.749255+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:57.749255+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264886,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1960}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8779","reported_seq":8898,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.556540+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:50:56.556540+0000","last_peered":"2024-09-18T11:50:56.556540+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:50:56.556540+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8779,"log_dups_size":0,"ondisk_log_size":8779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33715446,"num_objects":4116,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8800","reported_seq":8917,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.560584+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:50:56.560584+0000","last_peered":"2024-09-18T11:50:56.560584+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:50:56.560584+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8800,"log_dups_size":0,"ondisk_log_size":8800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32653312,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8800,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1867}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8920","reported_seq":9025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.528244+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:57.528244+0000","last_peered":"2024-09-18T11:50:57.528244+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:57.528244+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8920,"log_dups_size":0,"ondisk_log_size":8920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551606,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1980}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8729","reported_seq":8835,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.615707+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:57.615707+0000","last_peered":"2024-09-18T11:50:57.615707+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:57.615707+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8729,"log_dups_size":0,"ondisk_log_size":8729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33961206,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8727,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9006","reported_seq":11136,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.618848+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:50:56.618848+0000","last_peered":"2024-09-18T11:50:56.618848+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:50:56.618848+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34154988,"num_objects":4170,"num_object_clones":0,"num_object_copies":16680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1037,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8541","reported_seq":8653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.186848+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:57.186848+0000","last_peered":"2024-09-18T11:50:57.186848+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:57.186848+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33199350,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8863","reported_seq":8982,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.350525+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:50:56.350525+0000","last_peered":"2024-09-18T11:50:56.350525+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:50:56.350525+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8863,"log_dups_size":0,"ondisk_log_size":8863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34156401,"num_objects":4170,"num_object_clones":0,"num_object_copies":16680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8860,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8893","reported_seq":8983,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.582241+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:50:56.582241+0000","last_peered":"2024-09-18T11:50:56.582241+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:50:56.582241+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8893,"log_dups_size":0,"ondisk_log_size":8893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33897083,"num_objects":4138,"num_object_clones":0,"num_object_copies":16552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8892,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2054}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8816","reported_seq":8912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.267436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:57.267436+0000","last_peered":"2024-09-18T11:50:57.267436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:57.267436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8816,"log_dups_size":0,"ondisk_log_size":8816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34003579,"num_objects":4151,"num_object_clones":0,"num_object_copies":16604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8963","reported_seq":9086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:56.351616+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:50:56.351616+0000","last_peered":"2024-09-18T11:50:56.351616+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:50:56.351616+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8963,"log_dups_size":0,"ondisk_log_size":8963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33805558,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8961,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2004}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8817","reported_seq":13104,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:56.920548+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:56.920548+0000","last_peered":"2024-09-18T11:50:56.920548+0000","last_clean":"2024-09-18T11:50:56.920548+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:56.920548+0000","last_undegraded":"2024-09-18T11:50:56.920548+0000","last_fullsized":"2024-09-18T11:50:56.920548+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8817,"log_dups_size":0,"ondisk_log_size":8817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33469686,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8628","reported_seq":8716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.278969+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:57.278969+0000","last_peered":"2024-09-18T11:50:57.278969+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:57.278969+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8628,"log_dups_size":0,"ondisk_log_size":8628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33412342,"num_objects":4079,"num_object_clones":0,"num_object_copies":16316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8626,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":539239820,"num_objects":65831,"num_object_clones":0,"num_object_copies":263324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141256,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5535,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1078562816,"data_stored":1078562816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141283,"ondisk_log_size":141283,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566540,"kb_used_data":264516,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93805300,"statfs":{"total":96636764160,"available":96056627200,"internally_reserved":0,"allocated":270864384,"data_stored":270688918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,1,7,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":9.8179999999999996},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":9.782}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.425000000000001},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":21.395}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":11.705},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":21.431999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561300,"kb_used_data":263948,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93810540,"statfs":{"total":96636764160,"available":96061992960,"internally_reserved":0,"allocated":270282752,"data_stored":270107286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.5880000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.8769999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.9280000000000008},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":8.8290000000000006}]}]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562388,"kb_used_data":264588,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93809452,"statfs":{"total":96636764160,"available":96060878848,"internally_reserved":0,"allocated":270938112,"data_stored":270762646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,10,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":7.3600000000000003},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":8.8640000000000008},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.101}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.8559999999999999},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.0069999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573640,"kb_used_data":263360,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93798200,"statfs":{"total":96636764160,"available":96049356800,"internally_reserved":0,"allocated":269680640,"data_stored":269505174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,6,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":6.6600000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.7060000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":5.6420000000000003},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.98599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.9020000000000001},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":269029376,"data_stored":269029376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270143488,"data_stored":270143488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269406208,"data_stored":269406208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":269983744,"data_stored":269983744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:01.632 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:02.202 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:02.203 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:02.228 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-18T11:51:01.987727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":535233932,"num_objects":65342,"num_object_clones":0,"num_object_copies":261368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65342,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141745,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5594,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141772,"ondisk_log_size":141772,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2263420,"kb_used_data":1051772,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375223940,"statfs":{"total":386547056640,"available":384229314560,"internally_reserved":0,"allocated":1077014528,"data_stored":1076312664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,8,16,13,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22306816,"num_objects":-2723,"num_object_clones":0,"num_object_copies":-10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2723,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003529"},"pg_stats":[{"pgid":"1.f","version":"35'9104","reported_seq":9202,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.696673+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:01.696673+0000","last_peered":"2024-09-18T11:51:01.696673+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:01.696673+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9104,"log_dups_size":0,"ondisk_log_size":9104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33964032,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9104,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2043}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8924","reported_seq":13214,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:56.839529+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:56.839529+0000","last_peered":"2024-09-18T11:50:56.839529+0000","last_clean":"2024-09-18T11:50:56.839529+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:56.839529+0000","last_undegraded":"2024-09-18T11:50:56.839529+0000","last_fullsized":"2024-09-18T11:50:56.839529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8924,"log_dups_size":0,"ondisk_log_size":8924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33846518,"num_objects":4132,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8922,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8715","reported_seq":8830,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.244065+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:57.244065+0000","last_peered":"2024-09-18T11:50:57.244065+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:57.244065+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528467,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8714,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8865","reported_seq":8956,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.749255+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:57.749255+0000","last_peered":"2024-09-18T11:50:57.749255+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:57.749255+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264886,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1960}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8856","reported_seq":8975,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.551745+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:01.551745+0000","last_peered":"2024-09-18T11:51:01.551745+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:01.551745+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8856,"log_dups_size":0,"ondisk_log_size":8856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33084662,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8845","reported_seq":8962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.377111+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:01.377111+0000","last_peered":"2024-09-18T11:51:01.377111+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:01.377111+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32284672,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8845,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8920","reported_seq":9025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.528244+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:57.528244+0000","last_peered":"2024-09-18T11:50:57.528244+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:57.528244+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8920,"log_dups_size":0,"ondisk_log_size":8920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551606,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1980}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8729","reported_seq":8835,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.615707+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:57.615707+0000","last_peered":"2024-09-18T11:50:57.615707+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:57.615707+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8729,"log_dups_size":0,"ondisk_log_size":8729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33961206,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8727,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9069","reported_seq":11318,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.637537+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:01.637537+0000","last_peered":"2024-09-18T11:51:01.637537+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:01.637537+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9069,"log_dups_size":0,"ondisk_log_size":9069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33638892,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1096,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8541","reported_seq":8653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.186848+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:57.186848+0000","last_peered":"2024-09-18T11:50:57.186848+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:57.186848+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33199350,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8940","reported_seq":9058,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.685151+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:01.685151+0000","last_peered":"2024-09-18T11:51:01.685151+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:01.685151+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8940,"log_dups_size":0,"ondisk_log_size":8940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33525617,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8937,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8965","reported_seq":9055,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.623622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:01.623622+0000","last_peered":"2024-09-18T11:51:01.623622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:01.623622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8965,"log_dups_size":0,"ondisk_log_size":8965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33307259,"num_objects":4066,"num_object_clones":0,"num_object_copies":16264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8964,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8816","reported_seq":8912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.267436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:57.267436+0000","last_peered":"2024-09-18T11:50:57.267436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:57.267436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8816,"log_dups_size":0,"ondisk_log_size":8816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34003579,"num_objects":4151,"num_object_clones":0,"num_object_copies":16604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9038","reported_seq":9161,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.549626+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:01.549626+0000","last_peered":"2024-09-18T11:51:01.549626+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:01.549626+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9038,"log_dups_size":0,"ondisk_log_size":9038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33191158,"num_objects":4052,"num_object_clones":0,"num_object_copies":16208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1929}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8817","reported_seq":13104,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:56.920548+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:56.920548+0000","last_peered":"2024-09-18T11:50:56.920548+0000","last_clean":"2024-09-18T11:50:56.920548+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:56.920548+0000","last_undegraded":"2024-09-18T11:50:56.920548+0000","last_fullsized":"2024-09-18T11:50:56.920548+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8817,"log_dups_size":0,"ondisk_log_size":8817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33469686,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8628","reported_seq":8716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.278969+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:57.278969+0000","last_peered":"2024-09-18T11:50:57.278969+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:57.278969+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8628,"log_dups_size":0,"ondisk_log_size":8628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33412342,"num_objects":4079,"num_object_clones":0,"num_object_copies":16316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8626,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":535233932,"num_objects":65342,"num_object_clones":0,"num_object_copies":261368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65342,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141745,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5594,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073934336,"data_stored":1073934336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141772,"ondisk_log_size":141772,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566092,"kb_used_data":259876,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93805748,"statfs":{"total":96636764160,"available":96057085952,"internally_reserved":0,"allocated":266113024,"data_stored":265937558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.523},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.0510000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.3410000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":0.66600000000000004},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.0660000000000007}]}]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561300,"kb_used_data":263948,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93810540,"statfs":{"total":96636764160,"available":96061992960,"internally_reserved":0,"allocated":270282752,"data_stored":270107286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.5880000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.8769999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.9280000000000008},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":8.8290000000000006}]}]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562388,"kb_used_data":264588,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93809452,"statfs":{"total":96636764160,"available":96060878848,"internally_reserved":0,"allocated":270938112,"data_stored":270762646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,10,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":7.3600000000000003},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":8.8640000000000008},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.101}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.8559999999999999},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.0069999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573640,"kb_used_data":263360,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93798200,"statfs":{"total":96636764160,"available":96049356800,"internally_reserved":0,"allocated":269680640,"data_stored":269505174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,6,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":6.6600000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.7060000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":5.6420000000000003},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.98599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.9020000000000001},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":269029376,"data_stored":269029376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270143488,"data_stored":270143488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269406208,"data_stored":269406208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265355264,"data_stored":265355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:02.229 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:02.229 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9104", 'reported_seq': 9202, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.696673+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:01.696673+0000', 'last_peered': '2024-09-18T11:51:01.696673+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:01.696673+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9104, 'log_dups_size': 0, 'ondisk_log_size': 9104, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33964032, 'num_objects': 4146, 'num_object_clones': 0, 'num_object_copies': 16584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9104, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2043}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:02.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:02.230 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8715", 'reported_seq': 8830, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:57.244065+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:50:57.244065+0000', 'last_peered': '2024-09-18T11:50:57.244065+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:50:57.244065+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8715, 'log_dups_size': 0, 'ondisk_log_size': 8715, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33528467, 'num_objects': 4094, 'num_object_clones': 0, 'num_object_copies': 16376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4094, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8714, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2054}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:02.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:02.230 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8865", 'reported_seq': 8956, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:57.749255+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:50:57.749255+0000', 'last_peered': '2024-09-18T11:50:57.749255+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:50:57.749255+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8865, 'log_dups_size': 0, 'ondisk_log_size': 8865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33264886, 'num_objects': 4061, 'num_object_clones': 0, 'num_object_copies': 16244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8863, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1960}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:02.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:02.230 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8856", 'reported_seq': 8975, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.551745+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:01.551745+0000', 'last_peered': '2024-09-18T11:51:01.551745+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:01.551745+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8856, 'log_dups_size': 0, 'ondisk_log_size': 8856, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33084662, 'num_objects': 4039, 'num_object_clones': 0, 'num_object_copies': 16156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8854, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1987}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:02.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:02.230 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8845", 'reported_seq': 8962, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.377111+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:01.377111+0000', 'last_peered': '2024-09-18T11:51:01.377111+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:01.377111+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8845, 'log_dups_size': 0, 'ondisk_log_size': 8845, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32284672, 'num_objects': 3941, 'num_object_clones': 0, 'num_object_copies': 15764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8845, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1822}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:02.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:02.230 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8920", 'reported_seq': 9025, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:57.528244+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:50:57.528244+0000', 'last_peered': '2024-09-18T11:50:57.528244+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:50:57.528244+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8920, 'log_dups_size': 0, 'ondisk_log_size': 8920, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33551606, 'num_objects': 4096, 'num_object_clones': 0, 'num_object_copies': 16384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8918, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1980}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:02.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:02.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8729", 'reported_seq': 8835, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:57.615707+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:50:57.615707+0000', 'last_peered': '2024-09-18T11:50:57.615707+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:50:57.615707+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8729, 'log_dups_size': 0, 'ondisk_log_size': 8729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33961206, 'num_objects': 4146, 'num_object_clones': 0, 'num_object_copies': 16584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8727, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2132}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:02.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:02.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9069", 'reported_seq': 11318, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.637537+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:01.637537+0000', 'last_peered': '2024-09-18T11:51:01.637537+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:01.637537+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9069, 'log_dups_size': 0, 'ondisk_log_size': 9069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33638892, 'num_objects': 4107, 'num_object_clones': 0, 'num_object_copies': 16428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9066, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1096, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3062}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:02.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:02.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8541", 'reported_seq': 8653, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:57.186848+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:50:57.186848+0000', 'last_peered': '2024-09-18T11:50:57.186848+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:50:57.186848+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8541, 'log_dups_size': 0, 'ondisk_log_size': 8541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33199350, 'num_objects': 4053, 'num_object_clones': 0, 'num_object_copies': 16212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8539, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:02.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8940", 'reported_seq': 9058, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.685151+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:01.685151+0000', 'last_peered': '2024-09-18T11:51:01.685151+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:01.685151+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8940, 'log_dups_size': 0, 'ondisk_log_size': 8940, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33525617, 'num_objects': 4093, 'num_object_clones': 0, 'num_object_copies': 16372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8937, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8965", 'reported_seq': 9055, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.623622+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:01.623622+0000', 'last_peered': '2024-09-18T11:51:01.623622+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:01.623622+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8965, 'log_dups_size': 0, 'ondisk_log_size': 8965, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33307259, 'num_objects': 4066, 'num_object_clones': 0, 'num_object_copies': 16264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4066, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8964, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1982}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8816", 'reported_seq': 8912, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:57.267436+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:50:57.267436+0000', 'last_peered': '2024-09-18T11:50:57.267436+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:50:57.267436+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8816, 'log_dups_size': 0, 'ondisk_log_size': 8816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34003579, 'num_objects': 4151, 'num_object_clones': 0, 'num_object_copies': 16604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8815, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9038", 'reported_seq': 9161, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.549626+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:01.549626+0000', 'last_peered': '2024-09-18T11:51:01.549626+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:01.549626+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9038, 'log_dups_size': 0, 'ondisk_log_size': 9038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33191158, 'num_objects': 4052, 'num_object_clones': 0, 'num_object_copies': 16208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9036, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1929}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:02.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8628", 'reported_seq': 8716, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:50:57.278969+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:50:57.278969+0000', 'last_peered': '2024-09-18T11:50:57.278969+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:50:57.278969+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8628, 'log_dups_size': 0, 'ondisk_log_size': 8628, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33412342, 'num_objects': 4079, 'num_object_clones': 0, 'num_object_copies': 16316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8626, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2077}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:02.233 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:02.943 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27862/261368 objects degraded (10.660%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":461,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":65342,"data_bytes":535233932,"bytes_used":2317742080,"bytes_avail":384229314560,"bytes_total":386547056640,"degraded_objects":27862,"degraded_total":261368,"degraded_ratio":0.10660065501515105,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":226},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:02.973 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:02.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:03.488 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:03.508 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-18T11:51:01.987727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":535233932,"num_objects":65342,"num_object_clones":0,"num_object_copies":261368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65342,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141745,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5594,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141772,"ondisk_log_size":141772,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2263420,"kb_used_data":1051772,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375223940,"statfs":{"total":386547056640,"available":384229314560,"internally_reserved":0,"allocated":1077014528,"data_stored":1076312664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,8,16,13,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22306816,"num_objects":-2723,"num_object_clones":0,"num_object_copies":-10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2723,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003529"},"pg_stats":[{"pgid":"1.f","version":"35'9104","reported_seq":9202,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.696673+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:01.696673+0000","last_peered":"2024-09-18T11:51:01.696673+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:01.696673+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9104,"log_dups_size":0,"ondisk_log_size":9104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33964032,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9104,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2043}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8924","reported_seq":13214,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:56.839529+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:50:56.839529+0000","last_peered":"2024-09-18T11:50:56.839529+0000","last_clean":"2024-09-18T11:50:56.839529+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:50:56.839529+0000","last_undegraded":"2024-09-18T11:50:56.839529+0000","last_fullsized":"2024-09-18T11:50:56.839529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8924,"log_dups_size":0,"ondisk_log_size":8924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33846518,"num_objects":4132,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8922,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8715","reported_seq":8830,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.244065+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:50:57.244065+0000","last_peered":"2024-09-18T11:50:57.244065+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:50:57.244065+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528467,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8714,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8865","reported_seq":8956,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.749255+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:50:57.749255+0000","last_peered":"2024-09-18T11:50:57.749255+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:50:57.749255+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264886,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1960}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8856","reported_seq":8975,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.551745+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:01.551745+0000","last_peered":"2024-09-18T11:51:01.551745+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:01.551745+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8856,"log_dups_size":0,"ondisk_log_size":8856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33084662,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8845","reported_seq":8962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.377111+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:01.377111+0000","last_peered":"2024-09-18T11:51:01.377111+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:01.377111+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32284672,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8845,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8920","reported_seq":9025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.528244+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:50:57.528244+0000","last_peered":"2024-09-18T11:50:57.528244+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:50:57.528244+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8920,"log_dups_size":0,"ondisk_log_size":8920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551606,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1980}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8729","reported_seq":8835,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.615707+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:50:57.615707+0000","last_peered":"2024-09-18T11:50:57.615707+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:50:57.615707+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8729,"log_dups_size":0,"ondisk_log_size":8729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33961206,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8727,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9069","reported_seq":11318,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.637537+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:01.637537+0000","last_peered":"2024-09-18T11:51:01.637537+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:01.637537+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9069,"log_dups_size":0,"ondisk_log_size":9069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33638892,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1096,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8541","reported_seq":8653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.186848+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:50:57.186848+0000","last_peered":"2024-09-18T11:50:57.186848+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:50:57.186848+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33199350,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8940","reported_seq":9058,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.685151+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:01.685151+0000","last_peered":"2024-09-18T11:51:01.685151+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:01.685151+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8940,"log_dups_size":0,"ondisk_log_size":8940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33525617,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8937,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8965","reported_seq":9055,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.623622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:01.623622+0000","last_peered":"2024-09-18T11:51:01.623622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:01.623622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8965,"log_dups_size":0,"ondisk_log_size":8965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33307259,"num_objects":4066,"num_object_clones":0,"num_object_copies":16264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8964,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8816","reported_seq":8912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.267436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:50:57.267436+0000","last_peered":"2024-09-18T11:50:57.267436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:50:57.267436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8816,"log_dups_size":0,"ondisk_log_size":8816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34003579,"num_objects":4151,"num_object_clones":0,"num_object_copies":16604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9038","reported_seq":9161,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.549626+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:01.549626+0000","last_peered":"2024-09-18T11:51:01.549626+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:01.549626+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9038,"log_dups_size":0,"ondisk_log_size":9038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33191158,"num_objects":4052,"num_object_clones":0,"num_object_copies":16208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1929}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8817","reported_seq":13104,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:50:56.920548+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:50:56.920548+0000","last_peered":"2024-09-18T11:50:56.920548+0000","last_clean":"2024-09-18T11:50:56.920548+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:50:56.920548+0000","last_undegraded":"2024-09-18T11:50:56.920548+0000","last_fullsized":"2024-09-18T11:50:56.920548+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8817,"log_dups_size":0,"ondisk_log_size":8817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33469686,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8628","reported_seq":8716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:50:57.278969+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:50:57.278969+0000","last_peered":"2024-09-18T11:50:57.278969+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:50:57.278969+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8628,"log_dups_size":0,"ondisk_log_size":8628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33412342,"num_objects":4079,"num_object_clones":0,"num_object_copies":16316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8626,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":535233932,"num_objects":65342,"num_object_clones":0,"num_object_copies":261368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65342,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141745,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5594,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073934336,"data_stored":1073934336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141772,"ondisk_log_size":141772,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566092,"kb_used_data":259876,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93805748,"statfs":{"total":96636764160,"available":96057085952,"internally_reserved":0,"allocated":266113024,"data_stored":265937558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.523},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.0510000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.3410000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":0.66600000000000004},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.0660000000000007}]}]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561300,"kb_used_data":263948,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93810540,"statfs":{"total":96636764160,"available":96061992960,"internally_reserved":0,"allocated":270282752,"data_stored":270107286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.5880000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.8769999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.9280000000000008},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":8.8290000000000006}]}]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562388,"kb_used_data":264588,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93809452,"statfs":{"total":96636764160,"available":96060878848,"internally_reserved":0,"allocated":270938112,"data_stored":270762646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,10,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":7.3600000000000003},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":8.8640000000000008},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.101}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.8559999999999999},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":7.0069999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573640,"kb_used_data":263360,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93798200,"statfs":{"total":96636764160,"available":96049356800,"internally_reserved":0,"allocated":269680640,"data_stored":269505174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,6,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":6.6600000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.7060000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":5.6420000000000003},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.98599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.9020000000000001},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":269029376,"data_stored":269029376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270143488,"data_stored":270143488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269406208,"data_stored":269406208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265355264,"data_stored":265355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:06.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:07.085 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:07.116 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-18T11:51:05.988831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":529884556,"num_objects":64689,"num_object_clones":0,"num_object_copies":258756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64689,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142398,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5594,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142425,"ondisk_log_size":142425,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2245768,"kb_used_data":1038248,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375241592,"statfs":{"total":386547056640,"available":384247390208,"internally_reserved":0,"allocated":1063165952,"data_stored":1062464088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,5,14,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18702336,"num_objects":-2283,"num_object_clones":0,"num_object_copies":-9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2283,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"1.f","version":"35'9104","reported_seq":9202,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.696673+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:01.696673+0000","last_peered":"2024-09-18T11:51:01.696673+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:01.696673+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9104,"log_dups_size":0,"ondisk_log_size":9104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33964032,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9104,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2043}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8990","reported_seq":13278,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:01.929846+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:01.929846+0000","last_peered":"2024-09-18T11:51:01.929846+0000","last_clean":"2024-09-18T11:51:01.929846+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:01.929846+0000","last_undegraded":"2024-09-18T11:51:01.929846+0000","last_fullsized":"2024-09-18T11:51:01.929846+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8990,"log_dups_size":0,"ondisk_log_size":8990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33305846,"num_objects":4066,"num_object_clones":0,"num_object_copies":16264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8988,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8798","reported_seq":8913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.277636+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:02.277636+0000","last_peered":"2024-09-18T11:51:02.277636+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:02.277636+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8798,"log_dups_size":0,"ondisk_log_size":8798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32848531,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8797,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1971}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8929","reported_seq":9020,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.799240+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:02.799240+0000","last_peered":"2024-09-18T11:51:02.799240+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:02.799240+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8929,"log_dups_size":0,"ondisk_log_size":8929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32740598,"num_objects":3997,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8927,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8856","reported_seq":8975,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.551745+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:01.551745+0000","last_peered":"2024-09-18T11:51:01.551745+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:01.551745+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8856,"log_dups_size":0,"ondisk_log_size":8856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33084662,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8845","reported_seq":8962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.377111+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:01.377111+0000","last_peered":"2024-09-18T11:51:01.377111+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:01.377111+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32284672,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8845,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8988","reported_seq":9092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.725241+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:02.725241+0000","last_peered":"2024-09-18T11:51:02.725241+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:02.725241+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8988,"log_dups_size":0,"ondisk_log_size":8988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32994550,"num_objects":4028,"num_object_clones":0,"num_object_copies":16112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8986,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1912}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8800","reported_seq":8907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.824661+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:02.824661+0000","last_peered":"2024-09-18T11:51:02.824661+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:02.824661+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8800,"log_dups_size":0,"ondisk_log_size":8800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33379574,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8798,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9069","reported_seq":11318,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.637537+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:01.637537+0000","last_peered":"2024-09-18T11:51:01.637537+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:01.637537+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9069,"log_dups_size":0,"ondisk_log_size":9069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33638892,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1096,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8618","reported_seq":8730,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.317121+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:02.317121+0000","last_peered":"2024-09-18T11:51:02.317121+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:02.317121+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32568566,"num_objects":3976,"num_object_clones":0,"num_object_copies":15904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8616,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8940","reported_seq":9058,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.685151+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:01.685151+0000","last_peered":"2024-09-18T11:51:01.685151+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:01.685151+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8940,"log_dups_size":0,"ondisk_log_size":8940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33525617,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8937,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8965","reported_seq":9055,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.623622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:01.623622+0000","last_peered":"2024-09-18T11:51:01.623622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:01.623622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8965,"log_dups_size":0,"ondisk_log_size":8965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33307259,"num_objects":4066,"num_object_clones":0,"num_object_copies":16264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8964,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8895","reported_seq":8991,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.292305+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:02.292305+0000","last_peered":"2024-09-18T11:51:02.292305+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:02.292305+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8895,"log_dups_size":0,"ondisk_log_size":8895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33356411,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2026}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9038","reported_seq":9161,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.549626+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:01.549626+0000","last_peered":"2024-09-18T11:51:01.549626+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:01.549626+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9038,"log_dups_size":0,"ondisk_log_size":9038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33191158,"num_objects":4052,"num_object_clones":0,"num_object_copies":16208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1929}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8898","reported_seq":13183,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:01.941686+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:01.941686+0000","last_peered":"2024-09-18T11:51:01.941686+0000","last_clean":"2024-09-18T11:51:01.941686+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:01.941686+0000","last_undegraded":"2024-09-18T11:51:01.941686+0000","last_fullsized":"2024-09-18T11:51:01.941686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32806134,"num_objects":4005,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8896,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8692","reported_seq":8780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.677524+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:02.677524+0000","last_peered":"2024-09-18T11:51:02.677524+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:02.677524+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8692,"log_dups_size":0,"ondisk_log_size":8692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32888054,"num_objects":4015,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8690,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":529884556,"num_objects":64689,"num_object_clones":0,"num_object_copies":258756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64689,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142398,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5594,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1059885056,"data_stored":1059885056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142425,"ondisk_log_size":142425,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566092,"kb_used_data":259876,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93805748,"statfs":{"total":96636764160,"available":96057085952,"internally_reserved":0,"allocated":266113024,"data_stored":265937558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.523},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.0510000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.3410000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":0.66600000000000004},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.0660000000000007}]}]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560968,"kb_used_data":259444,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93810872,"statfs":{"total":96636764160,"available":96062332928,"internally_reserved":0,"allocated":265670656,"data_stored":265495190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.17199999999999999},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.23499999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":19.751000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.9020000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.20699999999999999},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.2509999999999994}]}]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562036,"kb_used_data":260060,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93809804,"statfs":{"total":96636764160,"available":96061239296,"internally_reserved":0,"allocated":266301440,"data_stored":266125974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.1850000000000005},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":9.9600000000000009}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":2.1920000000000002},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.3940000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":16.259},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":16.515999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556672,"kb_used_data":258868,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93815168,"statfs":{"total":96636764160,"available":96066732032,"internally_reserved":0,"allocated":265080832,"data_stored":264905366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":1.3759999999999999},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.8819999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.661},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":1.0840000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":1.345},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":4.9189999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":264368128,"data_stored":264368128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265457664,"data_stored":265457664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264704000,"data_stored":264704000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265355264,"data_stored":265355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:07.118 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:07.699 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:07.700 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:07.717 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-18T11:51:05.988831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":529884556,"num_objects":64689,"num_object_clones":0,"num_object_copies":258756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64689,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142398,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5594,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142425,"ondisk_log_size":142425,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2245768,"kb_used_data":1038248,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375241592,"statfs":{"total":386547056640,"available":384247390208,"internally_reserved":0,"allocated":1063165952,"data_stored":1062464088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,5,14,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18702336,"num_objects":-2283,"num_object_clones":0,"num_object_copies":-9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2283,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"1.f","version":"35'9104","reported_seq":9202,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.696673+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:01.696673+0000","last_peered":"2024-09-18T11:51:01.696673+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:01.696673+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9104,"log_dups_size":0,"ondisk_log_size":9104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33964032,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9104,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2043}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8990","reported_seq":13278,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:01.929846+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:01.929846+0000","last_peered":"2024-09-18T11:51:01.929846+0000","last_clean":"2024-09-18T11:51:01.929846+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:01.929846+0000","last_undegraded":"2024-09-18T11:51:01.929846+0000","last_fullsized":"2024-09-18T11:51:01.929846+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8990,"log_dups_size":0,"ondisk_log_size":8990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33305846,"num_objects":4066,"num_object_clones":0,"num_object_copies":16264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8988,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8798","reported_seq":8913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.277636+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:02.277636+0000","last_peered":"2024-09-18T11:51:02.277636+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:02.277636+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8798,"log_dups_size":0,"ondisk_log_size":8798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32848531,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8797,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1971}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8929","reported_seq":9020,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.799240+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:02.799240+0000","last_peered":"2024-09-18T11:51:02.799240+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:02.799240+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8929,"log_dups_size":0,"ondisk_log_size":8929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32740598,"num_objects":3997,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8927,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8856","reported_seq":8975,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.551745+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:01.551745+0000","last_peered":"2024-09-18T11:51:01.551745+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:01.551745+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8856,"log_dups_size":0,"ondisk_log_size":8856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33084662,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8845","reported_seq":8962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.377111+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:01.377111+0000","last_peered":"2024-09-18T11:51:01.377111+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:01.377111+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32284672,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8845,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8988","reported_seq":9092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.725241+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:02.725241+0000","last_peered":"2024-09-18T11:51:02.725241+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:02.725241+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8988,"log_dups_size":0,"ondisk_log_size":8988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32994550,"num_objects":4028,"num_object_clones":0,"num_object_copies":16112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8986,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1912}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8800","reported_seq":8907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.824661+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:02.824661+0000","last_peered":"2024-09-18T11:51:02.824661+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:02.824661+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8800,"log_dups_size":0,"ondisk_log_size":8800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33379574,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8798,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9069","reported_seq":11318,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.637537+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:01.637537+0000","last_peered":"2024-09-18T11:51:01.637537+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:01.637537+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9069,"log_dups_size":0,"ondisk_log_size":9069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33638892,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1096,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8618","reported_seq":8730,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.317121+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:02.317121+0000","last_peered":"2024-09-18T11:51:02.317121+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:02.317121+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32568566,"num_objects":3976,"num_object_clones":0,"num_object_copies":15904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8616,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8940","reported_seq":9058,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.685151+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:01.685151+0000","last_peered":"2024-09-18T11:51:01.685151+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:01.685151+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8940,"log_dups_size":0,"ondisk_log_size":8940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33525617,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8937,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8965","reported_seq":9055,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.623622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:01.623622+0000","last_peered":"2024-09-18T11:51:01.623622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:01.623622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8965,"log_dups_size":0,"ondisk_log_size":8965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33307259,"num_objects":4066,"num_object_clones":0,"num_object_copies":16264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8964,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8895","reported_seq":8991,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.292305+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:02.292305+0000","last_peered":"2024-09-18T11:51:02.292305+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:02.292305+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8895,"log_dups_size":0,"ondisk_log_size":8895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33356411,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2026}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9038","reported_seq":9161,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:01.549626+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:01.549626+0000","last_peered":"2024-09-18T11:51:01.549626+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:01.549626+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9038,"log_dups_size":0,"ondisk_log_size":9038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33191158,"num_objects":4052,"num_object_clones":0,"num_object_copies":16208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1929}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8898","reported_seq":13183,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:01.941686+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:01.941686+0000","last_peered":"2024-09-18T11:51:01.941686+0000","last_clean":"2024-09-18T11:51:01.941686+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:01.941686+0000","last_undegraded":"2024-09-18T11:51:01.941686+0000","last_fullsized":"2024-09-18T11:51:01.941686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32806134,"num_objects":4005,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8896,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8692","reported_seq":8780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:02.677524+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:02.677524+0000","last_peered":"2024-09-18T11:51:02.677524+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:02.677524+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8692,"log_dups_size":0,"ondisk_log_size":8692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32888054,"num_objects":4015,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8690,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":529884556,"num_objects":64689,"num_object_clones":0,"num_object_copies":258756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64689,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142398,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5594,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1059885056,"data_stored":1059885056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142425,"ondisk_log_size":142425,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566092,"kb_used_data":259876,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93805748,"statfs":{"total":96636764160,"available":96057085952,"internally_reserved":0,"allocated":266113024,"data_stored":265937558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.523},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":4.0510000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.3410000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":0.66600000000000004},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":8.0660000000000007}]}]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560968,"kb_used_data":259444,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93810872,"statfs":{"total":96636764160,"available":96062332928,"internally_reserved":0,"allocated":265670656,"data_stored":265495190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:05 2024","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":5.1950000000000003,"15min":2.7490000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":12.59,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.17199999999999999},{"interface":"front","average":{"1min":5.2270000000000003,"5min":4.0819999999999999,"15min":2.302},"min":{"1min":0.498,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":12.558,"5min":18.047000000000001,"15min":18.047000000000001},"last":0.23499999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":19.751000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.9020000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.20699999999999999},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":9.2509999999999994}]}]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562036,"kb_used_data":260060,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93809804,"statfs":{"total":96636764160,"available":96061239296,"internally_reserved":0,"allocated":266301440,"data_stored":266125974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":6.9580000000000002,"5min":4.5979999999999999,"15min":3.726},"min":{"1min":0.28499999999999998,"5min":0.159,"15min":0.159},"max":{"1min":21.030999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.1850000000000005},{"interface":"front","average":{"1min":5.96,"5min":4.9690000000000003,"15min":5.0579999999999998},"min":{"1min":0.57899999999999996,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":25.187000000000001,"5min":25.187000000000001,"15min":25.187000000000001},"last":9.9600000000000009}]},{"osd":2,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":7.2350000000000003,"5min":5.5800000000000001,"15min":4.4809999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":31.306000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":2.1920000000000002},{"interface":"front","average":{"1min":4.5949999999999998,"5min":4.4219999999999997,"15min":4.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19,"15min":0.19},"max":{"1min":15.736000000000001,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.3940000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:49:58 2024","interfaces":[{"interface":"back","average":{"1min":5.2300000000000004,"5min":4.5229999999999997,"15min":4.2949999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":16.193000000000001,"5min":17.390999999999998,"15min":17.390999999999998},"last":16.259},{"interface":"front","average":{"1min":7.343,"5min":5.8310000000000004,"15min":5.0190000000000001},"min":{"1min":0.247,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":22.138999999999999,"5min":25.440000000000001,"15min":25.440000000000001},"last":16.515999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556672,"kb_used_data":258868,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93815168,"statfs":{"total":96636764160,"available":96066732032,"internally_reserved":0,"allocated":265080832,"data_stored":264905366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":1.3759999999999999},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":4.8819999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.661},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":1.0840000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":1.345},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":4.9189999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":264368128,"data_stored":264368128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265457664,"data_stored":265457664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264704000,"data_stored":264704000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265355264,"data_stored":265355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:07.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:07.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9104", 'reported_seq': 9202, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.696673+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:01.696673+0000', 'last_peered': '2024-09-18T11:51:01.696673+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:01.696673+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9104, 'log_dups_size': 0, 'ondisk_log_size': 9104, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33964032, 'num_objects': 4146, 'num_object_clones': 0, 'num_object_copies': 16584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9104, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2043}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:07.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:07.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8798", 'reported_seq': 8913, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:02.277636+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:02.277636+0000', 'last_peered': '2024-09-18T11:51:02.277636+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:02.277636+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8798, 'log_dups_size': 0, 'ondisk_log_size': 8798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32848531, 'num_objects': 4011, 'num_object_clones': 0, 'num_object_copies': 16044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4011, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8797, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1971}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:07.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:07.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8929", 'reported_seq': 9020, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:02.799240+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:02.799240+0000', 'last_peered': '2024-09-18T11:51:02.799240+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:02.799240+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8929, 'log_dups_size': 0, 'ondisk_log_size': 8929, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32740598, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 15988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8927, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1896}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:07.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:07.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8856", 'reported_seq': 8975, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.551745+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:01.551745+0000', 'last_peered': '2024-09-18T11:51:01.551745+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:01.551745+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8856, 'log_dups_size': 0, 'ondisk_log_size': 8856, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33084662, 'num_objects': 4039, 'num_object_clones': 0, 'num_object_copies': 16156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8854, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1987}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:07.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:07.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8845", 'reported_seq': 8962, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.377111+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:01.377111+0000', 'last_peered': '2024-09-18T11:51:01.377111+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:01.377111+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8845, 'log_dups_size': 0, 'ondisk_log_size': 8845, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32284672, 'num_objects': 3941, 'num_object_clones': 0, 'num_object_copies': 15764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8845, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1822}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:07.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:07.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8988", 'reported_seq': 9092, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:02.725241+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:02.725241+0000', 'last_peered': '2024-09-18T11:51:02.725241+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:02.725241+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8988, 'log_dups_size': 0, 'ondisk_log_size': 8988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32994550, 'num_objects': 4028, 'num_object_clones': 0, 'num_object_copies': 16112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8986, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1912}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:07.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:07.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8800", 'reported_seq': 8907, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:02.824661+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:02.824661+0000', 'last_peered': '2024-09-18T11:51:02.824661+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:02.824661+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8800, 'log_dups_size': 0, 'ondisk_log_size': 8800, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33379574, 'num_objects': 4075, 'num_object_clones': 0, 'num_object_copies': 16300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8798, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2061}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:07.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:07.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9069", 'reported_seq': 11318, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.637537+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:01.637537+0000', 'last_peered': '2024-09-18T11:51:01.637537+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:01.637537+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9069, 'log_dups_size': 0, 'ondisk_log_size': 9069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33638892, 'num_objects': 4107, 'num_object_clones': 0, 'num_object_copies': 16428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9066, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1096, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3062}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:07.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:07.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8618", 'reported_seq': 8730, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:02.317121+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:02.317121+0000', 'last_peered': '2024-09-18T11:51:02.317121+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:02.317121+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8618, 'log_dups_size': 0, 'ondisk_log_size': 8618, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32568566, 'num_objects': 3976, 'num_object_clones': 0, 'num_object_copies': 15904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8616, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2013}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:07.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:07.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8940", 'reported_seq': 9058, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.685151+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:01.685151+0000', 'last_peered': '2024-09-18T11:51:01.685151+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:01.685151+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8940, 'log_dups_size': 0, 'ondisk_log_size': 8940, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33525617, 'num_objects': 4093, 'num_object_clones': 0, 'num_object_copies': 16372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8937, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:07.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:07.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8965", 'reported_seq': 9055, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.623622+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:01.623622+0000', 'last_peered': '2024-09-18T11:51:01.623622+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:01.623622+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8965, 'log_dups_size': 0, 'ondisk_log_size': 8965, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33307259, 'num_objects': 4066, 'num_object_clones': 0, 'num_object_copies': 16264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4066, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8964, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1982}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:07.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:07.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8895", 'reported_seq': 8991, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:02.292305+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:02.292305+0000', 'last_peered': '2024-09-18T11:51:02.292305+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:02.292305+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8895, 'log_dups_size': 0, 'ondisk_log_size': 8895, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33356411, 'num_objects': 4072, 'num_object_clones': 0, 'num_object_copies': 16288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8894, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2026}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:07.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:07.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9038", 'reported_seq': 9161, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:01.549626+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:01.549626+0000', 'last_peered': '2024-09-18T11:51:01.549626+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:01.549626+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9038, 'log_dups_size': 0, 'ondisk_log_size': 9038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33191158, 'num_objects': 4052, 'num_object_clones': 0, 'num_object_copies': 16208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9036, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1929}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:07.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:07.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8692", 'reported_seq': 8780, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:02.677524+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:02.677524+0000', 'last_peered': '2024-09-18T11:51:02.677524+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:02.677524+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8692, 'log_dups_size': 0, 'ondisk_log_size': 8692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32888054, 'num_objects': 4015, 'num_object_clones': 0, 'num_object_copies': 16060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8690, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2013}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:07.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:08.389 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:08.390 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27816/254260 objects degraded (10.940%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":466,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63565,"data_bytes":520676748,"bytes_used":2298191872,"bytes_avail":384248864768,"bytes_total":386547056640,"degraded_objects":27816,"degraded_total":254260,"degraded_ratio":0.10939982694879259,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":283},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:08.420 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:08.420 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:09.011 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:09.012 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:09.032 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-18T11:51:07.989467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":520676748,"num_objects":63565,"num_object_clones":0,"num_object_copies":254260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143522,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5640,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143549,"ondisk_log_size":143549,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2244328,"kb_used_data":1020220,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375243032,"statfs":{"total":386547056640,"available":384248864768,"internally_reserved":0,"allocated":1044705280,"data_stored":1044003416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,5,6,20,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27910144,"num_objects":-3407,"num_object_clones":0,"num_object_copies":-13628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3407,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003346"},"pg_stats":[{"pgid":"1.f","version":"35'9191","reported_seq":9290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:06.562249+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:06.562249+0000","last_peered":"2024-09-18T11:51:06.562249+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:06.562249+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9191,"log_dups_size":0,"ondisk_log_size":9191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33251328,"num_objects":4059,"num_object_clones":0,"num_object_copies":16236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9058","reported_seq":13346,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:06.905613+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:06.905613+0000","last_peered":"2024-09-18T11:51:06.905613+0000","last_clean":"2024-09-18T11:51:06.905613+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:06.905613+0000","last_undegraded":"2024-09-18T11:51:06.905613+0000","last_fullsized":"2024-09-18T11:51:06.905613+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9058,"log_dups_size":0,"ondisk_log_size":9058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32748790,"num_objects":3998,"num_object_clones":0,"num_object_copies":15992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9056,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8861","reported_seq":8976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.362323+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:07.362323+0000","last_peered":"2024-09-18T11:51:07.362323+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:07.362323+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8861,"log_dups_size":0,"ondisk_log_size":8861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32332435,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8860,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1908}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8989","reported_seq":9081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.633261+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:07.633261+0000","last_peered":"2024-09-18T11:51:07.633261+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:07.633261+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8989,"log_dups_size":0,"ondisk_log_size":8989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32249078,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8987,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'8921","reported_seq":9040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:06.620834+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:06.620834+0000","last_peered":"2024-09-18T11:51:06.620834+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:06.620834+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32552182,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8919,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8918","reported_seq":9035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:06.437565+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:06.437565+0000","last_peered":"2024-09-18T11:51:06.437565+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:06.437565+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8918,"log_dups_size":0,"ondisk_log_size":8918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31686656,"num_objects":3868,"num_object_clones":0,"num_object_copies":15472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9065","reported_seq":9170,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.551277+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:07.551277+0000","last_peered":"2024-09-18T11:51:07.551277+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:07.551277+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32363766,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9063,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1835}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8867","reported_seq":8974,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.850287+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:07.850287+0000","last_peered":"2024-09-18T11:51:07.850287+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:07.850287+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8867,"log_dups_size":0,"ondisk_log_size":8867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32830710,"num_objects":4008,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1994}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9155","reported_seq":11496,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:06.743839+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:06.743839+0000","last_peered":"2024-09-18T11:51:06.743839+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:06.743839+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9155,"log_dups_size":0,"ondisk_log_size":9155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32934380,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8688","reported_seq":8800,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.339849+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:07.339849+0000","last_peered":"2024-09-18T11:51:07.339849+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:07.339849+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8688,"log_dups_size":0,"ondisk_log_size":8688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31995126,"num_objects":3906,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8686,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8999","reported_seq":9118,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:06.653449+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:06.653449+0000","last_peered":"2024-09-18T11:51:06.653449+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:06.653449+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8999,"log_dups_size":0,"ondisk_log_size":8999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33042289,"num_objects":4034,"num_object_clones":0,"num_object_copies":16136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8996,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9039","reported_seq":9129,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:06.705827+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:06.705827+0000","last_peered":"2024-09-18T11:51:06.705827+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:06.705827+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9039,"log_dups_size":0,"ondisk_log_size":9039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32701051,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9038,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1908}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8964","reported_seq":9060,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.094615+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:07.094615+0000","last_peered":"2024-09-18T11:51:07.094615+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:07.094615+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32791163,"num_objects":4003,"num_object_clones":0,"num_object_copies":16012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8963,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1957}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9107","reported_seq":9230,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:06.616136+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:06.616136+0000","last_peered":"2024-09-18T11:51:06.616136+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:06.616136+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9107,"log_dups_size":0,"ondisk_log_size":9107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32625910,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9105,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8956","reported_seq":13243,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:06.844849+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:06.844849+0000","last_peered":"2024-09-18T11:51:06.844849+0000","last_clean":"2024-09-18T11:51:06.844849+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:06.844849+0000","last_undegraded":"2024-09-18T11:51:06.844849+0000","last_fullsized":"2024-09-18T11:51:06.844849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8956,"log_dups_size":0,"ondisk_log_size":8956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32330998,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8954,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8771","reported_seq":8858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.887570+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:07.887570+0000","last_peered":"2024-09-18T11:51:07.887570+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:07.887570+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8771,"log_dups_size":0,"ondisk_log_size":8771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32240886,"num_objects":3936,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8769,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":520676748,"num_objects":63565,"num_object_clones":0,"num_object_copies":254260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143522,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5640,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1041268736,"data_stored":1041268736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143549,"ondisk_log_size":143549,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565724,"kb_used_data":255380,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93806116,"statfs":{"total":96636764160,"available":96057462784,"internally_reserved":0,"allocated":261509120,"data_stored":261333654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,0,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":1.7},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":1.7470000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":1.724},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":0.55900000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":4.0990000000000002},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.589}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560564,"kb_used_data":254892,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93811276,"statfs":{"total":96636764160,"available":96062746624,"internally_reserved":0,"allocated":261009408,"data_stored":260833942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":3.0310000000000001},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":3.052}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.9369999999999998},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.1619999999999999},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":2.9990000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561624,"kb_used_data":255492,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93810216,"statfs":{"total":96636764160,"available":96061661184,"internally_reserved":0,"allocated":261623808,"data_stored":261448342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,10,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.90000000000000002},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.571}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":2.048},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.86299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":2.8490000000000002},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.2069999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556416,"kb_used_data":254456,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93815424,"statfs":{"total":96636764160,"available":96066994176,"internally_reserved":0,"allocated":260562944,"data_stored":260387478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":5.1180000000000003},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.89300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":5.9960000000000004},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.92000000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.2190000000000003},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":2.391}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259715072,"data_stored":259715072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260804608,"data_stored":260804608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260075520,"data_stored":260075520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":260673536,"data_stored":260673536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:12.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:12.559 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:12.560 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:12.584 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-18T11:51:11.998874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516572556,"num_objects":63064,"num_object_clones":0,"num_object_copies":252256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63064,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144023,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5696,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144050,"ondisk_log_size":144050,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2227512,"kb_used_data":1015880,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375259848,"statfs":{"total":386547056640,"available":384266084352,"internally_reserved":0,"allocated":1040261120,"data_stored":1039559256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,5,9,20,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22667264,"num_objects":-2767,"num_object_clones":0,"num_object_copies":-11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011583"},"pg_stats":[{"pgid":"1.f","version":"35'9261","reported_seq":9360,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.653475+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:11.653475+0000","last_peered":"2024-09-18T11:51:11.653475+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:11.653475+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9261,"log_dups_size":0,"ondisk_log_size":9261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32677888,"num_objects":3989,"num_object_clones":0,"num_object_copies":15956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9261,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9058","reported_seq":13346,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:06.905613+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:06.905613+0000","last_peered":"2024-09-18T11:51:06.905613+0000","last_clean":"2024-09-18T11:51:06.905613+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:06.905613+0000","last_undegraded":"2024-09-18T11:51:06.905613+0000","last_fullsized":"2024-09-18T11:51:06.905613+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9058,"log_dups_size":0,"ondisk_log_size":9058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32748790,"num_objects":3998,"num_object_clones":0,"num_object_copies":15992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9056,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8861","reported_seq":8976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.362323+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:07.362323+0000","last_peered":"2024-09-18T11:51:07.362323+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:07.362323+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8861,"log_dups_size":0,"ondisk_log_size":8861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32332435,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8860,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1908}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8989","reported_seq":9081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.633261+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:07.633261+0000","last_peered":"2024-09-18T11:51:07.633261+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:07.633261+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8989,"log_dups_size":0,"ondisk_log_size":8989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32249078,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8987,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9005","reported_seq":9124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.592664+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:11.592664+0000","last_peered":"2024-09-18T11:51:11.592664+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:11.592664+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9005,"log_dups_size":0,"ondisk_log_size":9005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31864054,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8980","reported_seq":9097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.486364+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:11.486364+0000","last_peered":"2024-09-18T11:51:11.486364+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:11.486364+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31178752,"num_objects":3806,"num_object_clones":0,"num_object_copies":15224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9065","reported_seq":9170,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.551277+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:07.551277+0000","last_peered":"2024-09-18T11:51:07.551277+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:07.551277+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32363766,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9063,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1835}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8867","reported_seq":8974,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.850287+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:07.850287+0000","last_peered":"2024-09-18T11:51:07.850287+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:07.850287+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8867,"log_dups_size":0,"ondisk_log_size":8867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32830710,"num_objects":4008,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1994}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9229","reported_seq":11679,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.749149+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:11.749149+0000","last_peered":"2024-09-18T11:51:11.749149+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:11.749149+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9229,"log_dups_size":0,"ondisk_log_size":9229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32328172,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1198,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8688","reported_seq":8800,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.339849+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:07.339849+0000","last_peered":"2024-09-18T11:51:07.339849+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:07.339849+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8688,"log_dups_size":0,"ondisk_log_size":8688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31995126,"num_objects":3906,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8686,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9070","reported_seq":9186,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.777246+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:11.777246+0000","last_peered":"2024-09-18T11:51:11.777246+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:11.777246+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9070,"log_dups_size":0,"ondisk_log_size":9070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32460657,"num_objects":3963,"num_object_clones":0,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9067,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9109","reported_seq":9200,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.738204+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:11.738204+0000","last_peered":"2024-09-18T11:51:11.738204+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:11.738204+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9109,"log_dups_size":0,"ondisk_log_size":9109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32127611,"num_objects":3922,"num_object_clones":0,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1838}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8964","reported_seq":9060,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.094615+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:07.094615+0000","last_peered":"2024-09-18T11:51:07.094615+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:07.094615+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32791163,"num_objects":4003,"num_object_clones":0,"num_object_copies":16012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8963,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1957}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9177","reported_seq":9300,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.683218+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:11.683218+0000","last_peered":"2024-09-18T11:51:11.683218+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:11.683218+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9177,"log_dups_size":0,"ondisk_log_size":9177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32052470,"num_objects":3913,"num_object_clones":0,"num_object_copies":15652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9175,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1790}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8956","reported_seq":13243,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:06.844849+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:06.844849+0000","last_peered":"2024-09-18T11:51:06.844849+0000","last_clean":"2024-09-18T11:51:06.844849+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:06.844849+0000","last_undegraded":"2024-09-18T11:51:06.844849+0000","last_fullsized":"2024-09-18T11:51:06.844849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8956,"log_dups_size":0,"ondisk_log_size":8956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32330998,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8954,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8771","reported_seq":8858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.887570+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:07.887570+0000","last_peered":"2024-09-18T11:51:07.887570+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:07.887570+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8771,"log_dups_size":0,"ondisk_log_size":8771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32240886,"num_objects":3936,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8769,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":516572556,"num_objects":63064,"num_object_clones":0,"num_object_copies":252256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63064,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144023,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5696,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1036705792,"data_stored":1036705792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144050,"ondisk_log_size":144050,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548908,"kb_used_data":251040,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93822932,"statfs":{"total":96636764160,"available":96074682368,"internally_reserved":0,"allocated":257064960,"data_stored":256889494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,3,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.1589999999999998},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.9219999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":0.88100000000000001},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.121}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560564,"kb_used_data":254892,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93811276,"statfs":{"total":96636764160,"available":96062746624,"internally_reserved":0,"allocated":261009408,"data_stored":260833942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":3.0310000000000001},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":3.052}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.9369999999999998},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.1619999999999999},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":2.9990000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561624,"kb_used_data":255492,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93810216,"statfs":{"total":96636764160,"available":96061661184,"internally_reserved":0,"allocated":261623808,"data_stored":261448342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,10,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.90000000000000002},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.571}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":2.048},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.86299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":2.8490000000000002},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.2069999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556416,"kb_used_data":254456,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93815424,"statfs":{"total":96636764160,"available":96066994176,"internally_reserved":0,"allocated":260562944,"data_stored":260387478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":5.1180000000000003},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.89300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":5.9960000000000004},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.92000000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.2190000000000003},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":2.391}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259715072,"data_stored":259715072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260804608,"data_stored":260804608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260075520,"data_stored":260075520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":256110592,"data_stored":256110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:12.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:13.136 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:13.136 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:13.158 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-18T11:51:11.998874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516572556,"num_objects":63064,"num_object_clones":0,"num_object_copies":252256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63064,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144023,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5696,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144050,"ondisk_log_size":144050,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2227512,"kb_used_data":1015880,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375259848,"statfs":{"total":386547056640,"available":384266084352,"internally_reserved":0,"allocated":1040261120,"data_stored":1039559256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,5,9,20,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22667264,"num_objects":-2767,"num_object_clones":0,"num_object_copies":-11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011583"},"pg_stats":[{"pgid":"1.f","version":"35'9261","reported_seq":9360,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.653475+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:11.653475+0000","last_peered":"2024-09-18T11:51:11.653475+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:11.653475+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9261,"log_dups_size":0,"ondisk_log_size":9261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32677888,"num_objects":3989,"num_object_clones":0,"num_object_copies":15956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9261,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9058","reported_seq":13346,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:06.905613+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:06.905613+0000","last_peered":"2024-09-18T11:51:06.905613+0000","last_clean":"2024-09-18T11:51:06.905613+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:06.905613+0000","last_undegraded":"2024-09-18T11:51:06.905613+0000","last_fullsized":"2024-09-18T11:51:06.905613+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9058,"log_dups_size":0,"ondisk_log_size":9058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32748790,"num_objects":3998,"num_object_clones":0,"num_object_copies":15992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9056,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8861","reported_seq":8976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.362323+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:07.362323+0000","last_peered":"2024-09-18T11:51:07.362323+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:07.362323+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8861,"log_dups_size":0,"ondisk_log_size":8861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32332435,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8860,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1908}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8989","reported_seq":9081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.633261+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:07.633261+0000","last_peered":"2024-09-18T11:51:07.633261+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:07.633261+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8989,"log_dups_size":0,"ondisk_log_size":8989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32249078,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8987,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9005","reported_seq":9124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.592664+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:11.592664+0000","last_peered":"2024-09-18T11:51:11.592664+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:11.592664+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9005,"log_dups_size":0,"ondisk_log_size":9005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31864054,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8980","reported_seq":9097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.486364+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:11.486364+0000","last_peered":"2024-09-18T11:51:11.486364+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:11.486364+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31178752,"num_objects":3806,"num_object_clones":0,"num_object_copies":15224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9065","reported_seq":9170,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.551277+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:07.551277+0000","last_peered":"2024-09-18T11:51:07.551277+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:07.551277+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32363766,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9063,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1835}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8867","reported_seq":8974,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.850287+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:07.850287+0000","last_peered":"2024-09-18T11:51:07.850287+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:07.850287+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8867,"log_dups_size":0,"ondisk_log_size":8867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32830710,"num_objects":4008,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1994}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9229","reported_seq":11679,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.749149+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:11.749149+0000","last_peered":"2024-09-18T11:51:11.749149+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:11.749149+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9229,"log_dups_size":0,"ondisk_log_size":9229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32328172,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1198,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8688","reported_seq":8800,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.339849+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:07.339849+0000","last_peered":"2024-09-18T11:51:07.339849+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:07.339849+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8688,"log_dups_size":0,"ondisk_log_size":8688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31995126,"num_objects":3906,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8686,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9070","reported_seq":9186,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.777246+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:11.777246+0000","last_peered":"2024-09-18T11:51:11.777246+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:11.777246+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9070,"log_dups_size":0,"ondisk_log_size":9070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32460657,"num_objects":3963,"num_object_clones":0,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9067,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9109","reported_seq":9200,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.738204+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:11.738204+0000","last_peered":"2024-09-18T11:51:11.738204+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:11.738204+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9109,"log_dups_size":0,"ondisk_log_size":9109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32127611,"num_objects":3922,"num_object_clones":0,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1838}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8964","reported_seq":9060,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.094615+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:07.094615+0000","last_peered":"2024-09-18T11:51:07.094615+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:07.094615+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32791163,"num_objects":4003,"num_object_clones":0,"num_object_copies":16012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8963,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1957}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9177","reported_seq":9300,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.683218+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:11.683218+0000","last_peered":"2024-09-18T11:51:11.683218+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:11.683218+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9177,"log_dups_size":0,"ondisk_log_size":9177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32052470,"num_objects":3913,"num_object_clones":0,"num_object_copies":15652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9175,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1790}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8956","reported_seq":13243,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:06.844849+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:06.844849+0000","last_peered":"2024-09-18T11:51:06.844849+0000","last_clean":"2024-09-18T11:51:06.844849+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:06.844849+0000","last_undegraded":"2024-09-18T11:51:06.844849+0000","last_fullsized":"2024-09-18T11:51:06.844849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8956,"log_dups_size":0,"ondisk_log_size":8956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32330998,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8954,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8771","reported_seq":8858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:07.887570+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:07.887570+0000","last_peered":"2024-09-18T11:51:07.887570+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:07.887570+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8771,"log_dups_size":0,"ondisk_log_size":8771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32240886,"num_objects":3936,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8769,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":516572556,"num_objects":63064,"num_object_clones":0,"num_object_copies":252256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63064,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144023,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5696,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1036705792,"data_stored":1036705792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144050,"ondisk_log_size":144050,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548908,"kb_used_data":251040,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93822932,"statfs":{"total":96636764160,"available":96074682368,"internally_reserved":0,"allocated":257064960,"data_stored":256889494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,3,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.1589999999999998},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.9219999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":0.88100000000000001},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.121}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560564,"kb_used_data":254892,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93811276,"statfs":{"total":96636764160,"available":96062746624,"internally_reserved":0,"allocated":261009408,"data_stored":260833942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":3.0310000000000001},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":3.052}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.9369999999999998},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.1619999999999999},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":2.9990000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561624,"kb_used_data":255492,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93810216,"statfs":{"total":96636764160,"available":96061661184,"internally_reserved":0,"allocated":261623808,"data_stored":261448342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,10,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.90000000000000002},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.571}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":2.048},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.86299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":2.8490000000000002},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":5.2069999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556416,"kb_used_data":254456,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93815424,"statfs":{"total":96636764160,"available":96066994176,"internally_reserved":0,"allocated":260562944,"data_stored":260387478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":5.1180000000000003},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.89300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:50:10 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":5.9500000000000002,"15min":2.544},"min":{"1min":0.33000000000000002,"5min":0.23999999999999999,"15min":0.14299999999999999},"max":{"1min":21.366,"5min":32.795000000000002,"15min":32.795000000000002},"last":5.9960000000000004},{"interface":"front","average":{"1min":6.4790000000000001,"5min":4.7329999999999997,"15min":2.3690000000000002},"min":{"1min":0.25,"5min":0.221,"15min":0.221},"max":{"1min":15.611000000000001,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.92000000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.2190000000000003},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":2.391}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259715072,"data_stored":259715072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260804608,"data_stored":260804608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260075520,"data_stored":260075520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":256110592,"data_stored":256110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:13.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:13.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9261", 'reported_seq': 9360, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:11.653475+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:11.653475+0000', 'last_peered': '2024-09-18T11:51:11.653475+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:11.653475+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9261, 'log_dups_size': 0, 'ondisk_log_size': 9261, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32677888, 'num_objects': 3989, 'num_object_clones': 0, 'num_object_copies': 15956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9261, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1886}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:13.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:13.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8861", 'reported_seq': 8976, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:07.362323+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:07.362323+0000', 'last_peered': '2024-09-18T11:51:07.362323+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:07.362323+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8861, 'log_dups_size': 0, 'ondisk_log_size': 8861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32332435, 'num_objects': 3948, 'num_object_clones': 0, 'num_object_copies': 15792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3948, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8860, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1908}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:13.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:13.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8989", 'reported_seq': 9081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:07.633261+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:07.633261+0000', 'last_peered': '2024-09-18T11:51:07.633261+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:07.633261+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8989, 'log_dups_size': 0, 'ondisk_log_size': 8989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32249078, 'num_objects': 3937, 'num_object_clones': 0, 'num_object_copies': 15748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8987, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1836}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:13.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:13.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9005", 'reported_seq': 9124, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:11.592664+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:11.592664+0000', 'last_peered': '2024-09-18T11:51:11.592664+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:11.592664+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9005, 'log_dups_size': 0, 'ondisk_log_size': 9005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31864054, 'num_objects': 3890, 'num_object_clones': 0, 'num_object_copies': 15560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9003, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1838}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:13.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:13.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8980", 'reported_seq': 9097, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:11.486364+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:11.486364+0000', 'last_peered': '2024-09-18T11:51:11.486364+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:11.486364+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8980, 'log_dups_size': 0, 'ondisk_log_size': 8980, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31178752, 'num_objects': 3806, 'num_object_clones': 0, 'num_object_copies': 15224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8980, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1687}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:13.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:13.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9065", 'reported_seq': 9170, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:07.551277+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:07.551277+0000', 'last_peered': '2024-09-18T11:51:07.551277+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:07.551277+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9065, 'log_dups_size': 0, 'ondisk_log_size': 9065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32363766, 'num_objects': 3951, 'num_object_clones': 0, 'num_object_copies': 15804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9063, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1835}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:13.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:13.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8867", 'reported_seq': 8974, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:07.850287+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:07.850287+0000', 'last_peered': '2024-09-18T11:51:07.850287+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:07.850287+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8867, 'log_dups_size': 0, 'ondisk_log_size': 8867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32830710, 'num_objects': 4008, 'num_object_clones': 0, 'num_object_copies': 16032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8865, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1994}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:13.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:13.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9229", 'reported_seq': 11679, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:11.749149+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:11.749149+0000', 'last_peered': '2024-09-18T11:51:11.749149+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:11.749149+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9229, 'log_dups_size': 0, 'ondisk_log_size': 9229, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32328172, 'num_objects': 3947, 'num_object_clones': 0, 'num_object_copies': 15788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 943, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9226, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1198, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3004}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:13.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:13.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8688", 'reported_seq': 8800, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:07.339849+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:07.339849+0000', 'last_peered': '2024-09-18T11:51:07.339849+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:07.339849+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8688, 'log_dups_size': 0, 'ondisk_log_size': 8688, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31995126, 'num_objects': 3906, 'num_object_clones': 0, 'num_object_copies': 15624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8686, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1943}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:13.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:13.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9070", 'reported_seq': 9186, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:11.777246+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:11.777246+0000', 'last_peered': '2024-09-18T11:51:11.777246+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:11.777246+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9070, 'log_dups_size': 0, 'ondisk_log_size': 9070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32460657, 'num_objects': 3963, 'num_object_clones': 0, 'num_object_copies': 15852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9067, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1929}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:13.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:13.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9109", 'reported_seq': 9200, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:11.738204+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:11.738204+0000', 'last_peered': '2024-09-18T11:51:11.738204+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:11.738204+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9109, 'log_dups_size': 0, 'ondisk_log_size': 9109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32127611, 'num_objects': 3922, 'num_object_clones': 0, 'num_object_copies': 15688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9108, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1838}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:13.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:13.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8964", 'reported_seq': 9060, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:07.094615+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:07.094615+0000', 'last_peered': '2024-09-18T11:51:07.094615+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:07.094615+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8964, 'log_dups_size': 0, 'ondisk_log_size': 8964, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32791163, 'num_objects': 4003, 'num_object_clones': 0, 'num_object_copies': 16012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8963, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:13.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:13.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9177", 'reported_seq': 9300, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:11.683218+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:11.683218+0000', 'last_peered': '2024-09-18T11:51:11.683218+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:11.683218+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9177, 'log_dups_size': 0, 'ondisk_log_size': 9177, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32052470, 'num_objects': 3913, 'num_object_clones': 0, 'num_object_copies': 15652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9175, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1790}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:13.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:13.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8771", 'reported_seq': 8858, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:07.887570+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:07.887570+0000', 'last_peered': '2024-09-18T11:51:07.887570+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:07.887570+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8771, 'log_dups_size': 0, 'ondisk_log_size': 8771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32240886, 'num_objects': 3936, 'num_object_clones': 0, 'num_object_copies': 15744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8769, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:13.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:13.827 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:13.827 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27760/252256 objects degraded (11.005%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":471,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63064,"data_bytes":516572556,"bytes_used":2280972288,"bytes_avail":384266084352,"bytes_total":386547056640,"degraded_objects":27760,"degraded_total":252256,"degraded_ratio":0.11004693644551565,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":230},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:13.853 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:13.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:14.418 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:14.440 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-18T11:51:14.000554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":511550860,"num_objects":62451,"num_object_clones":0,"num_object_copies":249804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144636,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5696,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144663,"ondisk_log_size":144663,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2226768,"kb_used_data":1002668,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375260592,"statfs":{"total":386547056640,"available":384266846208,"internally_reserved":0,"allocated":1026732032,"data_stored":1026030168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,2,12,13,17,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23683072,"num_objects":-2891,"num_object_clones":0,"num_object_copies":-11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2891,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012826"},"pg_stats":[{"pgid":"1.f","version":"35'9261","reported_seq":9360,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.653475+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:11.653475+0000","last_peered":"2024-09-18T11:51:11.653475+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:11.653475+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9261,"log_dups_size":0,"ondisk_log_size":9261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32677888,"num_objects":3989,"num_object_clones":0,"num_object_copies":15956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9261,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9121","reported_seq":13411,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:11.978989+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:11.978989+0000","last_peered":"2024-09-18T11:51:11.978989+0000","last_clean":"2024-09-18T11:51:11.978989+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:11.978989+0000","last_undegraded":"2024-09-18T11:51:11.978989+0000","last_fullsized":"2024-09-18T11:51:11.978989+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9121,"log_dups_size":0,"ondisk_log_size":9121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32232694,"num_objects":3935,"num_object_clones":0,"num_object_copies":15740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9119,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8914","reported_seq":9029,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.160906+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:12.160906+0000","last_peered":"2024-09-18T11:51:12.160906+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:12.160906+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8914,"log_dups_size":0,"ondisk_log_size":8914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31898259,"num_objects":3895,"num_object_clones":0,"num_object_copies":15580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3895,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8913,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1855}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9065","reported_seq":9157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.883769+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:12.883769+0000","last_peered":"2024-09-18T11:51:12.883769+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:12.883769+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31626486,"num_objects":3861,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9063,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9005","reported_seq":9124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.592664+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:11.592664+0000","last_peered":"2024-09-18T11:51:11.592664+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:11.592664+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9005,"log_dups_size":0,"ondisk_log_size":9005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31864054,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8980","reported_seq":9097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.486364+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:11.486364+0000","last_peered":"2024-09-18T11:51:11.486364+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:11.486364+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31178752,"num_objects":3806,"num_object_clones":0,"num_object_copies":15224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9139","reported_seq":9244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.864190+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:12.864190+0000","last_peered":"2024-09-18T11:51:12.864190+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:12.864190+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9139,"log_dups_size":0,"ondisk_log_size":9139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31757558,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1761}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8936","reported_seq":9041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.524312+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:12.524312+0000","last_peered":"2024-09-18T11:51:12.524312+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:12.524312+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8936,"log_dups_size":0,"ondisk_log_size":8936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32265462,"num_objects":3939,"num_object_clones":0,"num_object_copies":15756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8934,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1925}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9229","reported_seq":11679,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.749149+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:11.749149+0000","last_peered":"2024-09-18T11:51:11.749149+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:11.749149+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9229,"log_dups_size":0,"ondisk_log_size":9229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32328172,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1198,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8758","reported_seq":8870,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.380110+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:12.380110+0000","last_peered":"2024-09-18T11:51:12.380110+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:12.380110+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8758,"log_dups_size":0,"ondisk_log_size":8758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31421686,"num_objects":3836,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1873}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9070","reported_seq":9186,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.777246+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:11.777246+0000","last_peered":"2024-09-18T11:51:11.777246+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:11.777246+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9070,"log_dups_size":0,"ondisk_log_size":9070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32460657,"num_objects":3963,"num_object_clones":0,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9067,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9109","reported_seq":9200,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.738204+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:11.738204+0000","last_peered":"2024-09-18T11:51:11.738204+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:11.738204+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9109,"log_dups_size":0,"ondisk_log_size":9109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32127611,"num_objects":3922,"num_object_clones":0,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1838}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9032","reported_seq":9128,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.959100+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:11.959100+0000","last_peered":"2024-09-18T11:51:11.959100+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:11.959100+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9032,"log_dups_size":0,"ondisk_log_size":9032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32234107,"num_objects":3935,"num_object_clones":0,"num_object_copies":15740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9031,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9177","reported_seq":9300,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:11.683218+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:11.683218+0000","last_peered":"2024-09-18T11:51:11.683218+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:11.683218+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9177,"log_dups_size":0,"ondisk_log_size":9177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32052470,"num_objects":3913,"num_object_clones":0,"num_object_copies":15652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9175,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1790}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9026","reported_seq":13313,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:12.040288+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:12.040288+0000","last_peered":"2024-09-18T11:51:12.040288+0000","last_clean":"2024-09-18T11:51:12.040288+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:12.040288+0000","last_undegraded":"2024-09-18T11:51:12.040288+0000","last_fullsized":"2024-09-18T11:51:12.040288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9026,"log_dups_size":0,"ondisk_log_size":9026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31757558,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9024,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8841","reported_seq":8929,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.896724+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:12.896724+0000","last_peered":"2024-09-18T11:51:12.896724+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:12.896724+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8841,"log_dups_size":0,"ondisk_log_size":8841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31667446,"num_objects":3866,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8839,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1864}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":511550860,"num_objects":62451,"num_object_clones":0,"num_object_copies":249804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144636,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5696,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1023037440,"data_stored":1023037440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144663,"ondisk_log_size":144663,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548908,"kb_used_data":251040,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93822932,"statfs":{"total":96636764160,"available":96074682368,"internally_reserved":0,"allocated":257064960,"data_stored":256889494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,3,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.1589999999999998},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.9219999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":0.88100000000000001},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":3.121}]}]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560300,"kb_used_data":250468,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93811540,"statfs":{"total":96636764160,"available":96063016960,"internally_reserved":0,"allocated":256479232,"data_stored":256303766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.253},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":7.4379999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":13.675000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.7250000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.9750000000000001},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":4.25}]}]},{"osd":1,"up_from":29,"seq":124554051662,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561348,"kb_used_data":251068,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93810492,"statfs":{"total":96636764160,"available":96061943808,"internally_reserved":0,"allocated":257093632,"data_stored":256918166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.7230000000000001},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.1830000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":3.0299999999999998},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.96199999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.99399999999999999},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":8.6839999999999993}]}]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556212,"kb_used_data":250092,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93815628,"statfs":{"total":96636764160,"available":96067203072,"internally_reserved":0,"allocated":256094208,"data_stored":255918742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":6.8600000000000003},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.9800000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":16.535},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.218}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":6.1719999999999997},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":255143936,"data_stored":255143936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256217088,"data_stored":256217088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":255565824,"data_stored":255565824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":256110592,"data_stored":256110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:17.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:18.058 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:18.058 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:18.081 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-18T11:51:18.001685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":504628620,"num_objects":61606,"num_object_clones":0,"num_object_copies":246424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145481,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145508,"ondisk_log_size":145508,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2192656,"kb_used_data":989344,"kb_used_omap":14,"kb_used_meta":1203249,"kb_avail":375294704,"statfs":{"total":386547056640,"available":384301776896,"internally_reserved":0,"allocated":1013088256,"data_stored":1012386392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232127003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,6,8,14,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25255936,"num_objects":-3083,"num_object_clones":0,"num_object_copies":-12332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012854"},"pg_stats":[{"pgid":"1.f","version":"35'9327","reported_seq":9426,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.773797+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:16.773797+0000","last_peered":"2024-09-18T11:51:16.773797+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:16.773797+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9327,"log_dups_size":0,"ondisk_log_size":9327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32137216,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1820}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9193","reported_seq":13483,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:17.114275+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:17.114275+0000","last_peered":"2024-09-18T11:51:17.114275+0000","last_clean":"2024-09-18T11:51:17.114275+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:17.114275+0000","last_undegraded":"2024-09-18T11:51:17.114275+0000","last_fullsized":"2024-09-18T11:51:17.114275+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9193,"log_dups_size":0,"ondisk_log_size":9193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642870,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8987","reported_seq":9100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.485673+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:17.485673+0000","last_peered":"2024-09-18T11:51:17.485673+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:17.485673+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31300243,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8986,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9065","reported_seq":9157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.883769+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:12.883769+0000","last_peered":"2024-09-18T11:51:12.883769+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:12.883769+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31626486,"num_objects":3861,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9063,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9073","reported_seq":9192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.713189+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:16.713189+0000","last_peered":"2024-09-18T11:51:16.713189+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:16.713189+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9073,"log_dups_size":0,"ondisk_log_size":9073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31306998,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9071,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9059","reported_seq":9173,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.655313+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:16.655313+0000","last_peered":"2024-09-18T11:51:16.655313+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:16.655313+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9059,"log_dups_size":0,"ondisk_log_size":9059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30531584,"num_objects":3727,"num_object_clones":0,"num_object_copies":14908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9059,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9139","reported_seq":9244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.864190+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:12.864190+0000","last_peered":"2024-09-18T11:51:12.864190+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:12.864190+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9139,"log_dups_size":0,"ondisk_log_size":9139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31757558,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1761}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8936","reported_seq":9041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.524312+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:12.524312+0000","last_peered":"2024-09-18T11:51:12.524312+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:12.524312+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8936,"log_dups_size":0,"ondisk_log_size":8936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32265462,"num_objects":3939,"num_object_clones":0,"num_object_copies":15756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8934,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1925}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9290","reported_seq":11851,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.666808+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:16.666808+0000","last_peered":"2024-09-18T11:51:16.666808+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:16.666808+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31828460,"num_objects":3886,"num_object_clones":0,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1252,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8832","reported_seq":8944,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.233011+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:17.233011+0000","last_peered":"2024-09-18T11:51:17.233011+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:17.233011+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8832,"log_dups_size":0,"ondisk_log_size":8832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30815478,"num_objects":3762,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8830,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1799}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9128","reported_seq":9247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.651054+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:16.651054+0000","last_peered":"2024-09-18T11:51:16.651054+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:16.651054+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31985521,"num_objects":3905,"num_object_clones":0,"num_object_copies":15620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9182","reported_seq":9273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.644808+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:16.644808+0000","last_peered":"2024-09-18T11:51:16.644808+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:16.644808+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31529595,"num_objects":3849,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9181,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9099","reported_seq":9195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.354306+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:17.354306+0000","last_peered":"2024-09-18T11:51:17.354306+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:17.354306+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31685243,"num_objects":3868,"num_object_clones":0,"num_object_copies":15472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9098,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1822}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9256","reported_seq":9379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.440328+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:16.440328+0000","last_peered":"2024-09-18T11:51:16.440328+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:16.440328+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9256,"log_dups_size":0,"ondisk_log_size":9256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31405302,"num_objects":3834,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9254,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1711}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9101","reported_seq":13388,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:17.272246+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:17.272246+0000","last_peered":"2024-09-18T11:51:17.272246+0000","last_clean":"2024-09-18T11:51:17.272246+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:17.272246+0000","last_undegraded":"2024-09-18T11:51:17.272246+0000","last_fullsized":"2024-09-18T11:51:17.272246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9101,"log_dups_size":0,"ondisk_log_size":9101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143158,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9099,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8841","reported_seq":8929,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.896724+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:12.896724+0000","last_peered":"2024-09-18T11:51:12.896724+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:12.896724+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8841,"log_dups_size":0,"ondisk_log_size":8841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31667446,"num_objects":3866,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8839,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1864}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":504628620,"num_objects":61606,"num_object_clones":0,"num_object_copies":246424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145481,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1009344512,"data_stored":1009344512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145508,"ondisk_log_size":145508,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548628,"kb_used_data":246596,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93823212,"statfs":{"total":96636764160,"available":96074969088,"internally_reserved":0,"allocated":252514304,"data_stored":252338838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":18.882000000000001},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":5.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.452999999999999},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":10.417999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":1.9850000000000001},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":9.9740000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543340,"kb_used_data":245988,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93828500,"statfs":{"total":96636764160,"available":96080384000,"internally_reserved":0,"allocated":251891712,"data_stored":251716246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":10.779},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":10.816000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":1.9390000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.0129999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.7170000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544476,"kb_used_data":246668,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93827364,"statfs":{"total":96636764160,"available":96079220736,"internally_reserved":0,"allocated":252588032,"data_stored":252412566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.4059999999999999},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.9889999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":10.962999999999999},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.8289999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":15.193},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":3.3359999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556212,"kb_used_data":250092,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93815628,"statfs":{"total":96636764160,"available":96067203072,"internally_reserved":0,"allocated":256094208,"data_stored":255918742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":6.8600000000000003},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.9800000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":16.535},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.218}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":6.1719999999999997},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":255143936,"data_stored":255143936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251576320,"data_stored":251576320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250994688,"data_stored":250994688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251629568,"data_stored":251629568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:18.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:18.654 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:18.655 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:18.676 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-18T11:51:18.001685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":504628620,"num_objects":61606,"num_object_clones":0,"num_object_copies":246424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145481,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145508,"ondisk_log_size":145508,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2192656,"kb_used_data":989344,"kb_used_omap":14,"kb_used_meta":1203249,"kb_avail":375294704,"statfs":{"total":386547056640,"available":384301776896,"internally_reserved":0,"allocated":1013088256,"data_stored":1012386392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232127003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,6,8,14,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25255936,"num_objects":-3083,"num_object_clones":0,"num_object_copies":-12332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012854"},"pg_stats":[{"pgid":"1.f","version":"35'9327","reported_seq":9426,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.773797+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:16.773797+0000","last_peered":"2024-09-18T11:51:16.773797+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:16.773797+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9327,"log_dups_size":0,"ondisk_log_size":9327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32137216,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1820}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9193","reported_seq":13483,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:17.114275+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:17.114275+0000","last_peered":"2024-09-18T11:51:17.114275+0000","last_clean":"2024-09-18T11:51:17.114275+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:17.114275+0000","last_undegraded":"2024-09-18T11:51:17.114275+0000","last_fullsized":"2024-09-18T11:51:17.114275+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9193,"log_dups_size":0,"ondisk_log_size":9193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642870,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8987","reported_seq":9100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.485673+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:17.485673+0000","last_peered":"2024-09-18T11:51:17.485673+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:17.485673+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31300243,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8986,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9065","reported_seq":9157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.883769+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:12.883769+0000","last_peered":"2024-09-18T11:51:12.883769+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:12.883769+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31626486,"num_objects":3861,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9063,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9073","reported_seq":9192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.713189+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:16.713189+0000","last_peered":"2024-09-18T11:51:16.713189+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:16.713189+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9073,"log_dups_size":0,"ondisk_log_size":9073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31306998,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9071,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9059","reported_seq":9173,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.655313+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:16.655313+0000","last_peered":"2024-09-18T11:51:16.655313+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:16.655313+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9059,"log_dups_size":0,"ondisk_log_size":9059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30531584,"num_objects":3727,"num_object_clones":0,"num_object_copies":14908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9059,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9139","reported_seq":9244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.864190+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:12.864190+0000","last_peered":"2024-09-18T11:51:12.864190+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:12.864190+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9139,"log_dups_size":0,"ondisk_log_size":9139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31757558,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1761}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8936","reported_seq":9041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.524312+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:12.524312+0000","last_peered":"2024-09-18T11:51:12.524312+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:12.524312+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8936,"log_dups_size":0,"ondisk_log_size":8936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32265462,"num_objects":3939,"num_object_clones":0,"num_object_copies":15756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8934,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1925}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9290","reported_seq":11851,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.666808+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:16.666808+0000","last_peered":"2024-09-18T11:51:16.666808+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:16.666808+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31828460,"num_objects":3886,"num_object_clones":0,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1252,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8832","reported_seq":8944,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.233011+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:17.233011+0000","last_peered":"2024-09-18T11:51:17.233011+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:17.233011+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8832,"log_dups_size":0,"ondisk_log_size":8832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30815478,"num_objects":3762,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8830,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1799}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9128","reported_seq":9247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.651054+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:16.651054+0000","last_peered":"2024-09-18T11:51:16.651054+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:16.651054+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31985521,"num_objects":3905,"num_object_clones":0,"num_object_copies":15620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9182","reported_seq":9273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.644808+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:16.644808+0000","last_peered":"2024-09-18T11:51:16.644808+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:16.644808+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31529595,"num_objects":3849,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9181,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9099","reported_seq":9195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.354306+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:17.354306+0000","last_peered":"2024-09-18T11:51:17.354306+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:17.354306+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31685243,"num_objects":3868,"num_object_clones":0,"num_object_copies":15472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9098,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1822}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9256","reported_seq":9379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.440328+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:16.440328+0000","last_peered":"2024-09-18T11:51:16.440328+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:16.440328+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9256,"log_dups_size":0,"ondisk_log_size":9256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31405302,"num_objects":3834,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9254,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1711}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9101","reported_seq":13388,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:17.272246+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:17.272246+0000","last_peered":"2024-09-18T11:51:17.272246+0000","last_clean":"2024-09-18T11:51:17.272246+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:17.272246+0000","last_undegraded":"2024-09-18T11:51:17.272246+0000","last_fullsized":"2024-09-18T11:51:17.272246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9101,"log_dups_size":0,"ondisk_log_size":9101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143158,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9099,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8841","reported_seq":8929,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:12.896724+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:12.896724+0000","last_peered":"2024-09-18T11:51:12.896724+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:12.896724+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8841,"log_dups_size":0,"ondisk_log_size":8841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31667446,"num_objects":3866,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8839,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1864}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":504628620,"num_objects":61606,"num_object_clones":0,"num_object_copies":246424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145481,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1009344512,"data_stored":1009344512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145508,"ondisk_log_size":145508,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548628,"kb_used_data":246596,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93823212,"statfs":{"total":96636764160,"available":96074969088,"internally_reserved":0,"allocated":252514304,"data_stored":252338838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":18.882000000000001},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":5.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.452999999999999},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":10.417999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":1.9850000000000001},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":9.9740000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543340,"kb_used_data":245988,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93828500,"statfs":{"total":96636764160,"available":96080384000,"internally_reserved":0,"allocated":251891712,"data_stored":251716246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":10.779},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":10.816000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":1.9390000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.0129999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.7170000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544476,"kb_used_data":246668,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93827364,"statfs":{"total":96636764160,"available":96079220736,"internally_reserved":0,"allocated":252588032,"data_stored":252412566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.4059999999999999},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.9889999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":10.962999999999999},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.8289999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":15.193},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":3.3359999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556212,"kb_used_data":250092,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93815628,"statfs":{"total":96636764160,"available":96067203072,"internally_reserved":0,"allocated":256094208,"data_stored":255918742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":6.8600000000000003},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":6.9800000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":16.535},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":3.218}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":6.1719999999999997},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":255143936,"data_stored":255143936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251576320,"data_stored":251576320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250994688,"data_stored":250994688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251629568,"data_stored":251629568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:18.677 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:18.678 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9327", 'reported_seq': 9426, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:16.773797+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:16.773797+0000', 'last_peered': '2024-09-18T11:51:16.773797+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:16.773797+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9327, 'log_dups_size': 0, 'ondisk_log_size': 9327, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32137216, 'num_objects': 3923, 'num_object_clones': 0, 'num_object_copies': 15692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9327, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1820}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:18.678 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:18.678 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8987", 'reported_seq': 9100, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:17.485673+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:17.485673+0000', 'last_peered': '2024-09-18T11:51:17.485673+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:17.485673+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8987, 'log_dups_size': 0, 'ondisk_log_size': 8987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31300243, 'num_objects': 3822, 'num_object_clones': 0, 'num_object_copies': 15288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3822, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8986, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1782}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:18.678 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:18.678 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9065", 'reported_seq': 9157, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:12.883769+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:12.883769+0000', 'last_peered': '2024-09-18T11:51:12.883769+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:12.883769+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9065, 'log_dups_size': 0, 'ondisk_log_size': 9065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31626486, 'num_objects': 3861, 'num_object_clones': 0, 'num_object_copies': 15444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9063, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1760}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:18.678 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:18.678 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9073", 'reported_seq': 9192, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:16.713189+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:16.713189+0000', 'last_peered': '2024-09-18T11:51:16.713189+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:16.713189+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9073, 'log_dups_size': 0, 'ondisk_log_size': 9073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31306998, 'num_objects': 3822, 'num_object_clones': 0, 'num_object_copies': 15288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9071, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1770}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:18.678 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:18.678 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9059", 'reported_seq': 9173, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:16.655313+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:16.655313+0000', 'last_peered': '2024-09-18T11:51:16.655313+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:16.655313+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9059, 'log_dups_size': 0, 'ondisk_log_size': 9059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30531584, 'num_objects': 3727, 'num_object_clones': 0, 'num_object_copies': 14908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9059, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1608}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:18.679 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:18.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9139", 'reported_seq': 9244, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:12.864190+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:12.864190+0000', 'last_peered': '2024-09-18T11:51:12.864190+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:12.864190+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9139, 'log_dups_size': 0, 'ondisk_log_size': 9139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31757558, 'num_objects': 3877, 'num_object_clones': 0, 'num_object_copies': 15508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9137, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1761}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:18.679 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:18.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8936", 'reported_seq': 9041, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:12.524312+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:12.524312+0000', 'last_peered': '2024-09-18T11:51:12.524312+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:12.524312+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8936, 'log_dups_size': 0, 'ondisk_log_size': 8936, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32265462, 'num_objects': 3939, 'num_object_clones': 0, 'num_object_copies': 15756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8934, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1925}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:18.679 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:18.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9290", 'reported_seq': 11851, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:16.666808+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:16.666808+0000', 'last_peered': '2024-09-18T11:51:16.666808+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:16.666808+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9290, 'log_dups_size': 0, 'ondisk_log_size': 9290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31828460, 'num_objects': 3886, 'num_object_clones': 0, 'num_object_copies': 15544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 889, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9287, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1252, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2997}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:18.679 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:18.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8832", 'reported_seq': 8944, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:17.233011+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:17.233011+0000', 'last_peered': '2024-09-18T11:51:17.233011+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:17.233011+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8832, 'log_dups_size': 0, 'ondisk_log_size': 8832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30815478, 'num_objects': 3762, 'num_object_clones': 0, 'num_object_copies': 15048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8830, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1799}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:18.679 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:18.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9128", 'reported_seq': 9247, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:16.651054+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:16.651054+0000', 'last_peered': '2024-09-18T11:51:16.651054+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:16.651054+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9128, 'log_dups_size': 0, 'ondisk_log_size': 9128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31985521, 'num_objects': 3905, 'num_object_clones': 0, 'num_object_copies': 15620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9125, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1871}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:18.680 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:18.680 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9182", 'reported_seq': 9273, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:16.644808+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:16.644808+0000', 'last_peered': '2024-09-18T11:51:16.644808+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:16.644808+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9182, 'log_dups_size': 0, 'ondisk_log_size': 9182, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31529595, 'num_objects': 3849, 'num_object_clones': 0, 'num_object_copies': 15396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9181, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:18.680 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:18.680 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9099", 'reported_seq': 9195, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:17.354306+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:17.354306+0000', 'last_peered': '2024-09-18T11:51:17.354306+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:17.354306+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9099, 'log_dups_size': 0, 'ondisk_log_size': 9099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31685243, 'num_objects': 3868, 'num_object_clones': 0, 'num_object_copies': 15472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9098, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1822}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:18.680 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:18.680 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9256", 'reported_seq': 9379, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:16.440328+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:16.440328+0000', 'last_peered': '2024-09-18T11:51:16.440328+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:16.440328+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9256, 'log_dups_size': 0, 'ondisk_log_size': 9256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31405302, 'num_objects': 3834, 'num_object_clones': 0, 'num_object_copies': 15336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9254, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1711}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:18.680 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:18.680 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8841", 'reported_seq': 8929, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:12.896724+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:12.896724+0000', 'last_peered': '2024-09-18T11:51:12.896724+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:12.896724+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8841, 'log_dups_size': 0, 'ondisk_log_size': 8841, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31667446, 'num_objects': 3866, 'num_object_clones': 0, 'num_object_copies': 15464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8839, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1864}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:18.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:19.419 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27706/246424 objects degraded (11.243%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":477,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":61606,"data_bytes":504628620,"bytes_used":2245279744,"bytes_avail":384301776896,"bytes_total":386547056640,"degraded_objects":27706,"degraded_total":246424,"degraded_ratio":0.11243223062688699,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":256},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:19.420 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:19.420 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:20.015 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:20.015 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:20.031 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-09-18T11:51:20.002141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502506892,"num_objects":61347,"num_object_clones":0,"num_object_copies":245388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145740,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145767,"ondisk_log_size":145767,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2192304,"kb_used_data":984832,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375295056,"statfs":{"total":386547056640,"available":384302137344,"internally_reserved":0,"allocated":1008467968,"data_stored":1007766104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,9,7,9,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18169856,"num_objects":-2218,"num_object_clones":0,"num_object_copies":-8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2218,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012673"},"pg_stats":[{"pgid":"1.f","version":"35'9327","reported_seq":9426,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.773797+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:16.773797+0000","last_peered":"2024-09-18T11:51:16.773797+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:16.773797+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9327,"log_dups_size":0,"ondisk_log_size":9327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32137216,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1820}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9193","reported_seq":13483,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:17.114275+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:17.114275+0000","last_peered":"2024-09-18T11:51:17.114275+0000","last_clean":"2024-09-18T11:51:17.114275+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:17.114275+0000","last_undegraded":"2024-09-18T11:51:17.114275+0000","last_fullsized":"2024-09-18T11:51:17.114275+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9193,"log_dups_size":0,"ondisk_log_size":9193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642870,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8987","reported_seq":9100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.485673+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:17.485673+0000","last_peered":"2024-09-18T11:51:17.485673+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:17.485673+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31300243,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8986,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9131","reported_seq":9223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.968727+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:17.968727+0000","last_peered":"2024-09-18T11:51:17.968727+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:17.968727+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9131,"log_dups_size":0,"ondisk_log_size":9131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31085814,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9129,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9073","reported_seq":9192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.713189+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:16.713189+0000","last_peered":"2024-09-18T11:51:16.713189+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:16.713189+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9073,"log_dups_size":0,"ondisk_log_size":9073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31306998,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9071,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9059","reported_seq":9173,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.655313+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:16.655313+0000","last_peered":"2024-09-18T11:51:16.655313+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:16.655313+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9059,"log_dups_size":0,"ondisk_log_size":9059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30531584,"num_objects":3727,"num_object_clones":0,"num_object_copies":14908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9059,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9210","reported_seq":9315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.943620+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:17.943620+0000","last_peered":"2024-09-18T11:51:17.943620+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:17.943620+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9210,"log_dups_size":0,"ondisk_log_size":9210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31175926,"num_objects":3806,"num_object_clones":0,"num_object_copies":15224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9208,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1690}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9006","reported_seq":9113,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.850888+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:17.850888+0000","last_peered":"2024-09-18T11:51:17.850888+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:17.850888+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31692022,"num_objects":3869,"num_object_clones":0,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9004,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9290","reported_seq":11851,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.666808+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:16.666808+0000","last_peered":"2024-09-18T11:51:16.666808+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:16.666808+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31828460,"num_objects":3886,"num_object_clones":0,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1252,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8832","reported_seq":8944,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.233011+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:17.233011+0000","last_peered":"2024-09-18T11:51:17.233011+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:17.233011+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8832,"log_dups_size":0,"ondisk_log_size":8832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30815478,"num_objects":3762,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8830,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1799}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9128","reported_seq":9247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.651054+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:16.651054+0000","last_peered":"2024-09-18T11:51:16.651054+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:16.651054+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31985521,"num_objects":3905,"num_object_clones":0,"num_object_copies":15620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9182","reported_seq":9273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.644808+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:16.644808+0000","last_peered":"2024-09-18T11:51:16.644808+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:16.644808+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31529595,"num_objects":3849,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9181,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9099","reported_seq":9195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.354306+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:17.354306+0000","last_peered":"2024-09-18T11:51:17.354306+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:17.354306+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31685243,"num_objects":3868,"num_object_clones":0,"num_object_copies":15472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9098,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1822}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9256","reported_seq":9379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:16.440328+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:16.440328+0000","last_peered":"2024-09-18T11:51:16.440328+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:16.440328+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9256,"log_dups_size":0,"ondisk_log_size":9256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31405302,"num_objects":3834,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9254,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1711}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9101","reported_seq":13388,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:17.272246+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:17.272246+0000","last_peered":"2024-09-18T11:51:17.272246+0000","last_clean":"2024-09-18T11:51:17.272246+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:17.272246+0000","last_undegraded":"2024-09-18T11:51:17.272246+0000","last_fullsized":"2024-09-18T11:51:17.272246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9101,"log_dups_size":0,"ondisk_log_size":9101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143158,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9099,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8893","reported_seq":8981,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.766017+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:17.766017+0000","last_peered":"2024-09-18T11:51:17.766017+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:17.766017+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8893,"log_dups_size":0,"ondisk_log_size":8893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31241462,"num_objects":3814,"num_object_clones":0,"num_object_copies":15256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8891,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1812}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":502506892,"num_objects":61347,"num_object_clones":0,"num_object_copies":245388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145740,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1004740608,"data_stored":1004740608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145767,"ondisk_log_size":145767,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548628,"kb_used_data":246596,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93823212,"statfs":{"total":96636764160,"available":96074969088,"internally_reserved":0,"allocated":252514304,"data_stored":252338838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.0090000000000003,"5min":5.2530000000000001,"15min":3.6190000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.155},"max":{"1min":19.792000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":18.882000000000001},{"interface":"front","average":{"1min":6.5330000000000004,"5min":5.5650000000000004,"15min":3.6379999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":19.847999999999999,"5min":31.675000000000001,"15min":31.675000000000001},"last":5.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.452999999999999},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":10.417999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":5.2380000000000004,"15min":2.4039999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":19.866,"5min":29.087,"15min":29.087},"last":1.9850000000000001},{"interface":"front","average":{"1min":3.294,"5min":4.5529999999999999,"15min":3.2450000000000001},"min":{"1min":0.26200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":13.952,"5min":21.213000000000001,"15min":21.213000000000001},"last":9.9740000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543340,"kb_used_data":245988,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93828500,"statfs":{"total":96636764160,"available":96080384000,"internally_reserved":0,"allocated":251891712,"data_stored":251716246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":10.779},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":10.816000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":1.9390000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.0129999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.7170000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544476,"kb_used_data":246668,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93827364,"statfs":{"total":96636764160,"available":96079220736,"internally_reserved":0,"allocated":252588032,"data_stored":252412566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.4059999999999999},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.9889999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":10.962999999999999},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.8289999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":15.193},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":3.3359999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555860,"kb_used_data":245580,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93815980,"statfs":{"total":96636764160,"available":96067563520,"internally_reserved":0,"allocated":251473920,"data_stored":251298454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":3.488},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":7.9340000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.1360000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":6.0940000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":250540032,"data_stored":250540032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251576320,"data_stored":251576320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250994688,"data_stored":250994688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251629568,"data_stored":251629568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:23.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:23.639 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:23.639 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:23.662 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-18T11:51:22.006283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":498410892,"num_objects":60847,"num_object_clones":0,"num_object_copies":243388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146240,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5832,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146267,"ondisk_log_size":146267,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2192100,"kb_used_data":980476,"kb_used_omap":14,"kb_used_meta":1211569,"kb_avail":375295260,"statfs":{"total":386547056640,"available":384302346240,"internally_reserved":0,"allocated":1004007424,"data_stored":1003305560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240646683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,2,8,10,12,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22265856,"num_objects":-2718,"num_object_clones":0,"num_object_copies":-10872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.016164"},"pg_stats":[{"pgid":"1.f","version":"35'9407","reported_seq":9506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.870174+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:21.870174+0000","last_peered":"2024-09-18T11:51:21.870174+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:21.870174+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31481856,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9193","reported_seq":13483,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:17.114275+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:17.114275+0000","last_peered":"2024-09-18T11:51:17.114275+0000","last_clean":"2024-09-18T11:51:17.114275+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:17.114275+0000","last_undegraded":"2024-09-18T11:51:17.114275+0000","last_fullsized":"2024-09-18T11:51:17.114275+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9193,"log_dups_size":0,"ondisk_log_size":9193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642870,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'8987","reported_seq":9100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.485673+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:17.485673+0000","last_peered":"2024-09-18T11:51:17.485673+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:17.485673+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31300243,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8986,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9131","reported_seq":9223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.968727+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:17.968727+0000","last_peered":"2024-09-18T11:51:17.968727+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:17.968727+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9131,"log_dups_size":0,"ondisk_log_size":9131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31085814,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9129,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9153","reported_seq":9269,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.835851+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:21.835851+0000","last_peered":"2024-09-18T11:51:21.835851+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:21.835851+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30651638,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9124","reported_seq":9241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.431206+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:21.431206+0000","last_peered":"2024-09-18T11:51:21.431206+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:21.431206+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29999104,"num_objects":3662,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9124,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9210","reported_seq":9315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.943620+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:17.943620+0000","last_peered":"2024-09-18T11:51:17.943620+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:17.943620+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9210,"log_dups_size":0,"ondisk_log_size":9210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31175926,"num_objects":3806,"num_object_clones":0,"num_object_copies":15224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9208,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1690}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9006","reported_seq":9113,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.850888+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:17.850888+0000","last_peered":"2024-09-18T11:51:17.850888+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:17.850888+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31692022,"num_objects":3869,"num_object_clones":0,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9004,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9364","reported_seq":12089,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.900308+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:21.900308+0000","last_peered":"2024-09-18T11:51:21.900308+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:21.900308+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31222252,"num_objects":3812,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9361,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1334,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8832","reported_seq":8944,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.233011+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:17.233011+0000","last_peered":"2024-09-18T11:51:17.233011+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:17.233011+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8832,"log_dups_size":0,"ondisk_log_size":8832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30815478,"num_objects":3762,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8830,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1799}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9196","reported_seq":9315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.446051+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:21.446051+0000","last_peered":"2024-09-18T11:51:21.446051+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:21.446051+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9196,"log_dups_size":0,"ondisk_log_size":9196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31428465,"num_objects":3837,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9193,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9251","reported_seq":9341,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.871367+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:21.871367+0000","last_peered":"2024-09-18T11:51:21.871367+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:21.871367+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30964347,"num_objects":3780,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1696}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9099","reported_seq":9195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.354306+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:17.354306+0000","last_peered":"2024-09-18T11:51:17.354306+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:17.354306+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31685243,"num_objects":3868,"num_object_clones":0,"num_object_copies":15472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9098,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1822}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9320","reported_seq":9440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.767260+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:21.767260+0000","last_peered":"2024-09-18T11:51:21.767260+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:21.767260+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30881014,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9318,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1647}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9101","reported_seq":13388,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:17.272246+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:17.272246+0000","last_peered":"2024-09-18T11:51:17.272246+0000","last_clean":"2024-09-18T11:51:17.272246+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:17.272246+0000","last_undegraded":"2024-09-18T11:51:17.272246+0000","last_fullsized":"2024-09-18T11:51:17.272246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9101,"log_dups_size":0,"ondisk_log_size":9101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143158,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9099,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8893","reported_seq":8981,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:17.766017+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:17.766017+0000","last_peered":"2024-09-18T11:51:17.766017+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:17.766017+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8893,"log_dups_size":0,"ondisk_log_size":8893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31241462,"num_objects":3814,"num_object_clones":0,"num_object_copies":15256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8891,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1812}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":498410892,"num_objects":60847,"num_object_clones":0,"num_object_copies":243388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146240,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5832,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1000284160,"data_stored":1000284160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146267,"ondisk_log_size":146267,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548424,"kb_used_data":242240,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93823416,"statfs":{"total":96636764160,"available":96075177984,"internally_reserved":0,"allocated":248053760,"data_stored":247878294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.922000000000001},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":10.991}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.956},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":5.3019999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.0659999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.2670000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543340,"kb_used_data":245988,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93828500,"statfs":{"total":96636764160,"available":96080384000,"internally_reserved":0,"allocated":251891712,"data_stored":251716246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":10.779},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":10.816000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":1.9390000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.0129999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:50:17 2024","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":4.1280000000000001,"15min":2.9729999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.001000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.2189999999999999,"15min":2.206},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":9.8650000000000002,"5min":16.673999999999999,"15min":16.673999999999999},"last":1.7170000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544476,"kb_used_data":246668,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93827364,"statfs":{"total":96636764160,"available":96079220736,"internally_reserved":0,"allocated":252588032,"data_stored":252412566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.4059999999999999},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.9889999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":10.962999999999999},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.8289999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":15.193},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":3.3359999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555860,"kb_used_data":245580,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93815980,"statfs":{"total":96636764160,"available":96067563520,"internally_reserved":0,"allocated":251473920,"data_stored":251298454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":3.488},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":7.9340000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":6.1360000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:50:18 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":3.9740000000000002,"15min":2.8980000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.149999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":8.468,"5min":5.7869999999999999,"15min":2.9430000000000001},"min":{"1min":0.66600000000000004,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":19.129999999999999,"5min":19.129999999999999,"15min":19.129999999999999},"last":6.0940000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":250540032,"data_stored":250540032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251576320,"data_stored":251576320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250994688,"data_stored":250994688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247173120,"data_stored":247173120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:23.663 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:24.225 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:24.225 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:24.250 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-18T11:51:24.006949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493593996,"num_objects":60259,"num_object_clones":0,"num_object_copies":241036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60259,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146828,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5832,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146855,"ondisk_log_size":146855,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2175076,"kb_used_data":967600,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375312284,"statfs":{"total":386547056640,"available":384319778816,"internally_reserved":0,"allocated":990822400,"data_stored":990120536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,1,8,6,16,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22978560,"num_objects":-2805,"num_object_clones":0,"num_object_copies":-11220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2805,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008074"},"pg_stats":[{"pgid":"1.f","version":"35'9407","reported_seq":9506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.870174+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:21.870174+0000","last_peered":"2024-09-18T11:51:21.870174+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:21.870174+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31481856,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9249","reported_seq":13539,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:22.020252+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:22.020252+0000","last_peered":"2024-09-18T11:51:22.020252+0000","last_clean":"2024-09-18T11:51:22.020252+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:22.020252+0000","last_undegraded":"2024-09-18T11:51:22.020252+0000","last_fullsized":"2024-09-18T11:51:22.020252+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31184118,"num_objects":3807,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9059","reported_seq":9174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.422859+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:22.422859+0000","last_peered":"2024-09-18T11:51:22.422859+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:22.422859+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9059,"log_dups_size":0,"ondisk_log_size":9059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30710419,"num_objects":3750,"num_object_clones":0,"num_object_copies":15000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9058,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9195","reported_seq":9287,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:23.055650+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:23.055650+0000","last_peered":"2024-09-18T11:51:23.055650+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:23.055650+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9195,"log_dups_size":0,"ondisk_log_size":9195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30561526,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9193,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9153","reported_seq":9269,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.835851+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:21.835851+0000","last_peered":"2024-09-18T11:51:21.835851+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:21.835851+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30651638,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9124","reported_seq":9241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.431206+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:21.431206+0000","last_peered":"2024-09-18T11:51:21.431206+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:21.431206+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29999104,"num_objects":3662,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9124,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9283","reported_seq":9388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.938683+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:22.938683+0000","last_peered":"2024-09-18T11:51:22.938683+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:22.938683+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30577910,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9075","reported_seq":9182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.811269+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:22.811269+0000","last_peered":"2024-09-18T11:51:22.811269+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:22.811269+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9075,"log_dups_size":0,"ondisk_log_size":9075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31126774,"num_objects":3800,"num_object_clones":0,"num_object_copies":15200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9073,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9364","reported_seq":12089,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.900308+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:21.900308+0000","last_peered":"2024-09-18T11:51:21.900308+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:21.900308+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31222252,"num_objects":3812,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9361,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1334,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8886","reported_seq":8998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.422795+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:22.422795+0000","last_peered":"2024-09-18T11:51:22.422795+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:22.422795+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8886,"log_dups_size":0,"ondisk_log_size":8886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30373110,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8884,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9196","reported_seq":9315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.446051+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:21.446051+0000","last_peered":"2024-09-18T11:51:21.446051+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:21.446051+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9196,"log_dups_size":0,"ondisk_log_size":9196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31428465,"num_objects":3837,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9193,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9251","reported_seq":9341,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.871367+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:21.871367+0000","last_peered":"2024-09-18T11:51:21.871367+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:21.871367+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30964347,"num_objects":3780,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1696}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9182","reported_seq":9276,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.329769+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:22.329769+0000","last_peered":"2024-09-18T11:51:22.329769+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:22.329769+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31005307,"num_objects":3785,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9181,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9320","reported_seq":9440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.767260+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:21.767260+0000","last_peered":"2024-09-18T11:51:21.767260+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:21.767260+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30881014,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9318,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1647}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9149","reported_seq":13436,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:22.213506+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:22.213506+0000","last_peered":"2024-09-18T11:51:22.213506+0000","last_clean":"2024-09-18T11:51:22.213506+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:22.213506+0000","last_undegraded":"2024-09-18T11:51:22.213506+0000","last_fullsized":"2024-09-18T11:51:22.213506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9149,"log_dups_size":0,"ondisk_log_size":9149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30749942,"num_objects":3754,"num_object_clones":0,"num_object_copies":15016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9147,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8962","reported_seq":9050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.947390+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:22.947390+0000","last_peered":"2024-09-18T11:51:22.947390+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:22.947390+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30676214,"num_objects":3745,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8960,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1743}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":493593996,"num_objects":60259,"num_object_clones":0,"num_object_copies":241036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60259,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146828,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5832,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":987045888,"data_stored":987045888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146855,"ondisk_log_size":146855,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548424,"kb_used_data":242240,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93823416,"statfs":{"total":96636764160,"available":96075177984,"internally_reserved":0,"allocated":248053760,"data_stored":247878294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.922000000000001},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":10.991}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.956},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":5.3019999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.0659999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.2670000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543220,"kb_used_data":241696,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93828620,"statfs":{"total":96636764160,"available":96080506880,"internally_reserved":0,"allocated":247496704,"data_stored":247321238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":3.2839999999999998},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":3.3130000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.216},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.4649999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.1739999999999999},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.254}]}]},{"osd":1,"up_from":29,"seq":124554051664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544292,"kb_used_data":242332,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93827548,"statfs":{"total":96636764160,"available":96079409152,"internally_reserved":0,"allocated":248147968,"data_stored":247972502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,6,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.3330000000000002},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":8.0749999999999993}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":10.635},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.915}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":8.1159999999999997},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":10.601000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539140,"kb_used_data":241332,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93832700,"statfs":{"total":96636764160,"available":96084684800,"internally_reserved":0,"allocated":247123968,"data_stored":246948502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":28.059000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":11.105}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.83999999999999997},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":11.068}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.4450000000000003},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":28.129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246079488,"data_stored":246079488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":247209984,"data_stored":247209984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":246583296,"data_stored":246583296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247173120,"data_stored":247173120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:24.251 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:24.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9407", 'reported_seq': 9506, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:21.870174+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:21.870174+0000', 'last_peered': '2024-09-18T11:51:21.870174+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:21.870174+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9407, 'log_dups_size': 0, 'ondisk_log_size': 9407, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31481856, 'num_objects': 3843, 'num_object_clones': 0, 'num_object_copies': 15372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9407, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1740}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:24.251 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:24.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9059", 'reported_seq': 9174, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:22.422859+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:22.422859+0000', 'last_peered': '2024-09-18T11:51:22.422859+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:22.422859+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9059, 'log_dups_size': 0, 'ondisk_log_size': 9059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30710419, 'num_objects': 3750, 'num_object_clones': 0, 'num_object_copies': 15000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3750, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9058, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1710}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9195", 'reported_seq': 9287, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:23.055650+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:23.055650+0000', 'last_peered': '2024-09-18T11:51:23.055650+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:23.055650+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9195, 'log_dups_size': 0, 'ondisk_log_size': 9195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30561526, 'num_objects': 3731, 'num_object_clones': 0, 'num_object_copies': 14924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9193, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1630}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9153", 'reported_seq': 9269, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:21.835851+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:21.835851+0000', 'last_peered': '2024-09-18T11:51:21.835851+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:21.835851+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9153, 'log_dups_size': 0, 'ondisk_log_size': 9153, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30651638, 'num_objects': 3742, 'num_object_clones': 0, 'num_object_copies': 14968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9151, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1690}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9124", 'reported_seq': 9241, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:21.431206+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:21.431206+0000', 'last_peered': '2024-09-18T11:51:21.431206+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:21.431206+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9124, 'log_dups_size': 0, 'ondisk_log_size': 9124, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29999104, 'num_objects': 3662, 'num_object_clones': 0, 'num_object_copies': 14648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9124, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1543}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9283", 'reported_seq': 9388, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:22.938683+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:22.938683+0000', 'last_peered': '2024-09-18T11:51:22.938683+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:22.938683+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9283, 'log_dups_size': 0, 'ondisk_log_size': 9283, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30577910, 'num_objects': 3733, 'num_object_clones': 0, 'num_object_copies': 14932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9281, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1617}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9075", 'reported_seq': 9182, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:22.811269+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:22.811269+0000', 'last_peered': '2024-09-18T11:51:22.811269+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:22.811269+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9075, 'log_dups_size': 0, 'ondisk_log_size': 9075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31126774, 'num_objects': 3800, 'num_object_clones': 0, 'num_object_copies': 15200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9073, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1786}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:24.252 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:24.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9364", 'reported_seq': 12089, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:21.900308+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:21.900308+0000', 'last_peered': '2024-09-18T11:51:21.900308+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:21.900308+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9364, 'log_dups_size': 0, 'ondisk_log_size': 9364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31222252, 'num_objects': 3812, 'num_object_clones': 0, 'num_object_copies': 15248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 807, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9361, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1334, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3005}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:24.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:24.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8886", 'reported_seq': 8998, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:22.422795+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:22.422795+0000', 'last_peered': '2024-09-18T11:51:22.422795+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:22.422795+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8886, 'log_dups_size': 0, 'ondisk_log_size': 8886, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30373110, 'num_objects': 3708, 'num_object_clones': 0, 'num_object_copies': 14832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3708, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8884, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1745}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:24.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:24.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9196", 'reported_seq': 9315, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:21.446051+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:21.446051+0000', 'last_peered': '2024-09-18T11:51:21.446051+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:21.446051+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9196, 'log_dups_size': 0, 'ondisk_log_size': 9196, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31428465, 'num_objects': 3837, 'num_object_clones': 0, 'num_object_copies': 15348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9193, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1803}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:24.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:24.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9251", 'reported_seq': 9341, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:21.871367+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:21.871367+0000', 'last_peered': '2024-09-18T11:51:21.871367+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:21.871367+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9251, 'log_dups_size': 0, 'ondisk_log_size': 9251, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30964347, 'num_objects': 3780, 'num_object_clones': 0, 'num_object_copies': 15120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9250, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1696}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:24.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:24.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9182", 'reported_seq': 9276, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:22.329769+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:22.329769+0000', 'last_peered': '2024-09-18T11:51:22.329769+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:22.329769+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9182, 'log_dups_size': 0, 'ondisk_log_size': 9182, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31005307, 'num_objects': 3785, 'num_object_clones': 0, 'num_object_copies': 15140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9181, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1739}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:24.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:24.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9320", 'reported_seq': 9440, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:21.767260+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:21.767260+0000', 'last_peered': '2024-09-18T11:51:21.767260+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:21.767260+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9320, 'log_dups_size': 0, 'ondisk_log_size': 9320, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30881014, 'num_objects': 3770, 'num_object_clones': 0, 'num_object_copies': 15080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9318, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1647}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:24.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:24.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8962", 'reported_seq': 9050, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:22.947390+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:22.947390+0000', 'last_peered': '2024-09-18T11:51:22.947390+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:22.947390+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8962, 'log_dups_size': 0, 'ondisk_log_size': 8962, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30676214, 'num_objects': 3745, 'num_object_clones': 0, 'num_object_copies': 14980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3745, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8960, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1743}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:24.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:24.907 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:24.907 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27624/241036 objects degraded (11.461%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":482,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":60259,"data_bytes":493593996,"bytes_used":2227277824,"bytes_avail":384319778816,"bytes_total":386547056640,"degraded_objects":27624,"degraded_total":241036,"degraded_ratio":0.1146052871770192,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":233},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:24.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:25.450 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:25.450 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:25.468 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-18T11:51:24.006949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493593996,"num_objects":60259,"num_object_clones":0,"num_object_copies":241036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60259,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146828,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5832,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146855,"ondisk_log_size":146855,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2175076,"kb_used_data":967600,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375312284,"statfs":{"total":386547056640,"available":384319778816,"internally_reserved":0,"allocated":990822400,"data_stored":990120536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,1,8,6,16,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22978560,"num_objects":-2805,"num_object_clones":0,"num_object_copies":-11220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2805,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008074"},"pg_stats":[{"pgid":"1.f","version":"35'9407","reported_seq":9506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.870174+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:21.870174+0000","last_peered":"2024-09-18T11:51:21.870174+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:21.870174+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31481856,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9249","reported_seq":13539,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:22.020252+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:22.020252+0000","last_peered":"2024-09-18T11:51:22.020252+0000","last_clean":"2024-09-18T11:51:22.020252+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:22.020252+0000","last_undegraded":"2024-09-18T11:51:22.020252+0000","last_fullsized":"2024-09-18T11:51:22.020252+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31184118,"num_objects":3807,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9059","reported_seq":9174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.422859+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:22.422859+0000","last_peered":"2024-09-18T11:51:22.422859+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:22.422859+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9059,"log_dups_size":0,"ondisk_log_size":9059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30710419,"num_objects":3750,"num_object_clones":0,"num_object_copies":15000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9058,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9195","reported_seq":9287,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:23.055650+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:23.055650+0000","last_peered":"2024-09-18T11:51:23.055650+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:23.055650+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9195,"log_dups_size":0,"ondisk_log_size":9195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30561526,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9193,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9153","reported_seq":9269,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.835851+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:21.835851+0000","last_peered":"2024-09-18T11:51:21.835851+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:21.835851+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30651638,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9124","reported_seq":9241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.431206+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:21.431206+0000","last_peered":"2024-09-18T11:51:21.431206+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:21.431206+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29999104,"num_objects":3662,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9124,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9283","reported_seq":9388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.938683+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:22.938683+0000","last_peered":"2024-09-18T11:51:22.938683+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:22.938683+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30577910,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9075","reported_seq":9182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.811269+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:22.811269+0000","last_peered":"2024-09-18T11:51:22.811269+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:22.811269+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9075,"log_dups_size":0,"ondisk_log_size":9075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31126774,"num_objects":3800,"num_object_clones":0,"num_object_copies":15200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9073,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9364","reported_seq":12089,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.900308+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:21.900308+0000","last_peered":"2024-09-18T11:51:21.900308+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:21.900308+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31222252,"num_objects":3812,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9361,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1334,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8886","reported_seq":8998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.422795+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:22.422795+0000","last_peered":"2024-09-18T11:51:22.422795+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:22.422795+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8886,"log_dups_size":0,"ondisk_log_size":8886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30373110,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8884,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9196","reported_seq":9315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.446051+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:21.446051+0000","last_peered":"2024-09-18T11:51:21.446051+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:21.446051+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9196,"log_dups_size":0,"ondisk_log_size":9196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31428465,"num_objects":3837,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9193,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9251","reported_seq":9341,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.871367+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:21.871367+0000","last_peered":"2024-09-18T11:51:21.871367+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:21.871367+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30964347,"num_objects":3780,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1696}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9182","reported_seq":9276,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.329769+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:22.329769+0000","last_peered":"2024-09-18T11:51:22.329769+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:22.329769+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31005307,"num_objects":3785,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9181,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9320","reported_seq":9440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:21.767260+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:21.767260+0000","last_peered":"2024-09-18T11:51:21.767260+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:21.767260+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30881014,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9318,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1647}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9149","reported_seq":13436,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:22.213506+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:22.213506+0000","last_peered":"2024-09-18T11:51:22.213506+0000","last_clean":"2024-09-18T11:51:22.213506+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:22.213506+0000","last_undegraded":"2024-09-18T11:51:22.213506+0000","last_fullsized":"2024-09-18T11:51:22.213506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9149,"log_dups_size":0,"ondisk_log_size":9149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30749942,"num_objects":3754,"num_object_clones":0,"num_object_copies":15016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9147,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8962","reported_seq":9050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.947390+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:22.947390+0000","last_peered":"2024-09-18T11:51:22.947390+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:22.947390+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30676214,"num_objects":3745,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8960,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1743}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":493593996,"num_objects":60259,"num_object_clones":0,"num_object_copies":241036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60259,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146828,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5832,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":987045888,"data_stored":987045888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146855,"ondisk_log_size":146855,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548424,"kb_used_data":242240,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93823416,"statfs":{"total":96636764160,"available":96075177984,"internally_reserved":0,"allocated":248053760,"data_stored":247878294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.922000000000001},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":10.991}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.956},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":5.3019999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.0659999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.2670000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543220,"kb_used_data":241696,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93828620,"statfs":{"total":96636764160,"available":96080506880,"internally_reserved":0,"allocated":247496704,"data_stored":247321238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":3.2839999999999998},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":3.3130000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.216},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.4649999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.1739999999999999},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.254}]}]},{"osd":1,"up_from":29,"seq":124554051664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544292,"kb_used_data":242332,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93827548,"statfs":{"total":96636764160,"available":96079409152,"internally_reserved":0,"allocated":248147968,"data_stored":247972502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,6,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.3330000000000002},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":8.0749999999999993}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":10.635},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":3.915}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":8.1159999999999997},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":10.601000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539140,"kb_used_data":241332,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93832700,"statfs":{"total":96636764160,"available":96084684800,"internally_reserved":0,"allocated":247123968,"data_stored":246948502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":28.059000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":11.105}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.83999999999999997},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":11.068}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.4450000000000003},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":28.129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246079488,"data_stored":246079488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":247209984,"data_stored":247209984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":246583296,"data_stored":246583296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247173120,"data_stored":247173120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:28.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:29.008 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:29.009 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:29.030 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-18T11:51:28.008877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":486892940,"num_objects":59441,"num_object_clones":0,"num_object_copies":237764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59441,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147646,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5880,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147673,"ondisk_log_size":147673,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2174852,"kb_used_data":954900,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375312508,"statfs":{"total":386547056640,"available":384320008192,"internally_reserved":0,"allocated":977817600,"data_stored":977115736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,2,7,6,17,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24657920,"num_objects":-3010,"num_object_clones":0,"num_object_copies":-12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007633"},"pg_stats":[{"pgid":"1.f","version":"35'9477","reported_seq":9576,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.893879+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:26.893879+0000","last_peered":"2024-09-18T11:51:26.893879+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:26.893879+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9477,"log_dups_size":0,"ondisk_log_size":9477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30908416,"num_objects":3773,"num_object_clones":0,"num_object_copies":15092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9477,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1670}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9312","reported_seq":13602,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:27.223972+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:27.223972+0000","last_peered":"2024-09-18T11:51:27.223972+0000","last_clean":"2024-09-18T11:51:27.223972+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:27.223972+0000","last_undegraded":"2024-09-18T11:51:27.223972+0000","last_fullsized":"2024-09-18T11:51:27.223972+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30668022,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9132","reported_seq":9247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.546911+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:27.546911+0000","last_peered":"2024-09-18T11:51:27.546911+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:27.546911+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9132,"log_dups_size":0,"ondisk_log_size":9132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30112403,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9131,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1637}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9195","reported_seq":9287,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:23.055650+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:23.055650+0000","last_peered":"2024-09-18T11:51:23.055650+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:23.055650+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9195,"log_dups_size":0,"ondisk_log_size":9195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30561526,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9193,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9224","reported_seq":9343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.933925+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:26.933925+0000","last_peered":"2024-09-18T11:51:26.933925+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:26.933925+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9224,"log_dups_size":0,"ondisk_log_size":9224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30070006,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9197","reported_seq":9313,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.732672+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:26.732672+0000","last_peered":"2024-09-18T11:51:26.732672+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:26.732672+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29401088,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9197,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9283","reported_seq":9388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.938683+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:22.938683+0000","last_peered":"2024-09-18T11:51:22.938683+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:22.938683+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30577910,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9075","reported_seq":9182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.811269+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:22.811269+0000","last_peered":"2024-09-18T11:51:22.811269+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:22.811269+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9075,"log_dups_size":0,"ondisk_log_size":9075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31126774,"num_objects":3800,"num_object_clones":0,"num_object_copies":15200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9073,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9435","reported_seq":12256,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.922387+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:26.922387+0000","last_peered":"2024-09-18T11:51:26.922387+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:26.922387+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9435,"log_dups_size":0,"ondisk_log_size":9435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30640620,"num_objects":3741,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9432,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2982}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8947","reported_seq":9059,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.360314+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:27.360314+0000","last_peered":"2024-09-18T11:51:27.360314+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:27.360314+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8947,"log_dups_size":0,"ondisk_log_size":8947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873398,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8945,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1684}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9259","reported_seq":9378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.903449+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:26.903449+0000","last_peered":"2024-09-18T11:51:26.903449+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:26.903449+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30912369,"num_objects":3774,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9334","reported_seq":9425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.939491+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:26.939491+0000","last_peered":"2024-09-18T11:51:26.939491+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:26.939491+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9334,"log_dups_size":0,"ondisk_log_size":9334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30284411,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9333,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1613}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9250","reported_seq":9346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.470787+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:27.470787+0000","last_peered":"2024-09-18T11:51:27.470787+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:27.470787+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9250,"log_dups_size":0,"ondisk_log_size":9250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30448251,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1671}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9377","reported_seq":9500,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.698318+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:26.698318+0000","last_peered":"2024-09-18T11:51:26.698318+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:26.698318+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30414070,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9214","reported_seq":13501,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:27.352278+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:27.352278+0000","last_peered":"2024-09-18T11:51:27.352278+0000","last_clean":"2024-09-18T11:51:27.352278+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:27.352278+0000","last_undegraded":"2024-09-18T11:51:27.352278+0000","last_fullsized":"2024-09-18T11:51:27.352278+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9214,"log_dups_size":0,"ondisk_log_size":9214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30217462,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8962","reported_seq":9050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.947390+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:22.947390+0000","last_peered":"2024-09-18T11:51:22.947390+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:22.947390+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30676214,"num_objects":3745,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8960,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1743}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":486892940,"num_objects":59441,"num_object_clones":0,"num_object_copies":237764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59441,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147646,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5880,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":973594624,"data_stored":973594624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147673,"ondisk_log_size":147673,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548392,"kb_used_data":238048,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93823448,"statfs":{"total":96636764160,"available":96075210752,"internally_reserved":0,"allocated":243761152,"data_stored":243585686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.922000000000001},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":10.991}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.956},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":5.3019999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.0659999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.2670000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542980,"kb_used_data":237300,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93828860,"statfs":{"total":96636764160,"available":96080752640,"internally_reserved":0,"allocated":242995200,"data_stored":242819734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":0.86899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.3979999999999997},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.9430000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.8490000000000002},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":6.5099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544340,"kb_used_data":238220,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93827500,"statfs":{"total":96636764160,"available":96079360000,"internally_reserved":0,"allocated":243937280,"data_stored":243761814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.29799999999999999},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.391},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.95299999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":8.407}]}]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539140,"kb_used_data":241332,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93832700,"statfs":{"total":96636764160,"available":96084684800,"internally_reserved":0,"allocated":247123968,"data_stored":246948502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":28.059000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":11.105}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.83999999999999997},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":11.068}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.4450000000000003},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":28.129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246079488,"data_stored":246079488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242720768,"data_stored":242720768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242089984,"data_stored":242089984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242704384,"data_stored":242704384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:29.031 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:29.629 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:29.653 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-18T11:51:28.008877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":486892940,"num_objects":59441,"num_object_clones":0,"num_object_copies":237764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59441,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147646,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5880,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147673,"ondisk_log_size":147673,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2174852,"kb_used_data":954900,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375312508,"statfs":{"total":386547056640,"available":384320008192,"internally_reserved":0,"allocated":977817600,"data_stored":977115736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,2,7,6,17,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24657920,"num_objects":-3010,"num_object_clones":0,"num_object_copies":-12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007633"},"pg_stats":[{"pgid":"1.f","version":"35'9477","reported_seq":9576,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.893879+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:26.893879+0000","last_peered":"2024-09-18T11:51:26.893879+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:26.893879+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9477,"log_dups_size":0,"ondisk_log_size":9477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30908416,"num_objects":3773,"num_object_clones":0,"num_object_copies":15092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9477,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1670}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9312","reported_seq":13602,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:27.223972+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:27.223972+0000","last_peered":"2024-09-18T11:51:27.223972+0000","last_clean":"2024-09-18T11:51:27.223972+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:27.223972+0000","last_undegraded":"2024-09-18T11:51:27.223972+0000","last_fullsized":"2024-09-18T11:51:27.223972+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30668022,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9132","reported_seq":9247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.546911+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:27.546911+0000","last_peered":"2024-09-18T11:51:27.546911+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:27.546911+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9132,"log_dups_size":0,"ondisk_log_size":9132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30112403,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9131,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1637}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9195","reported_seq":9287,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:23.055650+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:23.055650+0000","last_peered":"2024-09-18T11:51:23.055650+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:23.055650+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9195,"log_dups_size":0,"ondisk_log_size":9195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30561526,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9193,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9224","reported_seq":9343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.933925+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:26.933925+0000","last_peered":"2024-09-18T11:51:26.933925+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:26.933925+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9224,"log_dups_size":0,"ondisk_log_size":9224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30070006,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9197","reported_seq":9313,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.732672+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:26.732672+0000","last_peered":"2024-09-18T11:51:26.732672+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:26.732672+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29401088,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9197,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9283","reported_seq":9388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.938683+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:22.938683+0000","last_peered":"2024-09-18T11:51:22.938683+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:22.938683+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30577910,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9075","reported_seq":9182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.811269+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:22.811269+0000","last_peered":"2024-09-18T11:51:22.811269+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:22.811269+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9075,"log_dups_size":0,"ondisk_log_size":9075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31126774,"num_objects":3800,"num_object_clones":0,"num_object_copies":15200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9073,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9435","reported_seq":12256,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.922387+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:26.922387+0000","last_peered":"2024-09-18T11:51:26.922387+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:26.922387+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9435,"log_dups_size":0,"ondisk_log_size":9435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30640620,"num_objects":3741,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9432,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2982}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8947","reported_seq":9059,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.360314+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:27.360314+0000","last_peered":"2024-09-18T11:51:27.360314+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:27.360314+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8947,"log_dups_size":0,"ondisk_log_size":8947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873398,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8945,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1684}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9259","reported_seq":9378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.903449+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:26.903449+0000","last_peered":"2024-09-18T11:51:26.903449+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:26.903449+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30912369,"num_objects":3774,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9334","reported_seq":9425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.939491+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:26.939491+0000","last_peered":"2024-09-18T11:51:26.939491+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:26.939491+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9334,"log_dups_size":0,"ondisk_log_size":9334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30284411,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9333,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1613}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9250","reported_seq":9346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.470787+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:27.470787+0000","last_peered":"2024-09-18T11:51:27.470787+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:27.470787+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9250,"log_dups_size":0,"ondisk_log_size":9250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30448251,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1671}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9377","reported_seq":9500,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.698318+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:26.698318+0000","last_peered":"2024-09-18T11:51:26.698318+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:26.698318+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30414070,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9214","reported_seq":13501,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:27.352278+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:27.352278+0000","last_peered":"2024-09-18T11:51:27.352278+0000","last_clean":"2024-09-18T11:51:27.352278+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:27.352278+0000","last_undegraded":"2024-09-18T11:51:27.352278+0000","last_fullsized":"2024-09-18T11:51:27.352278+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9214,"log_dups_size":0,"ondisk_log_size":9214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30217462,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'8962","reported_seq":9050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:22.947390+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:22.947390+0000","last_peered":"2024-09-18T11:51:22.947390+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:22.947390+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30676214,"num_objects":3745,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8960,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1743}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":486892940,"num_objects":59441,"num_object_clones":0,"num_object_copies":237764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59441,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147646,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5880,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":973594624,"data_stored":973594624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147673,"ondisk_log_size":147673,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548392,"kb_used_data":238048,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93823448,"statfs":{"total":96636764160,"available":96075210752,"internally_reserved":0,"allocated":243761152,"data_stored":243585686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.922000000000001},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":10.991}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.956},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":5.3019999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.0659999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.2670000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542980,"kb_used_data":237300,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93828860,"statfs":{"total":96636764160,"available":96080752640,"internally_reserved":0,"allocated":242995200,"data_stored":242819734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":0.86899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.3979999999999997},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.9430000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.8490000000000002},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":6.5099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544340,"kb_used_data":238220,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93827500,"statfs":{"total":96636764160,"available":96079360000,"internally_reserved":0,"allocated":243937280,"data_stored":243761814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.29799999999999999},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.391},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.95299999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":8.407}]}]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539140,"kb_used_data":241332,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93832700,"statfs":{"total":96636764160,"available":96084684800,"internally_reserved":0,"allocated":247123968,"data_stored":246948502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":28.059000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":11.105}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.83999999999999997},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":11.068}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.4450000000000003},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":28.129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246079488,"data_stored":246079488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242720768,"data_stored":242720768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242089984,"data_stored":242089984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242704384,"data_stored":242704384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:29.654 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:29.654 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9477", 'reported_seq': 9576, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:26.893879+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:26.893879+0000', 'last_peered': '2024-09-18T11:51:26.893879+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:26.893879+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9477, 'log_dups_size': 0, 'ondisk_log_size': 9477, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30908416, 'num_objects': 3773, 'num_object_clones': 0, 'num_object_copies': 15092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9477, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1670}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:29.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:29.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9132", 'reported_seq': 9247, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:27.546911+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:27.546911+0000', 'last_peered': '2024-09-18T11:51:27.546911+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:27.546911+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9132, 'log_dups_size': 0, 'ondisk_log_size': 9132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30112403, 'num_objects': 3677, 'num_object_clones': 0, 'num_object_copies': 14708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3677, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9131, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1637}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:29.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:29.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9195", 'reported_seq': 9287, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:23.055650+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:23.055650+0000', 'last_peered': '2024-09-18T11:51:23.055650+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:23.055650+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9195, 'log_dups_size': 0, 'ondisk_log_size': 9195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30561526, 'num_objects': 3731, 'num_object_clones': 0, 'num_object_copies': 14924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9193, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1630}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:29.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:29.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9224", 'reported_seq': 9343, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:26.933925+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:26.933925+0000', 'last_peered': '2024-09-18T11:51:26.933925+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:26.933925+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9224, 'log_dups_size': 0, 'ondisk_log_size': 9224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30070006, 'num_objects': 3671, 'num_object_clones': 0, 'num_object_copies': 14684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9222, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1619}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:29.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:29.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9197", 'reported_seq': 9313, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:26.732672+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:26.732672+0000', 'last_peered': '2024-09-18T11:51:26.732672+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:26.732672+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9197, 'log_dups_size': 0, 'ondisk_log_size': 9197, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29401088, 'num_objects': 3589, 'num_object_clones': 0, 'num_object_copies': 14356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9197, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1470}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:29.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:29.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9283", 'reported_seq': 9388, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:22.938683+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:22.938683+0000', 'last_peered': '2024-09-18T11:51:22.938683+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:22.938683+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9283, 'log_dups_size': 0, 'ondisk_log_size': 9283, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30577910, 'num_objects': 3733, 'num_object_clones': 0, 'num_object_copies': 14932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9281, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1617}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:29.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:29.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9075", 'reported_seq': 9182, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:22.811269+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:22.811269+0000', 'last_peered': '2024-09-18T11:51:22.811269+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:22.811269+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9075, 'log_dups_size': 0, 'ondisk_log_size': 9075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31126774, 'num_objects': 3800, 'num_object_clones': 0, 'num_object_copies': 15200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9073, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1786}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:29.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:29.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9435", 'reported_seq': 12256, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:26.922387+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:26.922387+0000', 'last_peered': '2024-09-18T11:51:26.922387+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:26.922387+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9435, 'log_dups_size': 0, 'ondisk_log_size': 9435, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30640620, 'num_objects': 3741, 'num_object_clones': 0, 'num_object_copies': 14964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 759, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9432, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1382, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2982}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:29.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:29.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8947", 'reported_seq': 9059, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:27.360314+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:27.360314+0000', 'last_peered': '2024-09-18T11:51:27.360314+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:27.360314+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8947, 'log_dups_size': 0, 'ondisk_log_size': 8947, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29873398, 'num_objects': 3647, 'num_object_clones': 0, 'num_object_copies': 14588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8945, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1684}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:29.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:29.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9259", 'reported_seq': 9378, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:26.903449+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:26.903449+0000', 'last_peered': '2024-09-18T11:51:26.903449+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:26.903449+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9259, 'log_dups_size': 0, 'ondisk_log_size': 9259, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30912369, 'num_objects': 3774, 'num_object_clones': 0, 'num_object_copies': 15096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9256, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1740}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:29.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:29.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9334", 'reported_seq': 9425, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:26.939491+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:26.939491+0000', 'last_peered': '2024-09-18T11:51:26.939491+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:26.939491+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9334, 'log_dups_size': 0, 'ondisk_log_size': 9334, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30284411, 'num_objects': 3697, 'num_object_clones': 0, 'num_object_copies': 14788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9333, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1613}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:29.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:29.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9250", 'reported_seq': 9346, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:27.470787+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:27.470787+0000', 'last_peered': '2024-09-18T11:51:27.470787+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:27.470787+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9250, 'log_dups_size': 0, 'ondisk_log_size': 9250, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30448251, 'num_objects': 3717, 'num_object_clones': 0, 'num_object_copies': 14868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9249, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:29.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:29.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9377", 'reported_seq': 9500, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:26.698318+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:26.698318+0000', 'last_peered': '2024-09-18T11:51:26.698318+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:26.698318+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9377, 'log_dups_size': 0, 'ondisk_log_size': 9377, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30414070, 'num_objects': 3713, 'num_object_clones': 0, 'num_object_copies': 14852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9375, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:29.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:29.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8962", 'reported_seq': 9050, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:22.947390+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:22.947390+0000', 'last_peered': '2024-09-18T11:51:22.947390+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:22.947390+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 8962, 'log_dups_size': 0, 'ondisk_log_size': 8962, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30676214, 'num_objects': 3745, 'num_object_clones': 0, 'num_object_copies': 14980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3745, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8960, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1743}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:29.657 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:30.335 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:30.335 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27576/236660 objects degraded (11.652%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":488,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59165,"data_bytes":484631948,"bytes_used":2226876416,"bytes_avail":384320180224,"bytes_total":386547056640,"degraded_objects":27576,"degraded_total":236660,"degraded_ratio":0.11652159215752557,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":203},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:30.364 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:30.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:30.939 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:30.939 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:30.962 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-18T11:51:30.009412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484631948,"num_objects":59165,"num_object_clones":0,"num_object_copies":236660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147922,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5880,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147949,"ondisk_log_size":147949,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2174684,"kb_used_data":950568,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375312676,"statfs":{"total":386547056640,"available":384320180224,"internally_reserved":0,"allocated":973381632,"data_stored":972679768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,2,7,6,17,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19996672,"num_objects":-2441,"num_object_clones":0,"num_object_copies":-9764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2441,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007726"},"pg_stats":[{"pgid":"1.f","version":"35'9477","reported_seq":9576,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.893879+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:26.893879+0000","last_peered":"2024-09-18T11:51:26.893879+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:26.893879+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9477,"log_dups_size":0,"ondisk_log_size":9477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30908416,"num_objects":3773,"num_object_clones":0,"num_object_copies":15092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9477,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1670}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9312","reported_seq":13602,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:27.223972+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:27.223972+0000","last_peered":"2024-09-18T11:51:27.223972+0000","last_clean":"2024-09-18T11:51:27.223972+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:27.223972+0000","last_undegraded":"2024-09-18T11:51:27.223972+0000","last_fullsized":"2024-09-18T11:51:27.223972+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30668022,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9132","reported_seq":9247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.546911+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:27.546911+0000","last_peered":"2024-09-18T11:51:27.546911+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:27.546911+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9132,"log_dups_size":0,"ondisk_log_size":9132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30112403,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9131,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1637}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9261","reported_seq":9353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.854104+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:27.854104+0000","last_peered":"2024-09-18T11:51:27.854104+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:27.854104+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9261,"log_dups_size":0,"ondisk_log_size":9261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30020854,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9259,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1564}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9224","reported_seq":9343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.933925+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:26.933925+0000","last_peered":"2024-09-18T11:51:26.933925+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:26.933925+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9224,"log_dups_size":0,"ondisk_log_size":9224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30070006,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9197","reported_seq":9313,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.732672+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:26.732672+0000","last_peered":"2024-09-18T11:51:26.732672+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:26.732672+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29401088,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9197,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9359","reported_seq":9462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.996717+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:27.996717+0000","last_peered":"2024-09-18T11:51:27.996717+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:27.996717+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9359,"log_dups_size":0,"ondisk_log_size":9359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29955318,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9357,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1541}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9141","reported_seq":9248,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.955121+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:27.955121+0000","last_peered":"2024-09-18T11:51:27.955121+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:27.955121+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9141,"log_dups_size":0,"ondisk_log_size":9141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30586102,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9139,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1720}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9435","reported_seq":12256,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.922387+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:26.922387+0000","last_peered":"2024-09-18T11:51:26.922387+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:26.922387+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9435,"log_dups_size":0,"ondisk_log_size":9435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30640620,"num_objects":3741,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9432,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2982}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8947","reported_seq":9059,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.360314+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:27.360314+0000","last_peered":"2024-09-18T11:51:27.360314+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:27.360314+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":8947,"log_dups_size":0,"ondisk_log_size":8947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873398,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8945,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1684}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9259","reported_seq":9378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.903449+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:26.903449+0000","last_peered":"2024-09-18T11:51:26.903449+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:26.903449+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30912369,"num_objects":3774,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9334","reported_seq":9425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.939491+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:26.939491+0000","last_peered":"2024-09-18T11:51:26.939491+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:26.939491+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9334,"log_dups_size":0,"ondisk_log_size":9334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30284411,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9333,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1613}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9250","reported_seq":9346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.470787+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:27.470787+0000","last_peered":"2024-09-18T11:51:27.470787+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:27.470787+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9250,"log_dups_size":0,"ondisk_log_size":9250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30448251,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1671}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9377","reported_seq":9500,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:26.698318+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:26.698318+0000","last_peered":"2024-09-18T11:51:26.698318+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:26.698318+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30414070,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9214","reported_seq":13501,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:27.352278+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:27.352278+0000","last_peered":"2024-09-18T11:51:27.352278+0000","last_clean":"2024-09-18T11:51:27.352278+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:27.352278+0000","last_undegraded":"2024-09-18T11:51:27.352278+0000","last_fullsized":"2024-09-18T11:51:27.352278+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9214,"log_dups_size":0,"ondisk_log_size":9214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30217462,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9030","reported_seq":9118,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:27.952068+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:27.952068+0000","last_peered":"2024-09-18T11:51:27.952068+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:27.952068+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9030,"log_dups_size":0,"ondisk_log_size":9030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30119158,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9028,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":484631948,"num_objects":59165,"num_object_clones":0,"num_object_copies":236660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147922,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5880,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":969134080,"data_stored":969134080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147949,"ondisk_log_size":147949,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548392,"kb_used_data":238048,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93823448,"statfs":{"total":96636764160,"available":96075210752,"internally_reserved":0,"allocated":243761152,"data_stored":243585686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.922000000000001},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":10.991}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":10.956},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":5.3019999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.0659999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.2670000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542980,"kb_used_data":237300,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93828860,"statfs":{"total":96636764160,"available":96080752640,"internally_reserved":0,"allocated":242995200,"data_stored":242819734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":0.86899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.3979999999999997},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.9430000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.8490000000000002},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":6.5099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544340,"kb_used_data":238220,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93827500,"statfs":{"total":96636764160,"available":96079360000,"internally_reserved":0,"allocated":243937280,"data_stored":243761814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.29799999999999999},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.391},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.95299999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":8.407}]}]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538972,"kb_used_data":237000,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93832868,"statfs":{"total":96636764160,"available":96084856832,"internally_reserved":0,"allocated":242688000,"data_stored":242512534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":28.059000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":11.105}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.83999999999999997},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":11.068}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.4450000000000003},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":28.129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241618944,"data_stored":241618944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242720768,"data_stored":242720768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242089984,"data_stored":242089984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242704384,"data_stored":242704384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:33.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:34.567 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-18T11:51:34.010394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":475620748,"num_objects":58065,"num_object_clones":0,"num_object_copies":232260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149022,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149049,"ondisk_log_size":149049,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2156208,"kb_used_data":932096,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375331152,"statfs":{"total":386547056640,"available":384339099648,"internally_reserved":0,"allocated":954466304,"data_stored":953764440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,7,1,7,10,16,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22790144,"num_objects":-2782,"num_object_clones":0,"num_object_copies":-11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2782,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004111"},"pg_stats":[{"pgid":"1.f","version":"35'9565","reported_seq":9664,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.773478+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:31.773478+0000","last_peered":"2024-09-18T11:51:31.773478+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:31.773478+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9565,"log_dups_size":0,"ondisk_log_size":9565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30187520,"num_objects":3685,"num_object_clones":0,"num_object_copies":14740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1582}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9377","reported_seq":13665,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:32.359642+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:32.359642+0000","last_peered":"2024-09-18T11:51:32.359642+0000","last_clean":"2024-09-18T11:51:32.359642+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:32.359642+0000","last_undegraded":"2024-09-18T11:51:32.359642+0000","last_fullsized":"2024-09-18T11:51:32.359642+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30135542,"num_objects":3679,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9200","reported_seq":9315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:32.386178+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:32.386178+0000","last_peered":"2024-09-18T11:51:32.386178+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:32.386178+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9200,"log_dups_size":0,"ondisk_log_size":9200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29555347,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9199,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9326","reported_seq":9418,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.160047+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:33.160047+0000","last_peered":"2024-09-18T11:51:33.160047+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:33.160047+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9326,"log_dups_size":0,"ondisk_log_size":9326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29488374,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9324,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1499}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9282","reported_seq":9401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.509848+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:31.509848+0000","last_peered":"2024-09-18T11:51:31.509848+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:31.509848+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29594870,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9263","reported_seq":9380,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.864282+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:31.864282+0000","last_peered":"2024-09-18T11:51:31.864282+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:31.864282+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9263,"log_dups_size":0,"ondisk_log_size":9263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28860416,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9263,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9422","reported_seq":9527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.056179+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:33.056179+0000","last_peered":"2024-09-18T11:51:33.056179+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:33.056179+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9422,"log_dups_size":0,"ondisk_log_size":9422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29439222,"num_objects":3594,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9420,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1478}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9219","reported_seq":9326,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.238928+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:33.238928+0000","last_peered":"2024-09-18T11:51:33.238928+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:33.238928+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9219,"log_dups_size":0,"ondisk_log_size":9219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29947126,"num_objects":3656,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9497","reported_seq":12404,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.857508+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:31.857508+0000","last_peered":"2024-09-18T11:51:31.857508+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:31.857508+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9497,"log_dups_size":0,"ondisk_log_size":9497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30132716,"num_objects":3679,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9016","reported_seq":9128,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:32.599340+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:32.599340+0000","last_peered":"2024-09-18T11:51:32.599340+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:32.599340+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9016,"log_dups_size":0,"ondisk_log_size":9016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29308150,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9014,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9328","reported_seq":9446,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.778494+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:31.778494+0000","last_peered":"2024-09-18T11:51:31.778494+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:31.778494+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30347121,"num_objects":3705,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1671}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9388","reported_seq":9479,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.763728+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:31.763728+0000","last_peered":"2024-09-18T11:51:31.763728+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:31.763728+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29842043,"num_objects":3643,"num_object_clones":0,"num_object_copies":14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9320","reported_seq":9416,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:32.569279+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:32.569279+0000","last_peered":"2024-09-18T11:51:32.569279+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:32.569279+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29874811,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9450","reported_seq":9573,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.873597+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:31.873597+0000","last_peered":"2024-09-18T11:51:31.873597+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:31.873597+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9450,"log_dups_size":0,"ondisk_log_size":9450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29816054,"num_objects":3640,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9448,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1517}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9292","reported_seq":13579,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:32.325249+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:32.325249+0000","last_peered":"2024-09-18T11:51:32.325249+0000","last_clean":"2024-09-18T11:51:32.325249+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:32.325249+0000","last_undegraded":"2024-09-18T11:51:32.325249+0000","last_fullsized":"2024-09-18T11:51:32.325249+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9292,"log_dups_size":0,"ondisk_log_size":9292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29578486,"num_objects":3611,"num_object_clones":0,"num_object_copies":14444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9290,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9104","reported_seq":9192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.109699+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:33.109699+0000","last_peered":"2024-09-18T11:51:33.109699+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:33.109699+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9104,"log_dups_size":0,"ondisk_log_size":9104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29512950,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":475620748,"num_objects":58065,"num_object_clones":0,"num_object_copies":232260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149022,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":951144448,"data_stored":951144448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149049,"ondisk_log_size":149049,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531508,"kb_used_data":233644,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93840332,"statfs":{"total":96636764160,"available":96092499968,"internally_reserved":0,"allocated":239251456,"data_stored":239075990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":8.2119999999999997},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":8.2430000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":8.2720000000000002},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.5199999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":3.0369999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.54100000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542792,"kb_used_data":232948,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93829048,"statfs":{"total":96636764160,"available":96080945152,"internally_reserved":0,"allocated":238538752,"data_stored":238363286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":2.4830000000000001},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":2.5139999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.0870000000000002},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.577}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":1.6020000000000001},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":8.0190000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543184,"kb_used_data":232900,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93828656,"statfs":{"total":96636764160,"available":96080543744,"internally_reserved":0,"allocated":238489600,"data_stored":238314134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,2,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.72499999999999998},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.4480000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":9.4440000000000008},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.69899999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":7.4880000000000004},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":9.4049999999999994}]}]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538724,"kb_used_data":232604,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93833116,"statfs":{"total":96636764160,"available":96085110784,"internally_reserved":0,"allocated":238186496,"data_stored":238011030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,1,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":10.451000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":34.786999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":4.1130000000000004},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.782}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":4.5190000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":4.46}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237051904,"data_stored":237051904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238174208,"data_stored":238174208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237658112,"data_stored":237658112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":238260224,"data_stored":238260224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:34.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:35.137 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:35.161 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-18T11:51:34.010394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":475620748,"num_objects":58065,"num_object_clones":0,"num_object_copies":232260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149022,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149049,"ondisk_log_size":149049,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2156208,"kb_used_data":932096,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375331152,"statfs":{"total":386547056640,"available":384339099648,"internally_reserved":0,"allocated":954466304,"data_stored":953764440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,7,1,7,10,16,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22790144,"num_objects":-2782,"num_object_clones":0,"num_object_copies":-11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2782,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004111"},"pg_stats":[{"pgid":"1.f","version":"35'9565","reported_seq":9664,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.773478+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:31.773478+0000","last_peered":"2024-09-18T11:51:31.773478+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:31.773478+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9565,"log_dups_size":0,"ondisk_log_size":9565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30187520,"num_objects":3685,"num_object_clones":0,"num_object_copies":14740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1582}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9377","reported_seq":13665,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:32.359642+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:32.359642+0000","last_peered":"2024-09-18T11:51:32.359642+0000","last_clean":"2024-09-18T11:51:32.359642+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:32.359642+0000","last_undegraded":"2024-09-18T11:51:32.359642+0000","last_fullsized":"2024-09-18T11:51:32.359642+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30135542,"num_objects":3679,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9200","reported_seq":9315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:32.386178+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:32.386178+0000","last_peered":"2024-09-18T11:51:32.386178+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:32.386178+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9200,"log_dups_size":0,"ondisk_log_size":9200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29555347,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9199,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9326","reported_seq":9418,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.160047+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:33.160047+0000","last_peered":"2024-09-18T11:51:33.160047+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:33.160047+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9326,"log_dups_size":0,"ondisk_log_size":9326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29488374,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9324,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1499}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9282","reported_seq":9401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.509848+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:31.509848+0000","last_peered":"2024-09-18T11:51:31.509848+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:31.509848+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29594870,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9263","reported_seq":9380,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.864282+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:31.864282+0000","last_peered":"2024-09-18T11:51:31.864282+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:31.864282+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9263,"log_dups_size":0,"ondisk_log_size":9263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28860416,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9263,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9422","reported_seq":9527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.056179+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:33.056179+0000","last_peered":"2024-09-18T11:51:33.056179+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:33.056179+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9422,"log_dups_size":0,"ondisk_log_size":9422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29439222,"num_objects":3594,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9420,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1478}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9219","reported_seq":9326,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.238928+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:33.238928+0000","last_peered":"2024-09-18T11:51:33.238928+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:33.238928+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9219,"log_dups_size":0,"ondisk_log_size":9219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29947126,"num_objects":3656,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9497","reported_seq":12404,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.857508+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:31.857508+0000","last_peered":"2024-09-18T11:51:31.857508+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:31.857508+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9497,"log_dups_size":0,"ondisk_log_size":9497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30132716,"num_objects":3679,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9016","reported_seq":9128,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:32.599340+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:32.599340+0000","last_peered":"2024-09-18T11:51:32.599340+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:32.599340+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9016,"log_dups_size":0,"ondisk_log_size":9016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29308150,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9014,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9328","reported_seq":9446,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.778494+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:31.778494+0000","last_peered":"2024-09-18T11:51:31.778494+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:31.778494+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30347121,"num_objects":3705,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1671}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9388","reported_seq":9479,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.763728+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:31.763728+0000","last_peered":"2024-09-18T11:51:31.763728+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:31.763728+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29842043,"num_objects":3643,"num_object_clones":0,"num_object_copies":14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9320","reported_seq":9416,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:32.569279+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:32.569279+0000","last_peered":"2024-09-18T11:51:32.569279+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:32.569279+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29874811,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9450","reported_seq":9573,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.873597+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:31.873597+0000","last_peered":"2024-09-18T11:51:31.873597+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:31.873597+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9450,"log_dups_size":0,"ondisk_log_size":9450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29816054,"num_objects":3640,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9448,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1517}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9292","reported_seq":13579,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:32.325249+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:32.325249+0000","last_peered":"2024-09-18T11:51:32.325249+0000","last_clean":"2024-09-18T11:51:32.325249+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:32.325249+0000","last_undegraded":"2024-09-18T11:51:32.325249+0000","last_fullsized":"2024-09-18T11:51:32.325249+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9292,"log_dups_size":0,"ondisk_log_size":9292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29578486,"num_objects":3611,"num_object_clones":0,"num_object_copies":14444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9290,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9104","reported_seq":9192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.109699+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:33.109699+0000","last_peered":"2024-09-18T11:51:33.109699+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:33.109699+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9104,"log_dups_size":0,"ondisk_log_size":9104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29512950,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":475620748,"num_objects":58065,"num_object_clones":0,"num_object_copies":232260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149022,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":951144448,"data_stored":951144448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149049,"ondisk_log_size":149049,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531508,"kb_used_data":233644,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93840332,"statfs":{"total":96636764160,"available":96092499968,"internally_reserved":0,"allocated":239251456,"data_stored":239075990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":8.2119999999999997},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":8.2430000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":8.2720000000000002},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.5199999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":3.0369999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.54100000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542792,"kb_used_data":232948,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93829048,"statfs":{"total":96636764160,"available":96080945152,"internally_reserved":0,"allocated":238538752,"data_stored":238363286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":2.4830000000000001},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":2.5139999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.0870000000000002},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.577}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":1.6020000000000001},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":8.0190000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543184,"kb_used_data":232900,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93828656,"statfs":{"total":96636764160,"available":96080543744,"internally_reserved":0,"allocated":238489600,"data_stored":238314134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,2,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.72499999999999998},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.4480000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":9.4440000000000008},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.69899999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":7.4880000000000004},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":9.4049999999999994}]}]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538724,"kb_used_data":232604,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93833116,"statfs":{"total":96636764160,"available":96085110784,"internally_reserved":0,"allocated":238186496,"data_stored":238011030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,1,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":10.451000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":34.786999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":4.1130000000000004},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.782}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":4.5190000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":4.46}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237051904,"data_stored":237051904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238174208,"data_stored":238174208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237658112,"data_stored":237658112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":238260224,"data_stored":238260224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:35.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:35.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9565", 'reported_seq': 9664, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:31.773478+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:31.773478+0000', 'last_peered': '2024-09-18T11:51:31.773478+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:31.773478+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9565, 'log_dups_size': 0, 'ondisk_log_size': 9565, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30187520, 'num_objects': 3685, 'num_object_clones': 0, 'num_object_copies': 14740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9565, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1582}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:35.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:35.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9200", 'reported_seq': 9315, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:32.386178+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:32.386178+0000', 'last_peered': '2024-09-18T11:51:32.386178+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:32.386178+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9200, 'log_dups_size': 0, 'ondisk_log_size': 9200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29555347, 'num_objects': 3609, 'num_object_clones': 0, 'num_object_copies': 14436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3609, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9199, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1569}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:35.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:35.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9326", 'reported_seq': 9418, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:33.160047+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:33.160047+0000', 'last_peered': '2024-09-18T11:51:33.160047+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:33.160047+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9326, 'log_dups_size': 0, 'ondisk_log_size': 9326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29488374, 'num_objects': 3600, 'num_object_clones': 0, 'num_object_copies': 14400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9324, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1499}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:35.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:35.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9282", 'reported_seq': 9401, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:31.509848+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:31.509848+0000', 'last_peered': '2024-09-18T11:51:31.509848+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:31.509848+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9282, 'log_dups_size': 0, 'ondisk_log_size': 9282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29594870, 'num_objects': 3613, 'num_object_clones': 0, 'num_object_copies': 14452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9280, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1561}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:35.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9263", 'reported_seq': 9380, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:31.864282+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:31.864282+0000', 'last_peered': '2024-09-18T11:51:31.864282+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:31.864282+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9263, 'log_dups_size': 0, 'ondisk_log_size': 9263, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28860416, 'num_objects': 3523, 'num_object_clones': 0, 'num_object_copies': 14092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9263, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1404}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9422", 'reported_seq': 9527, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:33.056179+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:33.056179+0000', 'last_peered': '2024-09-18T11:51:33.056179+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:33.056179+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9422, 'log_dups_size': 0, 'ondisk_log_size': 9422, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29439222, 'num_objects': 3594, 'num_object_clones': 0, 'num_object_copies': 14376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3594, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9420, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1478}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9219", 'reported_seq': 9326, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:33.238928+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:33.238928+0000', 'last_peered': '2024-09-18T11:51:33.238928+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:33.238928+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9219, 'log_dups_size': 0, 'ondisk_log_size': 9219, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29947126, 'num_objects': 3656, 'num_object_clones': 0, 'num_object_copies': 14624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9217, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1642}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9497", 'reported_seq': 12404, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:31.857508+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:31.857508+0000', 'last_peered': '2024-09-18T11:51:31.857508+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:31.857508+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9497, 'log_dups_size': 0, 'ondisk_log_size': 9497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30132716, 'num_objects': 3679, 'num_object_clones': 0, 'num_object_copies': 14716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 716, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9494, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1425, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2963}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9016", 'reported_seq': 9128, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:32.599340+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:32.599340+0000', 'last_peered': '2024-09-18T11:51:32.599340+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:32.599340+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9016, 'log_dups_size': 0, 'ondisk_log_size': 9016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29308150, 'num_objects': 3578, 'num_object_clones': 0, 'num_object_copies': 14312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9014, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1615}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:35.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9328", 'reported_seq': 9446, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:31.778494+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:31.778494+0000', 'last_peered': '2024-09-18T11:51:31.778494+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:31.778494+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9328, 'log_dups_size': 0, 'ondisk_log_size': 9328, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30347121, 'num_objects': 3705, 'num_object_clones': 0, 'num_object_copies': 14820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9325, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1671}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:35.165 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:35.165 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9388", 'reported_seq': 9479, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:31.763728+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:31.763728+0000', 'last_peered': '2024-09-18T11:51:31.763728+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:31.763728+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9388, 'log_dups_size': 0, 'ondisk_log_size': 9388, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29842043, 'num_objects': 3643, 'num_object_clones': 0, 'num_object_copies': 14572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9387, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:35.165 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:35.165 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9320", 'reported_seq': 9416, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:32.569279+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:32.569279+0000', 'last_peered': '2024-09-18T11:51:32.569279+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:32.569279+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9320, 'log_dups_size': 0, 'ondisk_log_size': 9320, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29874811, 'num_objects': 3647, 'num_object_clones': 0, 'num_object_copies': 14588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9319, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:35.165 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:35.165 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9450", 'reported_seq': 9573, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:31.873597+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:31.873597+0000', 'last_peered': '2024-09-18T11:51:31.873597+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:31.873597+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9450, 'log_dups_size': 0, 'ondisk_log_size': 9450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29816054, 'num_objects': 3640, 'num_object_clones': 0, 'num_object_copies': 14560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9448, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1517}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:35.165 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:35.165 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9104", 'reported_seq': 9192, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:33.109699+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:33.109699+0000', 'last_peered': '2024-09-18T11:51:33.109699+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:33.109699+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9104, 'log_dups_size': 0, 'ondisk_log_size': 9104, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29512950, 'num_objects': 3603, 'num_object_clones': 0, 'num_object_copies': 14412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9102, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:35.165 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:35.835 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:35.835 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27533/232260 objects degraded (11.854%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":493,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58065,"data_bytes":475620748,"bytes_used":2207956992,"bytes_avail":384339099648,"bytes_total":386547056640,"degraded_objects":27533,"degraded_total":232260,"degraded_ratio":0.11854387324550072,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":231},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:35.863 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:35.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:36.394 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:36.425 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-18T11:51:36.010877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":475620748,"num_objects":58065,"num_object_clones":0,"num_object_copies":232260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149022,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149049,"ondisk_log_size":149049,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2156208,"kb_used_data":932096,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375331152,"statfs":{"total":386547056640,"available":384339099648,"internally_reserved":0,"allocated":954466304,"data_stored":953764440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,7,1,7,10,16,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17973248,"num_objects":-2194,"num_object_clones":0,"num_object_copies":-8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2194,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003927"},"pg_stats":[{"pgid":"1.f","version":"35'9565","reported_seq":9664,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.773478+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:31.773478+0000","last_peered":"2024-09-18T11:51:31.773478+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:31.773478+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9565,"log_dups_size":0,"ondisk_log_size":9565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30187520,"num_objects":3685,"num_object_clones":0,"num_object_copies":14740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1582}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9377","reported_seq":13665,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:32.359642+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:32.359642+0000","last_peered":"2024-09-18T11:51:32.359642+0000","last_clean":"2024-09-18T11:51:32.359642+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:32.359642+0000","last_undegraded":"2024-09-18T11:51:32.359642+0000","last_fullsized":"2024-09-18T11:51:32.359642+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30135542,"num_objects":3679,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9200","reported_seq":9315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:32.386178+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:32.386178+0000","last_peered":"2024-09-18T11:51:32.386178+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:32.386178+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9200,"log_dups_size":0,"ondisk_log_size":9200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29555347,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9199,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9326","reported_seq":9418,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.160047+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:33.160047+0000","last_peered":"2024-09-18T11:51:33.160047+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:33.160047+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9326,"log_dups_size":0,"ondisk_log_size":9326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29488374,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9324,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1499}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9282","reported_seq":9401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.509848+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:31.509848+0000","last_peered":"2024-09-18T11:51:31.509848+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:31.509848+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29594870,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9263","reported_seq":9380,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.864282+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:31.864282+0000","last_peered":"2024-09-18T11:51:31.864282+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:31.864282+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9263,"log_dups_size":0,"ondisk_log_size":9263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28860416,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9263,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9422","reported_seq":9527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.056179+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:33.056179+0000","last_peered":"2024-09-18T11:51:33.056179+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:33.056179+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9422,"log_dups_size":0,"ondisk_log_size":9422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29439222,"num_objects":3594,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9420,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1478}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9219","reported_seq":9326,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.238928+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:33.238928+0000","last_peered":"2024-09-18T11:51:33.238928+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:33.238928+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9219,"log_dups_size":0,"ondisk_log_size":9219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29947126,"num_objects":3656,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9497","reported_seq":12404,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.857508+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:31.857508+0000","last_peered":"2024-09-18T11:51:31.857508+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:31.857508+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9497,"log_dups_size":0,"ondisk_log_size":9497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30132716,"num_objects":3679,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9016","reported_seq":9128,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:32.599340+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:32.599340+0000","last_peered":"2024-09-18T11:51:32.599340+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:32.599340+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9016,"log_dups_size":0,"ondisk_log_size":9016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29308150,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9014,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9328","reported_seq":9446,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.778494+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:31.778494+0000","last_peered":"2024-09-18T11:51:31.778494+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:31.778494+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30347121,"num_objects":3705,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1671}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9388","reported_seq":9479,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.763728+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:31.763728+0000","last_peered":"2024-09-18T11:51:31.763728+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:31.763728+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29842043,"num_objects":3643,"num_object_clones":0,"num_object_copies":14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9320","reported_seq":9416,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:32.569279+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:32.569279+0000","last_peered":"2024-09-18T11:51:32.569279+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:32.569279+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29874811,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9450","reported_seq":9573,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:31.873597+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:31.873597+0000","last_peered":"2024-09-18T11:51:31.873597+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:31.873597+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9450,"log_dups_size":0,"ondisk_log_size":9450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29816054,"num_objects":3640,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9448,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1517}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9292","reported_seq":13579,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:32.325249+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:32.325249+0000","last_peered":"2024-09-18T11:51:32.325249+0000","last_clean":"2024-09-18T11:51:32.325249+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:32.325249+0000","last_undegraded":"2024-09-18T11:51:32.325249+0000","last_fullsized":"2024-09-18T11:51:32.325249+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9292,"log_dups_size":0,"ondisk_log_size":9292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29578486,"num_objects":3611,"num_object_clones":0,"num_object_copies":14444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9290,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9104","reported_seq":9192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:33.109699+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:33.109699+0000","last_peered":"2024-09-18T11:51:33.109699+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:33.109699+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9104,"log_dups_size":0,"ondisk_log_size":9104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29512950,"num_objects":3603,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":475620748,"num_objects":58065,"num_object_clones":0,"num_object_copies":232260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149022,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":951144448,"data_stored":951144448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149049,"ondisk_log_size":149049,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531508,"kb_used_data":233644,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93840332,"statfs":{"total":96636764160,"available":96092499968,"internally_reserved":0,"allocated":239251456,"data_stored":239075990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":8.2119999999999997},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":8.2430000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":8.2720000000000002},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.5199999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":3.0369999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.54100000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542792,"kb_used_data":232948,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93829048,"statfs":{"total":96636764160,"available":96080945152,"internally_reserved":0,"allocated":238538752,"data_stored":238363286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":2.4830000000000001},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":2.5139999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.0870000000000002},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.577}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":1.6020000000000001},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":8.0190000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543184,"kb_used_data":232900,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93828656,"statfs":{"total":96636764160,"available":96080543744,"internally_reserved":0,"allocated":238489600,"data_stored":238314134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,2,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.72499999999999998},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.4480000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":9.4440000000000008},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.69899999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":7.4880000000000004},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":9.4049999999999994}]}]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538724,"kb_used_data":232604,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93833116,"statfs":{"total":96636764160,"available":96085110784,"internally_reserved":0,"allocated":238186496,"data_stored":238011030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,1,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":10.451000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":34.786999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":4.1130000000000004},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":7.782}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":4.5190000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":4.46}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237051904,"data_stored":237051904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238174208,"data_stored":238174208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237658112,"data_stored":237658112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":238260224,"data_stored":238260224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:39.428 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:40.031 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:40.032 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:40.051 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-18T11:51:40.011808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":466830732,"num_objects":56992,"num_object_clones":0,"num_object_copies":227968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150095,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5980,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150122,"ondisk_log_size":150122,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2122268,"kb_used_data":914796,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375365092,"statfs":{"total":386547056640,"available":384373854208,"internally_reserved":0,"allocated":936751104,"data_stored":936049240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,2,6,14,17,17,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20062208,"num_objects":-2449,"num_object_clones":0,"num_object_copies":-9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002930"},"pg_stats":[{"pgid":"1.f","version":"35'9622","reported_seq":9721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.972775+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:36.972775+0000","last_peered":"2024-09-18T11:51:36.972775+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:36.972775+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29720576,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9622,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9442","reported_seq":13732,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:37.286460+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:37.286460+0000","last_peered":"2024-09-18T11:51:37.286460+0000","last_clean":"2024-09-18T11:51:37.286460+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:37.286460+0000","last_undegraded":"2024-09-18T11:51:37.286460+0000","last_fullsized":"2024-09-18T11:51:37.286460+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9442,"log_dups_size":0,"ondisk_log_size":9442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29603062,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9440,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9283","reported_seq":9398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.342945+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:37.342945+0000","last_peered":"2024-09-18T11:51:37.342945+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:37.342945+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28875411,"num_objects":3526,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9282,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1486}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9389","reported_seq":9481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.186591+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:38.186591+0000","last_peered":"2024-09-18T11:51:38.186591+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:38.186591+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9389,"log_dups_size":0,"ondisk_log_size":9389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28972278,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9352","reported_seq":9471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.041615+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:37.041615+0000","last_peered":"2024-09-18T11:51:37.041615+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:37.041615+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29021430,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9328","reported_seq":9445,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.859337+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:36.859337+0000","last_peered":"2024-09-18T11:51:36.859337+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:36.859337+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28327936,"num_objects":3458,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9499","reported_seq":9604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.178096+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:38.178096+0000","last_peered":"2024-09-18T11:51:38.178096+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:38.178096+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9499,"log_dups_size":0,"ondisk_log_size":9499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28808438,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9497,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1401}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9296","reported_seq":9403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.395278+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:38.395278+0000","last_peered":"2024-09-18T11:51:38.395278+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:38.395278+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9296,"log_dups_size":0,"ondisk_log_size":9296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29316342,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9294,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1565}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9549","reported_seq":12570,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.970622+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:36.970622+0000","last_peered":"2024-09-18T11:51:36.970622+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:36.970622+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9549,"log_dups_size":0,"ondisk_log_size":9549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29706732,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9546,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1482,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9084","reported_seq":9195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.617189+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:37.617189+0000","last_peered":"2024-09-18T11:51:37.617189+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:37.617189+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9084,"log_dups_size":0,"ondisk_log_size":9084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28751094,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9082,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1547}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9396","reported_seq":9514,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.085615+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:37.085615+0000","last_peered":"2024-09-18T11:51:37.085615+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:37.085615+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29790065,"num_objects":3637,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9393,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9452","reported_seq":9543,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.011277+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:37.011277+0000","last_peered":"2024-09-18T11:51:37.011277+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:37.011277+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9452,"log_dups_size":0,"ondisk_log_size":9452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29317755,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9451,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1495}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9389","reported_seq":9485,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.448215+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:37.448215+0000","last_peered":"2024-09-18T11:51:37.448215+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:37.448215+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9389,"log_dups_size":0,"ondisk_log_size":9389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29309563,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9388,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1532}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9517","reported_seq":9640,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.932261+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:36.932261+0000","last_peered":"2024-09-18T11:51:36.932261+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:36.932261+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29267190,"num_objects":3573,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9348","reported_seq":13635,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:37.557305+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:37.557305+0000","last_peered":"2024-09-18T11:51:37.557305+0000","last_clean":"2024-09-18T11:51:37.557305+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:37.557305+0000","last_undegraded":"2024-09-18T11:51:37.557305+0000","last_fullsized":"2024-09-18T11:51:37.557305+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119734,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9346,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9176","reported_seq":9264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.247171+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:38.247171+0000","last_peered":"2024-09-18T11:51:38.247171+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:38.247171+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28923126,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":466830732,"num_objects":56992,"num_object_clones":0,"num_object_copies":227968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150095,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5980,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":933310464,"data_stored":933310464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150122,"ondisk_log_size":150122,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531372,"kb_used_data":229332,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93840468,"statfs":{"total":96636764160,"available":96092639232,"internally_reserved":0,"allocated":234835968,"data_stored":234660502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,8,7,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":17.158000000000001},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":16.193000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":6.6269999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.9260000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525952,"kb_used_data":228600,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93845888,"statfs":{"total":96636764160,"available":96098189312,"internally_reserved":0,"allocated":234086400,"data_stored":233910934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1,3,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":4.9850000000000003},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":5.0179999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.5540000000000003},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.492}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":3.5219999999999998},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":10.659000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526384,"kb_used_data":228584,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93845456,"statfs":{"total":96636764160,"available":96097746944,"internally_reserved":0,"allocated":234070016,"data_stored":233894550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.2870000000000008},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":12.313000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":23.850000000000001},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.1269999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":12.345000000000001},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":12.117000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538560,"kb_used_data":228280,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93833280,"statfs":{"total":96636764160,"available":96085278720,"internally_reserved":0,"allocated":233758720,"data_stored":233583254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,1,0,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":2.0659999999999998},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.22700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":2.0209999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":9.4700000000000006},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":17.488}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":232566784,"data_stored":232566784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233717760,"data_stored":233717760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233275392,"data_stored":233275392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233750528,"data_stored":233750528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:40.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:40.645 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:40.646 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:40.681 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-18T11:51:40.011808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":466830732,"num_objects":56992,"num_object_clones":0,"num_object_copies":227968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150095,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5980,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150122,"ondisk_log_size":150122,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2122268,"kb_used_data":914796,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375365092,"statfs":{"total":386547056640,"available":384373854208,"internally_reserved":0,"allocated":936751104,"data_stored":936049240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,2,6,14,17,17,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20062208,"num_objects":-2449,"num_object_clones":0,"num_object_copies":-9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002930"},"pg_stats":[{"pgid":"1.f","version":"35'9622","reported_seq":9721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.972775+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:36.972775+0000","last_peered":"2024-09-18T11:51:36.972775+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:36.972775+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29720576,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9622,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9442","reported_seq":13732,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:37.286460+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:37.286460+0000","last_peered":"2024-09-18T11:51:37.286460+0000","last_clean":"2024-09-18T11:51:37.286460+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:37.286460+0000","last_undegraded":"2024-09-18T11:51:37.286460+0000","last_fullsized":"2024-09-18T11:51:37.286460+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9442,"log_dups_size":0,"ondisk_log_size":9442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29603062,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9440,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9283","reported_seq":9398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.342945+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:37.342945+0000","last_peered":"2024-09-18T11:51:37.342945+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:37.342945+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28875411,"num_objects":3526,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9282,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1486}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9389","reported_seq":9481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.186591+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:38.186591+0000","last_peered":"2024-09-18T11:51:38.186591+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:38.186591+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9389,"log_dups_size":0,"ondisk_log_size":9389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28972278,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9352","reported_seq":9471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.041615+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:37.041615+0000","last_peered":"2024-09-18T11:51:37.041615+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:37.041615+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29021430,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9328","reported_seq":9445,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.859337+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:36.859337+0000","last_peered":"2024-09-18T11:51:36.859337+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:36.859337+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28327936,"num_objects":3458,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9499","reported_seq":9604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.178096+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:38.178096+0000","last_peered":"2024-09-18T11:51:38.178096+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:38.178096+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9499,"log_dups_size":0,"ondisk_log_size":9499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28808438,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9497,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1401}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9296","reported_seq":9403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.395278+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:38.395278+0000","last_peered":"2024-09-18T11:51:38.395278+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:38.395278+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9296,"log_dups_size":0,"ondisk_log_size":9296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29316342,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9294,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1565}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9549","reported_seq":12570,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.970622+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:36.970622+0000","last_peered":"2024-09-18T11:51:36.970622+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:36.970622+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9549,"log_dups_size":0,"ondisk_log_size":9549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29706732,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9546,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1482,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9084","reported_seq":9195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.617189+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:37.617189+0000","last_peered":"2024-09-18T11:51:37.617189+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:37.617189+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9084,"log_dups_size":0,"ondisk_log_size":9084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28751094,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9082,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1547}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9396","reported_seq":9514,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.085615+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:37.085615+0000","last_peered":"2024-09-18T11:51:37.085615+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:37.085615+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29790065,"num_objects":3637,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9393,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9452","reported_seq":9543,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.011277+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:37.011277+0000","last_peered":"2024-09-18T11:51:37.011277+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:37.011277+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9452,"log_dups_size":0,"ondisk_log_size":9452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29317755,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9451,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1495}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9389","reported_seq":9485,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.448215+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:37.448215+0000","last_peered":"2024-09-18T11:51:37.448215+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:37.448215+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9389,"log_dups_size":0,"ondisk_log_size":9389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29309563,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9388,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1532}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9517","reported_seq":9640,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.932261+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:36.932261+0000","last_peered":"2024-09-18T11:51:36.932261+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:36.932261+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29267190,"num_objects":3573,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9348","reported_seq":13635,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:37.557305+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:37.557305+0000","last_peered":"2024-09-18T11:51:37.557305+0000","last_clean":"2024-09-18T11:51:37.557305+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:37.557305+0000","last_undegraded":"2024-09-18T11:51:37.557305+0000","last_fullsized":"2024-09-18T11:51:37.557305+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119734,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9346,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9176","reported_seq":9264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.247171+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:38.247171+0000","last_peered":"2024-09-18T11:51:38.247171+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:38.247171+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28923126,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":466830732,"num_objects":56992,"num_object_clones":0,"num_object_copies":227968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150095,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5980,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":933310464,"data_stored":933310464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150122,"ondisk_log_size":150122,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531372,"kb_used_data":229332,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93840468,"statfs":{"total":96636764160,"available":96092639232,"internally_reserved":0,"allocated":234835968,"data_stored":234660502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,8,7,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":17.158000000000001},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":16.193000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":6.6269999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.9260000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525952,"kb_used_data":228600,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93845888,"statfs":{"total":96636764160,"available":96098189312,"internally_reserved":0,"allocated":234086400,"data_stored":233910934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1,3,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":4.9850000000000003},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":5.0179999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.5540000000000003},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.492}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":3.5219999999999998},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":10.659000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526384,"kb_used_data":228584,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93845456,"statfs":{"total":96636764160,"available":96097746944,"internally_reserved":0,"allocated":234070016,"data_stored":233894550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.2870000000000008},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":12.313000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":23.850000000000001},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.1269999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":12.345000000000001},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":12.117000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538560,"kb_used_data":228280,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93833280,"statfs":{"total":96636764160,"available":96085278720,"internally_reserved":0,"allocated":233758720,"data_stored":233583254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,1,0,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":2.0659999999999998},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.22700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":2.0209999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":9.4700000000000006},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":17.488}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":232566784,"data_stored":232566784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233717760,"data_stored":233717760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233275392,"data_stored":233275392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233750528,"data_stored":233750528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:40.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:40.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9622", 'reported_seq': 9721, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:36.972775+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:36.972775+0000', 'last_peered': '2024-09-18T11:51:36.972775+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:36.972775+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9622, 'log_dups_size': 0, 'ondisk_log_size': 9622, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29720576, 'num_objects': 3628, 'num_object_clones': 0, 'num_object_copies': 14512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9622, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1525}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:40.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:40.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9283", 'reported_seq': 9398, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:37.342945+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:37.342945+0000', 'last_peered': '2024-09-18T11:51:37.342945+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:37.342945+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9283, 'log_dups_size': 0, 'ondisk_log_size': 9283, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28875411, 'num_objects': 3526, 'num_object_clones': 0, 'num_object_copies': 14104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3526, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9282, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1486}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9389", 'reported_seq': 9481, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:38.186591+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:38.186591+0000', 'last_peered': '2024-09-18T11:51:38.186591+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:38.186591+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9389, 'log_dups_size': 0, 'ondisk_log_size': 9389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28972278, 'num_objects': 3537, 'num_object_clones': 0, 'num_object_copies': 14148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9387, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1436}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9352", 'reported_seq': 9471, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:37.041615+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:37.041615+0000', 'last_peered': '2024-09-18T11:51:37.041615+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:37.041615+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9352, 'log_dups_size': 0, 'ondisk_log_size': 9352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29021430, 'num_objects': 3543, 'num_object_clones': 0, 'num_object_copies': 14172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9350, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1491}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9328", 'reported_seq': 9445, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:36.859337+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:36.859337+0000', 'last_peered': '2024-09-18T11:51:36.859337+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:36.859337+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9328, 'log_dups_size': 0, 'ondisk_log_size': 9328, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28327936, 'num_objects': 3458, 'num_object_clones': 0, 'num_object_copies': 13832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9328, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1339}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9499", 'reported_seq': 9604, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:38.178096+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:38.178096+0000', 'last_peered': '2024-09-18T11:51:38.178096+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:38.178096+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9499, 'log_dups_size': 0, 'ondisk_log_size': 9499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28808438, 'num_objects': 3517, 'num_object_clones': 0, 'num_object_copies': 14068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9497, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1401}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9296", 'reported_seq': 9403, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:38.395278+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:38.395278+0000', 'last_peered': '2024-09-18T11:51:38.395278+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:38.395278+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9296, 'log_dups_size': 0, 'ondisk_log_size': 9296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29316342, 'num_objects': 3579, 'num_object_clones': 0, 'num_object_copies': 14316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9294, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1565}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:40.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:40.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9549", 'reported_seq': 12570, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:36.970622+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:36.970622+0000', 'last_peered': '2024-09-18T11:51:36.970622+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:36.970622+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9549, 'log_dups_size': 0, 'ondisk_log_size': 9549, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29706732, 'num_objects': 3627, 'num_object_clones': 0, 'num_object_copies': 14508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 659, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9546, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1482, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2968}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:40.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:40.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9084", 'reported_seq': 9195, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:37.617189+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:37.617189+0000', 'last_peered': '2024-09-18T11:51:37.617189+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:37.617189+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9084, 'log_dups_size': 0, 'ondisk_log_size': 9084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28751094, 'num_objects': 3510, 'num_object_clones': 0, 'num_object_copies': 14040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9082, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1547}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:40.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:40.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9396", 'reported_seq': 9514, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:37.085615+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:37.085615+0000', 'last_peered': '2024-09-18T11:51:37.085615+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:37.085615+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9396, 'log_dups_size': 0, 'ondisk_log_size': 9396, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29790065, 'num_objects': 3637, 'num_object_clones': 0, 'num_object_copies': 14548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9393, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1603}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:40.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:40.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9452", 'reported_seq': 9543, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:37.011277+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:37.011277+0000', 'last_peered': '2024-09-18T11:51:37.011277+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:37.011277+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9452, 'log_dups_size': 0, 'ondisk_log_size': 9452, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29317755, 'num_objects': 3579, 'num_object_clones': 0, 'num_object_copies': 14316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9451, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1495}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:40.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:40.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9389", 'reported_seq': 9485, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:37.448215+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:37.448215+0000', 'last_peered': '2024-09-18T11:51:37.448215+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:37.448215+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9389, 'log_dups_size': 0, 'ondisk_log_size': 9389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29309563, 'num_objects': 3578, 'num_object_clones': 0, 'num_object_copies': 14312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9388, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1532}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:40.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:40.685 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9517", 'reported_seq': 9640, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:36.932261+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:36.932261+0000', 'last_peered': '2024-09-18T11:51:36.932261+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:36.932261+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9517, 'log_dups_size': 0, 'ondisk_log_size': 9517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29267190, 'num_objects': 3573, 'num_object_clones': 0, 'num_object_copies': 14292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9515, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:40.685 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:40.685 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9176", 'reported_seq': 9264, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:38.247171+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:38.247171+0000', 'last_peered': '2024-09-18T11:51:38.247171+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:38.247171+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9176, 'log_dups_size': 0, 'ondisk_log_size': 9176, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28923126, 'num_objects': 3531, 'num_object_clones': 0, 'num_object_copies': 14124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9174, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1529}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:40.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:41.320 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:41.320 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27476/227968 objects degraded (12.053%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":499,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56992,"data_bytes":466830732,"bytes_used":2173202432,"bytes_avail":384373854208,"bytes_total":386547056640,"degraded_objects":27476,"degraded_total":227968,"degraded_ratio":0.1205256878158338,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":204},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:41.342 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:41.342 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:41.923 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:41.950 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-18T11:51:40.011808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":466830732,"num_objects":56992,"num_object_clones":0,"num_object_copies":227968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150095,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5980,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150122,"ondisk_log_size":150122,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2122268,"kb_used_data":914796,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375365092,"statfs":{"total":386547056640,"available":384373854208,"internally_reserved":0,"allocated":936751104,"data_stored":936049240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,2,6,14,17,17,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20062208,"num_objects":-2449,"num_object_clones":0,"num_object_copies":-9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002930"},"pg_stats":[{"pgid":"1.f","version":"35'9622","reported_seq":9721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.972775+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:36.972775+0000","last_peered":"2024-09-18T11:51:36.972775+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:36.972775+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29720576,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9622,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9442","reported_seq":13732,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:37.286460+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:37.286460+0000","last_peered":"2024-09-18T11:51:37.286460+0000","last_clean":"2024-09-18T11:51:37.286460+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:37.286460+0000","last_undegraded":"2024-09-18T11:51:37.286460+0000","last_fullsized":"2024-09-18T11:51:37.286460+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9442,"log_dups_size":0,"ondisk_log_size":9442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29603062,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9440,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9283","reported_seq":9398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.342945+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:37.342945+0000","last_peered":"2024-09-18T11:51:37.342945+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:37.342945+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28875411,"num_objects":3526,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9282,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1486}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9389","reported_seq":9481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.186591+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:38.186591+0000","last_peered":"2024-09-18T11:51:38.186591+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:38.186591+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9389,"log_dups_size":0,"ondisk_log_size":9389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28972278,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9352","reported_seq":9471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.041615+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:37.041615+0000","last_peered":"2024-09-18T11:51:37.041615+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:37.041615+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29021430,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9328","reported_seq":9445,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.859337+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:36.859337+0000","last_peered":"2024-09-18T11:51:36.859337+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:36.859337+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28327936,"num_objects":3458,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9499","reported_seq":9604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.178096+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:38.178096+0000","last_peered":"2024-09-18T11:51:38.178096+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:38.178096+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9499,"log_dups_size":0,"ondisk_log_size":9499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28808438,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9497,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1401}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9296","reported_seq":9403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.395278+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:38.395278+0000","last_peered":"2024-09-18T11:51:38.395278+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:38.395278+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9296,"log_dups_size":0,"ondisk_log_size":9296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29316342,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9294,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1565}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9549","reported_seq":12570,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.970622+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:36.970622+0000","last_peered":"2024-09-18T11:51:36.970622+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:36.970622+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9549,"log_dups_size":0,"ondisk_log_size":9549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29706732,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9546,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1482,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9084","reported_seq":9195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.617189+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:37.617189+0000","last_peered":"2024-09-18T11:51:37.617189+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:37.617189+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9084,"log_dups_size":0,"ondisk_log_size":9084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28751094,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9082,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1547}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9396","reported_seq":9514,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.085615+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:37.085615+0000","last_peered":"2024-09-18T11:51:37.085615+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:37.085615+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29790065,"num_objects":3637,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9393,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9452","reported_seq":9543,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.011277+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:37.011277+0000","last_peered":"2024-09-18T11:51:37.011277+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:37.011277+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9452,"log_dups_size":0,"ondisk_log_size":9452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29317755,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9451,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1495}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9389","reported_seq":9485,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:37.448215+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:37.448215+0000","last_peered":"2024-09-18T11:51:37.448215+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:37.448215+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9389,"log_dups_size":0,"ondisk_log_size":9389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29309563,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9388,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1532}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9517","reported_seq":9640,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:36.932261+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:36.932261+0000","last_peered":"2024-09-18T11:51:36.932261+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:36.932261+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29267190,"num_objects":3573,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9348","reported_seq":13635,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:37.557305+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:37.557305+0000","last_peered":"2024-09-18T11:51:37.557305+0000","last_clean":"2024-09-18T11:51:37.557305+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:37.557305+0000","last_undegraded":"2024-09-18T11:51:37.557305+0000","last_fullsized":"2024-09-18T11:51:37.557305+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119734,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9346,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9176","reported_seq":9264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:38.247171+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:38.247171+0000","last_peered":"2024-09-18T11:51:38.247171+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:38.247171+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28923126,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":466830732,"num_objects":56992,"num_object_clones":0,"num_object_copies":227968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150095,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5980,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":933310464,"data_stored":933310464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150122,"ondisk_log_size":150122,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531372,"kb_used_data":229332,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93840468,"statfs":{"total":96636764160,"available":96092639232,"internally_reserved":0,"allocated":234835968,"data_stored":234660502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,8,7,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":17.158000000000001},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":16.193000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":6.6269999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.9260000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525952,"kb_used_data":228600,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93845888,"statfs":{"total":96636764160,"available":96098189312,"internally_reserved":0,"allocated":234086400,"data_stored":233910934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1,3,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":4.9850000000000003},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":5.0179999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.5540000000000003},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.492}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":3.5219999999999998},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":10.659000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526384,"kb_used_data":228584,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93845456,"statfs":{"total":96636764160,"available":96097746944,"internally_reserved":0,"allocated":234070016,"data_stored":233894550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.2870000000000008},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":12.313000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":23.850000000000001},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":5.1269999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":12.345000000000001},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":12.117000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538560,"kb_used_data":228280,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93833280,"statfs":{"total":96636764160,"available":96085278720,"internally_reserved":0,"allocated":233758720,"data_stored":233583254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,1,0,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":2.0659999999999998},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.22700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":2.0209999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":9.4700000000000006},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":17.488}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":232566784,"data_stored":232566784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233717760,"data_stored":233717760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233275392,"data_stored":233275392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233750528,"data_stored":233750528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:44.952 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:45.504 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:45.505 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:45.515 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-18T11:51:44.012700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":457917836,"num_objects":55904,"num_object_clones":0,"num_object_copies":223616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151183,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151210,"ondisk_log_size":151210,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2104180,"kb_used_data":896708,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375383180,"statfs":{"total":386547056640,"available":384392376320,"internally_reserved":0,"allocated":918228992,"data_stored":917527128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,8,5,14,5,20,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22863872,"num_objects":-2791,"num_object_clones":0,"num_object_copies":-11164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2791,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002792"},"pg_stats":[{"pgid":"1.f","version":"35'9695","reported_seq":9794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:41.960940+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:41.960940+0000","last_peered":"2024-09-18T11:51:41.960940+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:41.960940+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9695,"log_dups_size":0,"ondisk_log_size":9695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29122560,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9695,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9512","reported_seq":13802,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:42.574790+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:42.574790+0000","last_peered":"2024-09-18T11:51:42.574790+0000","last_clean":"2024-09-18T11:51:42.574790+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:42.574790+0000","last_undegraded":"2024-09-18T11:51:42.574790+0000","last_fullsized":"2024-09-18T11:51:42.574790+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29029622,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9359","reported_seq":9474,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.564336+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:42.564336+0000","last_peered":"2024-09-18T11:51:42.564336+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:42.564336+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9359,"log_dups_size":0,"ondisk_log_size":9359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28252819,"num_objects":3450,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9358,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1410}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9440","reported_seq":9532,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.135202+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:43.135202+0000","last_peered":"2024-09-18T11:51:43.135202+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:43.135202+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9440,"log_dups_size":0,"ondisk_log_size":9440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28554486,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9438,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1385}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9421","reported_seq":9539,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:41.992078+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:41.992078+0000","last_peered":"2024-09-18T11:51:41.992078+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:41.992078+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9421,"log_dups_size":0,"ondisk_log_size":9421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456182,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9419,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9397","reported_seq":9513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.153789+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:42.153789+0000","last_peered":"2024-09-18T11:51:42.153789+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:42.153789+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9397,"log_dups_size":0,"ondisk_log_size":9397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27762688,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9397,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1270}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9567","reported_seq":9672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.321751+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:43.321751+0000","last_peered":"2024-09-18T11:51:43.321751+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:43.321751+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9567,"log_dups_size":0,"ondisk_log_size":9567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28251382,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1333}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9366","reported_seq":9473,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.309829+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:43.309829+0000","last_peered":"2024-09-18T11:51:43.309829+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:43.309829+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28742902,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1495}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9626","reported_seq":12760,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.156516+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:42.156516+0000","last_peered":"2024-09-18T11:51:42.156516+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:42.156516+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29075948,"num_objects":3550,"num_object_clones":0,"num_object_copies":14200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9623,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1539,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9153","reported_seq":9265,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.552235+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:42.552235+0000","last_peered":"2024-09-18T11:51:42.552235+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:42.552235+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28185846,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9448","reported_seq":9567,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:41.923563+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:41.923563+0000","last_peered":"2024-09-18T11:51:41.923563+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:41.923563+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9448,"log_dups_size":0,"ondisk_log_size":9448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364081,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9445,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9525","reported_seq":9615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.141068+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:42.141068+0000","last_peered":"2024-09-18T11:51:42.141068+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:42.141068+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9525,"log_dups_size":0,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28719739,"num_objects":3506,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9524,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1422}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9449","reported_seq":9545,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.488800+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:42.488800+0000","last_peered":"2024-09-18T11:51:42.488800+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:42.488800+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28818043,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9448,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1472}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9587","reported_seq":9710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.098293+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:42.098293+0000","last_peered":"2024-09-18T11:51:42.098293+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:42.098293+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9587,"log_dups_size":0,"ondisk_log_size":9587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28693750,"num_objects":3503,"num_object_clones":0,"num_object_copies":14012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9585,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1380}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9424","reported_seq":13711,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:42.661242+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:42.661242+0000","last_peered":"2024-09-18T11:51:42.661242+0000","last_clean":"2024-09-18T11:51:42.661242+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:42.661242+0000","last_undegraded":"2024-09-18T11:51:42.661242+0000","last_fullsized":"2024-09-18T11:51:42.661242+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9424,"log_dups_size":0,"ondisk_log_size":9424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28497142,"num_objects":3479,"num_object_clones":0,"num_object_copies":13916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9422,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9241","reported_seq":9329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.416223+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:43.416223+0000","last_peered":"2024-09-18T11:51:43.416223+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:43.416223+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9241,"log_dups_size":0,"ondisk_log_size":9241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28390646,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9239,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":457917836,"num_objects":55904,"num_object_clones":0,"num_object_copies":223616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151183,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":915587072,"data_stored":915587072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151210,"ondisk_log_size":151210,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530444,"kb_used_data":224260,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93841396,"statfs":{"total":96636764160,"available":96093589504,"internally_reserved":0,"allocated":229642240,"data_stored":229466774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,3,0,4,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.181},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":5.04}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":5.0670000000000002},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.9409999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":12.069000000000001},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.36299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525808,"kb_used_data":224296,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93846032,"statfs":{"total":96636764160,"available":96098336768,"internally_reserved":0,"allocated":229679104,"data_stored":229503638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":12.289999999999999},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":12.319000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":16.372},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":16.885999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":12.347},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":17.888999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526212,"kb_used_data":224236,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93845628,"statfs":{"total":96636764160,"available":96097923072,"internally_reserved":0,"allocated":229617664,"data_stored":229442198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,2,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":10.807},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.1669999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":8.8100000000000005},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":6.6980000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.1379999999999999},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":10.449999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521716,"kb_used_data":223916,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93850124,"statfs":{"total":96636764160,"available":96102526976,"internally_reserved":0,"allocated":229289984,"data_stored":229114518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,0,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":8.9009999999999998},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.81299999999999994}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":8.7829999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":8.827},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":8.7509999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228167680,"data_stored":228167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":229220352,"data_stored":229220352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":228880384,"data_stored":228880384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229318656,"data_stored":229318656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:45.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:46.090 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:46.091 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:46.116 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-18T11:51:46.013211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":457917836,"num_objects":55904,"num_object_clones":0,"num_object_copies":223616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151183,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151210,"ondisk_log_size":151210,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2104180,"kb_used_data":896708,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375383180,"statfs":{"total":386547056640,"available":384392376320,"internally_reserved":0,"allocated":918228992,"data_stored":917527128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,8,5,14,5,20,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17702912,"num_objects":-2161,"num_object_clones":0,"num_object_copies":-8644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002816"},"pg_stats":[{"pgid":"1.f","version":"35'9695","reported_seq":9794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:41.960940+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:41.960940+0000","last_peered":"2024-09-18T11:51:41.960940+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:41.960940+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9695,"log_dups_size":0,"ondisk_log_size":9695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29122560,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9695,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9512","reported_seq":13802,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:42.574790+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:42.574790+0000","last_peered":"2024-09-18T11:51:42.574790+0000","last_clean":"2024-09-18T11:51:42.574790+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:42.574790+0000","last_undegraded":"2024-09-18T11:51:42.574790+0000","last_fullsized":"2024-09-18T11:51:42.574790+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29029622,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9359","reported_seq":9474,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.564336+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:42.564336+0000","last_peered":"2024-09-18T11:51:42.564336+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:42.564336+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9359,"log_dups_size":0,"ondisk_log_size":9359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28252819,"num_objects":3450,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9358,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1410}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9440","reported_seq":9532,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.135202+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:43.135202+0000","last_peered":"2024-09-18T11:51:43.135202+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:43.135202+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9440,"log_dups_size":0,"ondisk_log_size":9440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28554486,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9438,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1385}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9421","reported_seq":9539,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:41.992078+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:41.992078+0000","last_peered":"2024-09-18T11:51:41.992078+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:41.992078+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9421,"log_dups_size":0,"ondisk_log_size":9421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456182,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9419,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9397","reported_seq":9513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.153789+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:42.153789+0000","last_peered":"2024-09-18T11:51:42.153789+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:42.153789+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9397,"log_dups_size":0,"ondisk_log_size":9397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27762688,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9397,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1270}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9567","reported_seq":9672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.321751+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:43.321751+0000","last_peered":"2024-09-18T11:51:43.321751+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:43.321751+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9567,"log_dups_size":0,"ondisk_log_size":9567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28251382,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1333}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9366","reported_seq":9473,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.309829+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:43.309829+0000","last_peered":"2024-09-18T11:51:43.309829+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:43.309829+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28742902,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1495}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9626","reported_seq":12760,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.156516+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:42.156516+0000","last_peered":"2024-09-18T11:51:42.156516+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:42.156516+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29075948,"num_objects":3550,"num_object_clones":0,"num_object_copies":14200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9623,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1539,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9153","reported_seq":9265,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.552235+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:42.552235+0000","last_peered":"2024-09-18T11:51:42.552235+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:42.552235+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28185846,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9448","reported_seq":9567,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:41.923563+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:41.923563+0000","last_peered":"2024-09-18T11:51:41.923563+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:41.923563+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9448,"log_dups_size":0,"ondisk_log_size":9448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364081,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9445,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9525","reported_seq":9615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.141068+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:42.141068+0000","last_peered":"2024-09-18T11:51:42.141068+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:42.141068+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9525,"log_dups_size":0,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28719739,"num_objects":3506,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9524,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1422}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9449","reported_seq":9545,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.488800+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:42.488800+0000","last_peered":"2024-09-18T11:51:42.488800+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:42.488800+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28818043,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9448,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1472}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9587","reported_seq":9710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.098293+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:42.098293+0000","last_peered":"2024-09-18T11:51:42.098293+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:42.098293+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9587,"log_dups_size":0,"ondisk_log_size":9587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28693750,"num_objects":3503,"num_object_clones":0,"num_object_copies":14012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9585,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1380}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9424","reported_seq":13711,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:42.661242+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:42.661242+0000","last_peered":"2024-09-18T11:51:42.661242+0000","last_clean":"2024-09-18T11:51:42.661242+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:42.661242+0000","last_undegraded":"2024-09-18T11:51:42.661242+0000","last_fullsized":"2024-09-18T11:51:42.661242+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9424,"log_dups_size":0,"ondisk_log_size":9424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28497142,"num_objects":3479,"num_object_clones":0,"num_object_copies":13916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9422,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9241","reported_seq":9329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.416223+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:43.416223+0000","last_peered":"2024-09-18T11:51:43.416223+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:43.416223+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9241,"log_dups_size":0,"ondisk_log_size":9241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28390646,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9239,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":457917836,"num_objects":55904,"num_object_clones":0,"num_object_copies":223616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151183,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":915587072,"data_stored":915587072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151210,"ondisk_log_size":151210,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530444,"kb_used_data":224260,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93841396,"statfs":{"total":96636764160,"available":96093589504,"internally_reserved":0,"allocated":229642240,"data_stored":229466774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,3,0,4,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.181},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":5.04}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":5.0670000000000002},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.9409999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":12.069000000000001},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.36299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525808,"kb_used_data":224296,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93846032,"statfs":{"total":96636764160,"available":96098336768,"internally_reserved":0,"allocated":229679104,"data_stored":229503638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":12.289999999999999},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":12.319000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":16.372},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":16.885999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":12.347},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":17.888999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526212,"kb_used_data":224236,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93845628,"statfs":{"total":96636764160,"available":96097923072,"internally_reserved":0,"allocated":229617664,"data_stored":229442198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,2,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":10.807},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.1669999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":8.8100000000000005},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":6.6980000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.1379999999999999},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":10.449999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521716,"kb_used_data":223916,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93850124,"statfs":{"total":96636764160,"available":96102526976,"internally_reserved":0,"allocated":229289984,"data_stored":229114518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,0,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":8.9009999999999998},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.81299999999999994}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":8.7829999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":8.827},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":8.7509999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228167680,"data_stored":228167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":229220352,"data_stored":229220352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":228880384,"data_stored":228880384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229318656,"data_stored":229318656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:46.117 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:46.117 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9695", 'reported_seq': 9794, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:41.960940+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:41.960940+0000', 'last_peered': '2024-09-18T11:51:41.960940+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:41.960940+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9695, 'log_dups_size': 0, 'ondisk_log_size': 9695, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29122560, 'num_objects': 3555, 'num_object_clones': 0, 'num_object_copies': 14220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9695, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1452}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:46.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:46.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9359", 'reported_seq': 9474, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:42.564336+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:42.564336+0000', 'last_peered': '2024-09-18T11:51:42.564336+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:42.564336+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9359, 'log_dups_size': 0, 'ondisk_log_size': 9359, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28252819, 'num_objects': 3450, 'num_object_clones': 0, 'num_object_copies': 13800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3450, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9358, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1410}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:46.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:46.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9440", 'reported_seq': 9532, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:43.135202+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:43.135202+0000', 'last_peered': '2024-09-18T11:51:43.135202+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:43.135202+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9440, 'log_dups_size': 0, 'ondisk_log_size': 9440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28554486, 'num_objects': 3486, 'num_object_clones': 0, 'num_object_copies': 13944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9438, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1385}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:46.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:46.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9421", 'reported_seq': 9539, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:41.992078+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:41.992078+0000', 'last_peered': '2024-09-18T11:51:41.992078+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:41.992078+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9421, 'log_dups_size': 0, 'ondisk_log_size': 9421, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28456182, 'num_objects': 3474, 'num_object_clones': 0, 'num_object_copies': 13896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9419, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1422}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:46.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:46.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9397", 'reported_seq': 9513, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:42.153789+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:42.153789+0000', 'last_peered': '2024-09-18T11:51:42.153789+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:42.153789+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9397, 'log_dups_size': 0, 'ondisk_log_size': 9397, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27762688, 'num_objects': 3389, 'num_object_clones': 0, 'num_object_copies': 13556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9397, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1270}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:46.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:46.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9567", 'reported_seq': 9672, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:43.321751+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:43.321751+0000', 'last_peered': '2024-09-18T11:51:43.321751+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:43.321751+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9567, 'log_dups_size': 0, 'ondisk_log_size': 9567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28251382, 'num_objects': 3449, 'num_object_clones': 0, 'num_object_copies': 13796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9565, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1333}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:46.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:46.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9366", 'reported_seq': 9473, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:43.309829+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:43.309829+0000', 'last_peered': '2024-09-18T11:51:43.309829+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:43.309829+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9366, 'log_dups_size': 0, 'ondisk_log_size': 9366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28742902, 'num_objects': 3509, 'num_object_clones': 0, 'num_object_copies': 14036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9364, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1495}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:46.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:46.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9626", 'reported_seq': 12760, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:42.156516+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:42.156516+0000', 'last_peered': '2024-09-18T11:51:42.156516+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:42.156516+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9626, 'log_dups_size': 0, 'ondisk_log_size': 9626, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29075948, 'num_objects': 3550, 'num_object_clones': 0, 'num_object_copies': 14200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 602, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9623, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1539, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2948}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:46.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:46.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9153", 'reported_seq': 9265, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:42.552235+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:42.552235+0000', 'last_peered': '2024-09-18T11:51:42.552235+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:42.552235+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9153, 'log_dups_size': 0, 'ondisk_log_size': 9153, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28185846, 'num_objects': 3441, 'num_object_clones': 0, 'num_object_copies': 13764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9151, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1478}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:46.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:46.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9448", 'reported_seq': 9567, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:41.923563+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:41.923563+0000', 'last_peered': '2024-09-18T11:51:41.923563+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:41.923563+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9448, 'log_dups_size': 0, 'ondisk_log_size': 9448, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29364081, 'num_objects': 3585, 'num_object_clones': 0, 'num_object_copies': 14340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9445, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1551}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:46.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:46.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9525", 'reported_seq': 9615, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:42.141068+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:42.141068+0000', 'last_peered': '2024-09-18T11:51:42.141068+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:42.141068+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9525, 'log_dups_size': 0, 'ondisk_log_size': 9525, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28719739, 'num_objects': 3506, 'num_object_clones': 0, 'num_object_copies': 14024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9524, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1422}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:46.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:46.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9449", 'reported_seq': 9545, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:42.488800+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:42.488800+0000', 'last_peered': '2024-09-18T11:51:42.488800+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:42.488800+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9449, 'log_dups_size': 0, 'ondisk_log_size': 9449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28818043, 'num_objects': 3518, 'num_object_clones': 0, 'num_object_copies': 14072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9448, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:46.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:46.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9587", 'reported_seq': 9710, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:42.098293+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:42.098293+0000', 'last_peered': '2024-09-18T11:51:42.098293+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:42.098293+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9587, 'log_dups_size': 0, 'ondisk_log_size': 9587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28693750, 'num_objects': 3503, 'num_object_clones': 0, 'num_object_copies': 14012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9585, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1380}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:46.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:46.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9241", 'reported_seq': 9329, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:43.416223+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:43.416223+0000', 'last_peered': '2024-09-18T11:51:43.416223+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:43.416223+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9241, 'log_dups_size': 0, 'ondisk_log_size': 9241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28390646, 'num_objects': 3466, 'num_object_clones': 0, 'num_object_copies': 13864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9239, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:46.121 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:46.842 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:46.842 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27419/223616 objects degraded (12.262%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":504,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55904,"data_bytes":457917836,"bytes_used":2154680320,"bytes_avail":384392376320,"bytes_total":386547056640,"degraded_objects":27419,"degraded_total":223616,"degraded_ratio":0.1226164496279336,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:46.869 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:46.870 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:47.451 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:47.451 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:47.479 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-18T11:51:46.013211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":457917836,"num_objects":55904,"num_object_clones":0,"num_object_copies":223616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151183,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151210,"ondisk_log_size":151210,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2104180,"kb_used_data":896708,"kb_used_omap":14,"kb_used_meta":1207409,"kb_avail":375383180,"statfs":{"total":386547056640,"available":384392376320,"internally_reserved":0,"allocated":918228992,"data_stored":917527128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236386843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,8,5,14,5,20,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17702912,"num_objects":-2161,"num_object_clones":0,"num_object_copies":-8644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002816"},"pg_stats":[{"pgid":"1.f","version":"35'9695","reported_seq":9794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:41.960940+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:41.960940+0000","last_peered":"2024-09-18T11:51:41.960940+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:41.960940+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9695,"log_dups_size":0,"ondisk_log_size":9695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29122560,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9695,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9512","reported_seq":13802,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:42.574790+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:42.574790+0000","last_peered":"2024-09-18T11:51:42.574790+0000","last_clean":"2024-09-18T11:51:42.574790+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:42.574790+0000","last_undegraded":"2024-09-18T11:51:42.574790+0000","last_fullsized":"2024-09-18T11:51:42.574790+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29029622,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9359","reported_seq":9474,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.564336+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:42.564336+0000","last_peered":"2024-09-18T11:51:42.564336+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:42.564336+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9359,"log_dups_size":0,"ondisk_log_size":9359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28252819,"num_objects":3450,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9358,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1410}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9440","reported_seq":9532,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.135202+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:43.135202+0000","last_peered":"2024-09-18T11:51:43.135202+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:43.135202+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9440,"log_dups_size":0,"ondisk_log_size":9440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28554486,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9438,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1385}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9421","reported_seq":9539,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:41.992078+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:41.992078+0000","last_peered":"2024-09-18T11:51:41.992078+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:41.992078+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9421,"log_dups_size":0,"ondisk_log_size":9421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456182,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9419,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9397","reported_seq":9513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.153789+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:42.153789+0000","last_peered":"2024-09-18T11:51:42.153789+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:42.153789+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9397,"log_dups_size":0,"ondisk_log_size":9397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27762688,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9397,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1270}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9567","reported_seq":9672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.321751+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:43.321751+0000","last_peered":"2024-09-18T11:51:43.321751+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:43.321751+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9567,"log_dups_size":0,"ondisk_log_size":9567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28251382,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1333}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9366","reported_seq":9473,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.309829+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:43.309829+0000","last_peered":"2024-09-18T11:51:43.309829+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:43.309829+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28742902,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1495}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9626","reported_seq":12760,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.156516+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:42.156516+0000","last_peered":"2024-09-18T11:51:42.156516+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:42.156516+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29075948,"num_objects":3550,"num_object_clones":0,"num_object_copies":14200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9623,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1539,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9153","reported_seq":9265,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.552235+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:42.552235+0000","last_peered":"2024-09-18T11:51:42.552235+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:42.552235+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28185846,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9448","reported_seq":9567,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:41.923563+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:41.923563+0000","last_peered":"2024-09-18T11:51:41.923563+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:41.923563+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9448,"log_dups_size":0,"ondisk_log_size":9448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364081,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9445,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9525","reported_seq":9615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.141068+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:42.141068+0000","last_peered":"2024-09-18T11:51:42.141068+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:42.141068+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9525,"log_dups_size":0,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28719739,"num_objects":3506,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9524,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1422}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9449","reported_seq":9545,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.488800+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:42.488800+0000","last_peered":"2024-09-18T11:51:42.488800+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:42.488800+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28818043,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9448,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1472}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9587","reported_seq":9710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:42.098293+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:42.098293+0000","last_peered":"2024-09-18T11:51:42.098293+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:42.098293+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9587,"log_dups_size":0,"ondisk_log_size":9587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28693750,"num_objects":3503,"num_object_clones":0,"num_object_copies":14012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9585,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1380}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9424","reported_seq":13711,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:42.661242+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:42.661242+0000","last_peered":"2024-09-18T11:51:42.661242+0000","last_clean":"2024-09-18T11:51:42.661242+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:42.661242+0000","last_undegraded":"2024-09-18T11:51:42.661242+0000","last_fullsized":"2024-09-18T11:51:42.661242+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9424,"log_dups_size":0,"ondisk_log_size":9424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28497142,"num_objects":3479,"num_object_clones":0,"num_object_copies":13916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9422,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9241","reported_seq":9329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:43.416223+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:43.416223+0000","last_peered":"2024-09-18T11:51:43.416223+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:43.416223+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9241,"log_dups_size":0,"ondisk_log_size":9241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28390646,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9239,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":457917836,"num_objects":55904,"num_object_clones":0,"num_object_copies":223616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151183,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":915587072,"data_stored":915587072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151210,"ondisk_log_size":151210,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530444,"kb_used_data":224260,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93841396,"statfs":{"total":96636764160,"available":96093589504,"internally_reserved":0,"allocated":229642240,"data_stored":229466774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,3,0,4,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.181},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":5.04}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":5.0670000000000002},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":4.9409999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":12.069000000000001},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.36299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525808,"kb_used_data":224296,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93846032,"statfs":{"total":96636764160,"available":96098336768,"internally_reserved":0,"allocated":229679104,"data_stored":229503638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":12.289999999999999},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":12.319000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":16.372},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":16.885999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":12.347},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":17.888999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526212,"kb_used_data":224236,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93845628,"statfs":{"total":96636764160,"available":96097923072,"internally_reserved":0,"allocated":229617664,"data_stored":229442198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,2,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":10.807},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.1669999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":8.8100000000000005},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":6.6980000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":6.1379999999999999},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":10.449999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521716,"kb_used_data":223916,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93850124,"statfs":{"total":96636764160,"available":96102526976,"internally_reserved":0,"allocated":229289984,"data_stored":229114518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,0,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":8.9009999999999998},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":0.81299999999999994}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":8.7829999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":8.827},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":8.7509999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228167680,"data_stored":228167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":229220352,"data_stored":229220352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":228880384,"data_stored":228880384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229318656,"data_stored":229318656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:50.481 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:51.115 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:51.115 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:51.138 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-18T11:51:50.014144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449299852,"num_objects":54852,"num_object_clones":0,"num_object_copies":219408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152235,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6098,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152262,"ondisk_log_size":152262,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2104724,"kb_used_data":880620,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375382636,"statfs":{"total":386547056640,"available":384391819264,"internally_reserved":0,"allocated":901754880,"data_stored":901053016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,6,22,8,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19898368,"num_objects":-2429,"num_object_clones":0,"num_object_copies":-9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002826"},"pg_stats":[{"pgid":"1.f","version":"35'9766","reported_seq":9865,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.115206+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:47.115206+0000","last_peered":"2024-09-18T11:51:47.115206+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:47.115206+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9766,"log_dups_size":0,"ondisk_log_size":9766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28540928,"num_objects":3484,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1381}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9573","reported_seq":13863,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:47.636446+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:47.636446+0000","last_peered":"2024-09-18T11:51:47.636446+0000","last_clean":"2024-09-18T11:51:47.636446+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:47.636446+0000","last_undegraded":"2024-09-18T11:51:47.636446+0000","last_fullsized":"2024-09-18T11:51:47.636446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9573,"log_dups_size":0,"ondisk_log_size":9573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28529910,"num_objects":3483,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9431","reported_seq":9546,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.533278+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:47.533278+0000","last_peered":"2024-09-18T11:51:47.533278+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:47.533278+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27662995,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9430,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9517","reported_seq":9609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:48.205945+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:48.205945+0000","last_peered":"2024-09-18T11:51:48.205945+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:48.205945+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27923702,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9492","reported_seq":9611,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.121986+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:47.121986+0000","last_peered":"2024-09-18T11:51:47.121986+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:47.121986+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9492,"log_dups_size":0,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27874550,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9490,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9458","reported_seq":9573,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.111107+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:47.111107+0000","last_peered":"2024-09-18T11:51:47.111107+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:47.111107+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9458,"log_dups_size":0,"ondisk_log_size":9458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9636","reported_seq":9741,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:48.270974+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:48.270974+0000","last_peered":"2024-09-18T11:51:48.270974+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:48.270974+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27686134,"num_objects":3380,"num_object_clones":0,"num_object_copies":13520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1264}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9430","reported_seq":9537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:48.349200+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:48.349200+0000","last_peered":"2024-09-18T11:51:48.349200+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:48.349200+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9430,"log_dups_size":0,"ondisk_log_size":9430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28218614,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9686","reported_seq":12942,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.065323+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:47.065323+0000","last_peered":"2024-09-18T11:51:47.065323+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:47.065323+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9686,"log_dups_size":0,"ondisk_log_size":9686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28584428,"num_objects":3490,"num_object_clones":0,"num_object_copies":13960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1600,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9215","reported_seq":9327,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.603478+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:47.603478+0000","last_peered":"2024-09-18T11:51:47.603478+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:47.603478+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9215,"log_dups_size":0,"ondisk_log_size":9215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27677942,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1416}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9513","reported_seq":9632,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.055162+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:47.055162+0000","last_peered":"2024-09-18T11:51:47.055162+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:47.055162+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9513,"log_dups_size":0,"ondisk_log_size":9513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28831601,"num_objects":3520,"num_object_clones":0,"num_object_copies":14080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9593","reported_seq":9684,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:46.966178+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:46.966178+0000","last_peered":"2024-09-18T11:51:46.966178+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:46.966178+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9593,"log_dups_size":0,"ondisk_log_size":9593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28162683,"num_objects":3438,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9592,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1354}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9506","reported_seq":9602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.604706+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:47.604706+0000","last_peered":"2024-09-18T11:51:47.604706+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:47.604706+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9506,"log_dups_size":0,"ondisk_log_size":9506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28351099,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9505,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9642","reported_seq":9765,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.012824+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:47.012824+0000","last_peered":"2024-09-18T11:51:47.012824+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:47.012824+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28243190,"num_objects":3448,"num_object_clones":0,"num_object_copies":13792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9640,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9504","reported_seq":13791,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:47.732732+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:47.732732+0000","last_peered":"2024-09-18T11:51:47.732732+0000","last_clean":"2024-09-18T11:51:47.732732+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:47.732732+0000","last_undegraded":"2024-09-18T11:51:47.732732+0000","last_fullsized":"2024-09-18T11:51:47.732732+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27841782,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9300","reported_seq":9388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.965670+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:47.965670+0000","last_peered":"2024-09-18T11:51:47.965670+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:47.965670+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9300,"log_dups_size":0,"ondisk_log_size":9300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27907318,"num_objects":3407,"num_object_clones":0,"num_object_copies":13628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":449299852,"num_objects":54852,"num_object_clones":0,"num_object_copies":219408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152235,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6098,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":898539520,"data_stored":898539520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152262,"ondisk_log_size":152262,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530404,"kb_used_data":220060,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93841436,"statfs":{"total":96636764160,"available":96093630464,"internally_reserved":0,"allocated":225341440,"data_stored":225165974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,8,0,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.1029999999999998},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":6.9130000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":7.1379999999999999},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":3.1709999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.2119999999999997},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":7.5709999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525732,"kb_used_data":220060,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93846108,"statfs":{"total":96636764160,"available":96098414592,"internally_reserved":0,"allocated":225341440,"data_stored":225165974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1,1,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.4969999999999999},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":6.5199999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.3289999999999997},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.98999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":7.9909999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526844,"kb_used_data":220720,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93844996,"statfs":{"total":96636764160,"available":96097275904,"internally_reserved":0,"allocated":226017280,"data_stored":225841814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,9,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.2999999999999998},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":11.895},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.5389999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.443},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.4480000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521744,"kb_used_data":219780,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93850096,"statfs":{"total":96636764160,"available":96102498304,"internally_reserved":0,"allocated":225054720,"data_stored":224879254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":9.2050000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":8.9399999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":5.4450000000000003},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":8.8949999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":9.173},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":25.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":223911936,"data_stored":223911936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224948224,"data_stored":224948224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224641024,"data_stored":224641024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225038336,"data_stored":225038336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:51.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:51.710 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:51.710 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:51.738 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-18T11:51:50.014144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449299852,"num_objects":54852,"num_object_clones":0,"num_object_copies":219408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152235,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6098,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152262,"ondisk_log_size":152262,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2104724,"kb_used_data":880620,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375382636,"statfs":{"total":386547056640,"available":384391819264,"internally_reserved":0,"allocated":901754880,"data_stored":901053016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,6,22,8,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19898368,"num_objects":-2429,"num_object_clones":0,"num_object_copies":-9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002826"},"pg_stats":[{"pgid":"1.f","version":"35'9766","reported_seq":9865,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.115206+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:47.115206+0000","last_peered":"2024-09-18T11:51:47.115206+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:47.115206+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9766,"log_dups_size":0,"ondisk_log_size":9766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28540928,"num_objects":3484,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1381}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9573","reported_seq":13863,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:47.636446+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:47.636446+0000","last_peered":"2024-09-18T11:51:47.636446+0000","last_clean":"2024-09-18T11:51:47.636446+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:47.636446+0000","last_undegraded":"2024-09-18T11:51:47.636446+0000","last_fullsized":"2024-09-18T11:51:47.636446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9573,"log_dups_size":0,"ondisk_log_size":9573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28529910,"num_objects":3483,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9431","reported_seq":9546,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.533278+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:47.533278+0000","last_peered":"2024-09-18T11:51:47.533278+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:47.533278+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27662995,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9430,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9517","reported_seq":9609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:48.205945+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:48.205945+0000","last_peered":"2024-09-18T11:51:48.205945+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:48.205945+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27923702,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9492","reported_seq":9611,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.121986+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:47.121986+0000","last_peered":"2024-09-18T11:51:47.121986+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:47.121986+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9492,"log_dups_size":0,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27874550,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9490,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9458","reported_seq":9573,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.111107+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:47.111107+0000","last_peered":"2024-09-18T11:51:47.111107+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:47.111107+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9458,"log_dups_size":0,"ondisk_log_size":9458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9636","reported_seq":9741,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:48.270974+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:48.270974+0000","last_peered":"2024-09-18T11:51:48.270974+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:48.270974+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27686134,"num_objects":3380,"num_object_clones":0,"num_object_copies":13520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1264}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9430","reported_seq":9537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:48.349200+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:48.349200+0000","last_peered":"2024-09-18T11:51:48.349200+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:48.349200+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9430,"log_dups_size":0,"ondisk_log_size":9430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28218614,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9686","reported_seq":12942,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.065323+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:47.065323+0000","last_peered":"2024-09-18T11:51:47.065323+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:47.065323+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9686,"log_dups_size":0,"ondisk_log_size":9686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28584428,"num_objects":3490,"num_object_clones":0,"num_object_copies":13960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1600,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9215","reported_seq":9327,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.603478+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:47.603478+0000","last_peered":"2024-09-18T11:51:47.603478+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:47.603478+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9215,"log_dups_size":0,"ondisk_log_size":9215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27677942,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1416}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9513","reported_seq":9632,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.055162+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:47.055162+0000","last_peered":"2024-09-18T11:51:47.055162+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:47.055162+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9513,"log_dups_size":0,"ondisk_log_size":9513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28831601,"num_objects":3520,"num_object_clones":0,"num_object_copies":14080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9593","reported_seq":9684,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:46.966178+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:46.966178+0000","last_peered":"2024-09-18T11:51:46.966178+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:46.966178+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9593,"log_dups_size":0,"ondisk_log_size":9593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28162683,"num_objects":3438,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9592,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1354}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9506","reported_seq":9602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.604706+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:47.604706+0000","last_peered":"2024-09-18T11:51:47.604706+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:47.604706+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9506,"log_dups_size":0,"ondisk_log_size":9506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28351099,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9505,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9642","reported_seq":9765,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.012824+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:47.012824+0000","last_peered":"2024-09-18T11:51:47.012824+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:47.012824+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28243190,"num_objects":3448,"num_object_clones":0,"num_object_copies":13792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9640,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9504","reported_seq":13791,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:47.732732+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:47.732732+0000","last_peered":"2024-09-18T11:51:47.732732+0000","last_clean":"2024-09-18T11:51:47.732732+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:47.732732+0000","last_undegraded":"2024-09-18T11:51:47.732732+0000","last_fullsized":"2024-09-18T11:51:47.732732+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27841782,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9300","reported_seq":9388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.965670+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:47.965670+0000","last_peered":"2024-09-18T11:51:47.965670+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:47.965670+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9300,"log_dups_size":0,"ondisk_log_size":9300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27907318,"num_objects":3407,"num_object_clones":0,"num_object_copies":13628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":449299852,"num_objects":54852,"num_object_clones":0,"num_object_copies":219408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152235,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6098,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":898539520,"data_stored":898539520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152262,"ondisk_log_size":152262,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530404,"kb_used_data":220060,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93841436,"statfs":{"total":96636764160,"available":96093630464,"internally_reserved":0,"allocated":225341440,"data_stored":225165974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,8,0,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.1029999999999998},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":6.9130000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":7.1379999999999999},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":3.1709999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.2119999999999997},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":7.5709999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525732,"kb_used_data":220060,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93846108,"statfs":{"total":96636764160,"available":96098414592,"internally_reserved":0,"allocated":225341440,"data_stored":225165974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1,1,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.4969999999999999},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":6.5199999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.3289999999999997},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.98999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":7.9909999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526844,"kb_used_data":220720,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93844996,"statfs":{"total":96636764160,"available":96097275904,"internally_reserved":0,"allocated":226017280,"data_stored":225841814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,9,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.2999999999999998},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":11.895},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.5389999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.443},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.4480000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521744,"kb_used_data":219780,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93850096,"statfs":{"total":96636764160,"available":96102498304,"internally_reserved":0,"allocated":225054720,"data_stored":224879254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":9.2050000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":8.9399999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":5.4450000000000003},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":8.8949999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":9.173},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":25.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":223911936,"data_stored":223911936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224948224,"data_stored":224948224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224641024,"data_stored":224641024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225038336,"data_stored":225038336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:51.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:51.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9766", 'reported_seq': 9865, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:47.115206+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:47.115206+0000', 'last_peered': '2024-09-18T11:51:47.115206+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:47.115206+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9766, 'log_dups_size': 0, 'ondisk_log_size': 9766, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28540928, 'num_objects': 3484, 'num_object_clones': 0, 'num_object_copies': 13936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9766, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1381}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:51.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:51.741 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9431", 'reported_seq': 9546, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:47.533278+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:47.533278+0000', 'last_peered': '2024-09-18T11:51:47.533278+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:47.533278+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9431, 'log_dups_size': 0, 'ondisk_log_size': 9431, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27662995, 'num_objects': 3378, 'num_object_clones': 0, 'num_object_copies': 13512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3378, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9430, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1338}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:51.741 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:51.741 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9517", 'reported_seq': 9609, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:48.205945+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:48.205945+0000', 'last_peered': '2024-09-18T11:51:48.205945+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:48.205945+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9517, 'log_dups_size': 0, 'ondisk_log_size': 9517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27923702, 'num_objects': 3409, 'num_object_clones': 0, 'num_object_copies': 13636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9515, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1308}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:51.741 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:51.741 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9492", 'reported_seq': 9611, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:47.121986+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:47.121986+0000', 'last_peered': '2024-09-18T11:51:47.121986+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:47.121986+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9492, 'log_dups_size': 0, 'ondisk_log_size': 9492, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27874550, 'num_objects': 3403, 'num_object_clones': 0, 'num_object_copies': 13612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9490, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:51.741 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:51.741 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9458", 'reported_seq': 9573, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:47.111107+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:47.111107+0000', 'last_peered': '2024-09-18T11:51:47.111107+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:47.111107+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9458, 'log_dups_size': 0, 'ondisk_log_size': 9458, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27262976, 'num_objects': 3328, 'num_object_clones': 0, 'num_object_copies': 13312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9458, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1209}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:51.742 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:51.742 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9636", 'reported_seq': 9741, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:48.270974+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:48.270974+0000', 'last_peered': '2024-09-18T11:51:48.270974+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:48.270974+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9636, 'log_dups_size': 0, 'ondisk_log_size': 9636, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27686134, 'num_objects': 3380, 'num_object_clones': 0, 'num_object_copies': 13520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9634, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1264}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:51.742 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:51.742 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9430", 'reported_seq': 9537, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:48.349200+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:48.349200+0000', 'last_peered': '2024-09-18T11:51:48.349200+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:48.349200+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9430, 'log_dups_size': 0, 'ondisk_log_size': 9430, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28218614, 'num_objects': 3445, 'num_object_clones': 0, 'num_object_copies': 13780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9428, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1431}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:51.743 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:51.743 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9686", 'reported_seq': 12942, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:47.065323+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:47.065323+0000', 'last_peered': '2024-09-18T11:51:47.065323+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:47.065323+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9686, 'log_dups_size': 0, 'ondisk_log_size': 9686, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28584428, 'num_objects': 3490, 'num_object_clones': 0, 'num_object_copies': 13960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 541, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9683, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1600, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2949}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:51.743 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:51.743 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9215", 'reported_seq': 9327, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:47.603478+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:47.603478+0000', 'last_peered': '2024-09-18T11:51:47.603478+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:47.603478+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9215, 'log_dups_size': 0, 'ondisk_log_size': 9215, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27677942, 'num_objects': 3379, 'num_object_clones': 0, 'num_object_copies': 13516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9213, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1416}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:51.743 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:51.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9513", 'reported_seq': 9632, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:47.055162+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:47.055162+0000', 'last_peered': '2024-09-18T11:51:47.055162+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:47.055162+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9513, 'log_dups_size': 0, 'ondisk_log_size': 9513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28831601, 'num_objects': 3520, 'num_object_clones': 0, 'num_object_copies': 14080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9510, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1486}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:51.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:51.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9593", 'reported_seq': 9684, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:46.966178+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:46.966178+0000', 'last_peered': '2024-09-18T11:51:46.966178+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:46.966178+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9593, 'log_dups_size': 0, 'ondisk_log_size': 9593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28162683, 'num_objects': 3438, 'num_object_clones': 0, 'num_object_copies': 13752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9592, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1354}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:51.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:51.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9506", 'reported_seq': 9602, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:47.604706+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:47.604706+0000', 'last_peered': '2024-09-18T11:51:47.604706+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:47.604706+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9506, 'log_dups_size': 0, 'ondisk_log_size': 9506, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28351099, 'num_objects': 3461, 'num_object_clones': 0, 'num_object_copies': 13844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9505, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1415}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:51.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:51.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9642", 'reported_seq': 9765, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:47.012824+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:47.012824+0000', 'last_peered': '2024-09-18T11:51:47.012824+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:47.012824+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9642, 'log_dups_size': 0, 'ondisk_log_size': 9642, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28243190, 'num_objects': 3448, 'num_object_clones': 0, 'num_object_copies': 13792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3448, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9640, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1325}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:51.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:51.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9300", 'reported_seq': 9388, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:47.965670+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:47.965670+0000', 'last_peered': '2024-09-18T11:51:47.965670+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:47.965670+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9300, 'log_dups_size': 0, 'ondisk_log_size': 9300, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27907318, 'num_objects': 3407, 'num_object_clones': 0, 'num_object_copies': 13628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9298, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1405}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:51.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:52.481 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:52.481 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27358/219408 objects degraded (12.469%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":510,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54852,"data_bytes":449299852,"bytes_used":2155237376,"bytes_avail":384391819264,"bytes_total":386547056640,"degraded_objects":27358,"degraded_total":219408,"degraded_ratio":0.12469007511120835,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:52.511 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:52.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:53.124 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:53.126 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:53.150 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-18T11:51:52.014607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449299852,"num_objects":54852,"num_object_clones":0,"num_object_copies":219408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152235,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6098,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152262,"ondisk_log_size":152262,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2104724,"kb_used_data":880620,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375382636,"statfs":{"total":386547056640,"available":384391819264,"internally_reserved":0,"allocated":901754880,"data_stored":901053016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,6,22,8,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17530880,"num_objects":-2140,"num_object_clones":0,"num_object_copies":-8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002799"},"pg_stats":[{"pgid":"1.f","version":"35'9766","reported_seq":9865,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.115206+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:47.115206+0000","last_peered":"2024-09-18T11:51:47.115206+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:47.115206+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9766,"log_dups_size":0,"ondisk_log_size":9766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28540928,"num_objects":3484,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1381}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9573","reported_seq":13863,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:47.636446+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:47.636446+0000","last_peered":"2024-09-18T11:51:47.636446+0000","last_clean":"2024-09-18T11:51:47.636446+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:47.636446+0000","last_undegraded":"2024-09-18T11:51:47.636446+0000","last_fullsized":"2024-09-18T11:51:47.636446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9573,"log_dups_size":0,"ondisk_log_size":9573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28529910,"num_objects":3483,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9431","reported_seq":9546,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.533278+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:47.533278+0000","last_peered":"2024-09-18T11:51:47.533278+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:47.533278+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27662995,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9430,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9517","reported_seq":9609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:48.205945+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:48.205945+0000","last_peered":"2024-09-18T11:51:48.205945+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:48.205945+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27923702,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9492","reported_seq":9611,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.121986+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:47.121986+0000","last_peered":"2024-09-18T11:51:47.121986+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:47.121986+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9492,"log_dups_size":0,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27874550,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9490,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9458","reported_seq":9573,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.111107+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:47.111107+0000","last_peered":"2024-09-18T11:51:47.111107+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:47.111107+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9458,"log_dups_size":0,"ondisk_log_size":9458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9636","reported_seq":9741,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:48.270974+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:48.270974+0000","last_peered":"2024-09-18T11:51:48.270974+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:48.270974+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27686134,"num_objects":3380,"num_object_clones":0,"num_object_copies":13520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1264}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9430","reported_seq":9537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:48.349200+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:48.349200+0000","last_peered":"2024-09-18T11:51:48.349200+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:48.349200+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9430,"log_dups_size":0,"ondisk_log_size":9430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28218614,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9686","reported_seq":12942,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.065323+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:47.065323+0000","last_peered":"2024-09-18T11:51:47.065323+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:47.065323+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9686,"log_dups_size":0,"ondisk_log_size":9686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28584428,"num_objects":3490,"num_object_clones":0,"num_object_copies":13960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1600,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9215","reported_seq":9327,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.603478+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:47.603478+0000","last_peered":"2024-09-18T11:51:47.603478+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:47.603478+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9215,"log_dups_size":0,"ondisk_log_size":9215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27677942,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1416}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9513","reported_seq":9632,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.055162+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:47.055162+0000","last_peered":"2024-09-18T11:51:47.055162+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:47.055162+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9513,"log_dups_size":0,"ondisk_log_size":9513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28831601,"num_objects":3520,"num_object_clones":0,"num_object_copies":14080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9593","reported_seq":9684,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:46.966178+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:46.966178+0000","last_peered":"2024-09-18T11:51:46.966178+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:46.966178+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9593,"log_dups_size":0,"ondisk_log_size":9593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28162683,"num_objects":3438,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9592,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1354}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9506","reported_seq":9602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.604706+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:47.604706+0000","last_peered":"2024-09-18T11:51:47.604706+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:47.604706+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9506,"log_dups_size":0,"ondisk_log_size":9506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28351099,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9505,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9642","reported_seq":9765,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.012824+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:47.012824+0000","last_peered":"2024-09-18T11:51:47.012824+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:47.012824+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28243190,"num_objects":3448,"num_object_clones":0,"num_object_copies":13792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9640,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9504","reported_seq":13791,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:47.732732+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:47.732732+0000","last_peered":"2024-09-18T11:51:47.732732+0000","last_clean":"2024-09-18T11:51:47.732732+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:47.732732+0000","last_undegraded":"2024-09-18T11:51:47.732732+0000","last_fullsized":"2024-09-18T11:51:47.732732+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27841782,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9300","reported_seq":9388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:47.965670+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:47.965670+0000","last_peered":"2024-09-18T11:51:47.965670+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:47.965670+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9300,"log_dups_size":0,"ondisk_log_size":9300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27907318,"num_objects":3407,"num_object_clones":0,"num_object_copies":13628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":449299852,"num_objects":54852,"num_object_clones":0,"num_object_copies":219408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152235,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6098,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":898539520,"data_stored":898539520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152262,"ondisk_log_size":152262,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530404,"kb_used_data":220060,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93841436,"statfs":{"total":96636764160,"available":96093630464,"internally_reserved":0,"allocated":225341440,"data_stored":225165974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,8,0,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.1029999999999998},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":6.9130000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":7.1379999999999999},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":3.1709999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.2119999999999997},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":7.5709999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525732,"kb_used_data":220060,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93846108,"statfs":{"total":96636764160,"available":96098414592,"internally_reserved":0,"allocated":225341440,"data_stored":225165974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1,1,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.4969999999999999},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":6.5199999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.3289999999999997},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.98999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":7.9909999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526844,"kb_used_data":220720,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93844996,"statfs":{"total":96636764160,"available":96097275904,"internally_reserved":0,"allocated":226017280,"data_stored":225841814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,9,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":2.2999999999999998},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":11.895},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":7.5389999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.443},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":4.4480000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521744,"kb_used_data":219780,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93850096,"statfs":{"total":96636764160,"available":96102498304,"internally_reserved":0,"allocated":225054720,"data_stored":224879254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":9.2050000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":8.9399999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":5.4450000000000003},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":8.8949999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":9.173},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":25.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":223911936,"data_stored":223911936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224948224,"data_stored":224948224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224641024,"data_stored":224641024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225038336,"data_stored":225038336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:56.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:56.943 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:56.943 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:56.974 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-18T11:51:56.015583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":440681868,"num_objects":53800,"num_object_clones":0,"num_object_copies":215200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153287,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6157,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153314,"ondisk_log_size":153314,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2087636,"kb_used_data":863536,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375399724,"statfs":{"total":386547056640,"available":384409317376,"internally_reserved":0,"allocated":884260864,"data_stored":883559000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,5,4,9,11,16,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17235968,"num_objects":-2104,"num_object_clones":0,"num_object_copies":-8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2104,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002883"},"pg_stats":[{"pgid":"1.f","version":"35'9832","reported_seq":9930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:51.995937+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:51.995937+0000","last_peered":"2024-09-18T11:51:51.995937+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:51.995937+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28000256,"num_objects":3418,"num_object_clones":0,"num_object_copies":13672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9832,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9641","reported_seq":13931,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:52.722662+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:52.722662+0000","last_peered":"2024-09-18T11:51:52.722662+0000","last_clean":"2024-09-18T11:51:52.722662+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:52.722662+0000","last_undegraded":"2024-09-18T11:51:52.722662+0000","last_fullsized":"2024-09-18T11:51:52.722662+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9641,"log_dups_size":0,"ondisk_log_size":9641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27972854,"num_objects":3415,"num_object_clones":0,"num_object_copies":13660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9639,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9494","reported_seq":9609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.429385+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:52.429385+0000","last_peered":"2024-09-18T11:51:52.429385+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:52.429385+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9494,"log_dups_size":0,"ondisk_log_size":9494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27146899,"num_objects":3315,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9493,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1275}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9589","reported_seq":9681,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.385765+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:53.385765+0000","last_peered":"2024-09-18T11:51:53.385765+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:53.385765+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9589,"log_dups_size":0,"ondisk_log_size":9589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27333878,"num_objects":3337,"num_object_clones":0,"num_object_copies":13348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9587,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9546","reported_seq":9665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:51.931264+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:51.931264+0000","last_peered":"2024-09-18T11:51:51.931264+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:51.931264+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9546,"log_dups_size":0,"ondisk_log_size":9546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432182,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1297}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9521","reported_seq":9638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:51.892834+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:51.892834+0000","last_peered":"2024-09-18T11:51:51.892834+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:51.892834+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9521,"log_dups_size":0,"ondisk_log_size":9521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26746880,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9521,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9696","reported_seq":9801,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.276500+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:53.276500+0000","last_peered":"2024-09-18T11:51:53.276500+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:53.276500+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9696,"log_dups_size":0,"ondisk_log_size":9696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27194614,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9694,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1204}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9490","reported_seq":9597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.291153+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:53.291153+0000","last_peered":"2024-09-18T11:51:53.291153+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:53.291153+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9490,"log_dups_size":0,"ondisk_log_size":9490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27727094,"num_objects":3385,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9488,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9755","reported_seq":13130,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.164887+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:52.164887+0000","last_peered":"2024-09-18T11:51:52.164887+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:52.164887+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28019180,"num_objects":3421,"num_object_clones":0,"num_object_copies":13684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9752,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1659,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9282","reported_seq":9394,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.538475+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:52.538475+0000","last_peered":"2024-09-18T11:51:52.538475+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:52.538475+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27129078,"num_objects":3312,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9584","reported_seq":9703,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.086279+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:52.086279+0000","last_peered":"2024-09-18T11:51:52.086279+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:52.086279+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28249969,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9581,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9660","reported_seq":9748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.181144+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:52.181144+0000","last_peered":"2024-09-18T11:51:52.181144+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:52.181144+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9660,"log_dups_size":0,"ondisk_log_size":9660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27613819,"num_objects":3371,"num_object_clones":0,"num_object_copies":13484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9659,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1287}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9579","reported_seq":9675,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.485668+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:52.485668+0000","last_peered":"2024-09-18T11:51:52.485668+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:52.485668+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9579,"log_dups_size":0,"ondisk_log_size":9579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27753083,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9578,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1342}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9709","reported_seq":9832,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.028793+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:52.028793+0000","last_peered":"2024-09-18T11:51:52.028793+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:52.028793+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27694326,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1258}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9564","reported_seq":13851,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:52.510022+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:52.510022+0000","last_peered":"2024-09-18T11:51:52.510022+0000","last_clean":"2024-09-18T11:51:52.510022+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:52.510022+0000","last_undegraded":"2024-09-18T11:51:52.510022+0000","last_fullsized":"2024-09-18T11:51:52.510022+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9564,"log_dups_size":0,"ondisk_log_size":9564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27350262,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9372","reported_seq":9460,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.406953+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:53.406953+0000","last_peered":"2024-09-18T11:51:53.406953+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:53.406953+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9372,"log_dups_size":0,"ondisk_log_size":9372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27317494,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9370,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":440681868,"num_objects":53800,"num_object_clones":0,"num_object_copies":215200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153287,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6157,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":881213440,"data_stored":881213440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153314,"ondisk_log_size":153314,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513712,"kb_used_data":215848,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93858128,"statfs":{"total":96636764160,"available":96110723072,"internally_reserved":0,"allocated":221028352,"data_stored":220852886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,3,1,2,3,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.1579999999999999},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":9.9130000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":3.0659999999999998},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":32.067}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":1.7789999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":6.3680000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525628,"kb_used_data":215796,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93846212,"statfs":{"total":96636764160,"available":96098521088,"internally_reserved":0,"allocated":220975104,"data_stored":220799638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,0,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":4.806},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":4.8360000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":13.832000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.217}]}]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526648,"kb_used_data":216368,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93845192,"statfs":{"total":96636764160,"available":96097476608,"internally_reserved":0,"allocated":221560832,"data_stored":221385366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,6,8,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":6.6550000000000002},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":2.9550000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":23.151},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.6579999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":3.1880000000000002},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":16.262}]}]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521648,"kb_used_data":215524,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93850192,"statfs":{"total":96636764160,"available":96102596608,"internally_reserved":0,"allocated":220696576,"data_stored":220521110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":24.771000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":5.4880000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.7729999999999997},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":5.5229999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":16.911000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":5.548}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":219635712,"data_stored":219635712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":220569600,"data_stored":220569600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220319744,"data_stored":220319744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":220688384,"data_stored":220688384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:56.975 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:57.565 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:57.566 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:57.582 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-18T11:51:56.015583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":440681868,"num_objects":53800,"num_object_clones":0,"num_object_copies":215200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153287,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6157,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153314,"ondisk_log_size":153314,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2087636,"kb_used_data":863536,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375399724,"statfs":{"total":386547056640,"available":384409317376,"internally_reserved":0,"allocated":884260864,"data_stored":883559000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,5,4,9,11,16,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17235968,"num_objects":-2104,"num_object_clones":0,"num_object_copies":-8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2104,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002883"},"pg_stats":[{"pgid":"1.f","version":"35'9832","reported_seq":9930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:51.995937+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:51.995937+0000","last_peered":"2024-09-18T11:51:51.995937+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:51.995937+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28000256,"num_objects":3418,"num_object_clones":0,"num_object_copies":13672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9832,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9641","reported_seq":13931,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:52.722662+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:52.722662+0000","last_peered":"2024-09-18T11:51:52.722662+0000","last_clean":"2024-09-18T11:51:52.722662+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:52.722662+0000","last_undegraded":"2024-09-18T11:51:52.722662+0000","last_fullsized":"2024-09-18T11:51:52.722662+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9641,"log_dups_size":0,"ondisk_log_size":9641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27972854,"num_objects":3415,"num_object_clones":0,"num_object_copies":13660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9639,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9494","reported_seq":9609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.429385+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:52.429385+0000","last_peered":"2024-09-18T11:51:52.429385+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:52.429385+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9494,"log_dups_size":0,"ondisk_log_size":9494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27146899,"num_objects":3315,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9493,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1275}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9589","reported_seq":9681,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.385765+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:53.385765+0000","last_peered":"2024-09-18T11:51:53.385765+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:53.385765+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9589,"log_dups_size":0,"ondisk_log_size":9589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27333878,"num_objects":3337,"num_object_clones":0,"num_object_copies":13348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9587,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9546","reported_seq":9665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:51.931264+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:51.931264+0000","last_peered":"2024-09-18T11:51:51.931264+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:51.931264+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9546,"log_dups_size":0,"ondisk_log_size":9546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432182,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1297}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9521","reported_seq":9638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:51.892834+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:51.892834+0000","last_peered":"2024-09-18T11:51:51.892834+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:51.892834+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9521,"log_dups_size":0,"ondisk_log_size":9521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26746880,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9521,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9696","reported_seq":9801,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.276500+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:53.276500+0000","last_peered":"2024-09-18T11:51:53.276500+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:53.276500+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9696,"log_dups_size":0,"ondisk_log_size":9696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27194614,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9694,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1204}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9490","reported_seq":9597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.291153+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:53.291153+0000","last_peered":"2024-09-18T11:51:53.291153+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:53.291153+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9490,"log_dups_size":0,"ondisk_log_size":9490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27727094,"num_objects":3385,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9488,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9755","reported_seq":13130,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.164887+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:52.164887+0000","last_peered":"2024-09-18T11:51:52.164887+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:52.164887+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28019180,"num_objects":3421,"num_object_clones":0,"num_object_copies":13684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9752,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1659,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9282","reported_seq":9394,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.538475+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:52.538475+0000","last_peered":"2024-09-18T11:51:52.538475+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:52.538475+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27129078,"num_objects":3312,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9584","reported_seq":9703,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.086279+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:52.086279+0000","last_peered":"2024-09-18T11:51:52.086279+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:52.086279+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28249969,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9581,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9660","reported_seq":9748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.181144+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:52.181144+0000","last_peered":"2024-09-18T11:51:52.181144+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:52.181144+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9660,"log_dups_size":0,"ondisk_log_size":9660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27613819,"num_objects":3371,"num_object_clones":0,"num_object_copies":13484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9659,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1287}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9579","reported_seq":9675,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.485668+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:52.485668+0000","last_peered":"2024-09-18T11:51:52.485668+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:52.485668+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9579,"log_dups_size":0,"ondisk_log_size":9579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27753083,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9578,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1342}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9709","reported_seq":9832,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:52.028793+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:52.028793+0000","last_peered":"2024-09-18T11:51:52.028793+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:52.028793+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27694326,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1258}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9564","reported_seq":13851,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:52.510022+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:52.510022+0000","last_peered":"2024-09-18T11:51:52.510022+0000","last_clean":"2024-09-18T11:51:52.510022+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:52.510022+0000","last_undegraded":"2024-09-18T11:51:52.510022+0000","last_fullsized":"2024-09-18T11:51:52.510022+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9564,"log_dups_size":0,"ondisk_log_size":9564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27350262,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9372","reported_seq":9460,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.406953+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:53.406953+0000","last_peered":"2024-09-18T11:51:53.406953+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:53.406953+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9372,"log_dups_size":0,"ondisk_log_size":9372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27317494,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9370,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":440681868,"num_objects":53800,"num_object_clones":0,"num_object_copies":215200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153287,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6157,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":881213440,"data_stored":881213440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153314,"ondisk_log_size":153314,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513712,"kb_used_data":215848,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93858128,"statfs":{"total":96636764160,"available":96110723072,"internally_reserved":0,"allocated":221028352,"data_stored":220852886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,3,1,2,3,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.1579999999999999},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":9.9130000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2409999999999997,"5min":5.8380000000000001,"15min":5.601},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":26.899999999999999,"5min":26.899999999999999,"15min":26.899999999999999},"last":3.0659999999999998},{"interface":"front","average":{"1min":4.0289999999999999,"5min":4.6619999999999999,"15min":4.3540000000000001},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.279999999999999,"5min":25.573,"15min":25.573},"last":32.067}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":1.7789999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":6.3680000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525628,"kb_used_data":215796,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93846212,"statfs":{"total":96636764160,"available":96098521088,"internally_reserved":0,"allocated":220975104,"data_stored":220799638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,0,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":4.806},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":4.8360000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":13.832000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.217}]}]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526648,"kb_used_data":216368,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93845192,"statfs":{"total":96636764160,"available":96097476608,"internally_reserved":0,"allocated":221560832,"data_stored":221385366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,6,8,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":6.6550000000000002},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":2.9550000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":23.151},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":2.6579999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":3.1880000000000002},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":16.262}]}]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521648,"kb_used_data":215524,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93850192,"statfs":{"total":96636764160,"available":96102596608,"internally_reserved":0,"allocated":220696576,"data_stored":220521110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":24.771000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":5.4880000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.7729999999999997},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":5.5229999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":16.911000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":5.548}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":219635712,"data_stored":219635712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":220569600,"data_stored":220569600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220319744,"data_stored":220319744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":220688384,"data_stored":220688384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:51:57.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:51:57.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9832", 'reported_seq': 9930, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:51.995937+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:51.995937+0000', 'last_peered': '2024-09-18T11:51:51.995937+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:51.995937+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9832, 'log_dups_size': 0, 'ondisk_log_size': 9832, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28000256, 'num_objects': 3418, 'num_object_clones': 0, 'num_object_copies': 13672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9832, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:57.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:51:57.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9494", 'reported_seq': 9609, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:52.429385+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:52.429385+0000', 'last_peered': '2024-09-18T11:51:52.429385+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:52.429385+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9494, 'log_dups_size': 0, 'ondisk_log_size': 9494, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27146899, 'num_objects': 3315, 'num_object_clones': 0, 'num_object_copies': 13260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9493, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1275}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:57.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:51:57.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9589", 'reported_seq': 9681, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:53.385765+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:53.385765+0000', 'last_peered': '2024-09-18T11:51:53.385765+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:53.385765+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9589, 'log_dups_size': 0, 'ondisk_log_size': 9589, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27333878, 'num_objects': 3337, 'num_object_clones': 0, 'num_object_copies': 13348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9587, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1236}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:57.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:51:57.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9546", 'reported_seq': 9665, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:51.931264+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:51.931264+0000', 'last_peered': '2024-09-18T11:51:51.931264+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:51.931264+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9546, 'log_dups_size': 0, 'ondisk_log_size': 9546, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27432182, 'num_objects': 3349, 'num_object_clones': 0, 'num_object_copies': 13396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9544, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1297}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:57.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:51:57.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9521", 'reported_seq': 9638, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:51.892834+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:51.892834+0000', 'last_peered': '2024-09-18T11:51:51.892834+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:51.892834+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9521, 'log_dups_size': 0, 'ondisk_log_size': 9521, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26746880, 'num_objects': 3265, 'num_object_clones': 0, 'num_object_copies': 13060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9521, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1146}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:57.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:51:57.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9696", 'reported_seq': 9801, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:53.276500+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:53.276500+0000', 'last_peered': '2024-09-18T11:51:53.276500+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:53.276500+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9696, 'log_dups_size': 0, 'ondisk_log_size': 9696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27194614, 'num_objects': 3320, 'num_object_clones': 0, 'num_object_copies': 13280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9694, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1204}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:57.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:51:57.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9490", 'reported_seq': 9597, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:53.291153+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:53.291153+0000', 'last_peered': '2024-09-18T11:51:53.291153+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:53.291153+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9490, 'log_dups_size': 0, 'ondisk_log_size': 9490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27727094, 'num_objects': 3385, 'num_object_clones': 0, 'num_object_copies': 13540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9488, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1371}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:57.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:51:57.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9755", 'reported_seq': 13130, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:52.164887+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:52.164887+0000', 'last_peered': '2024-09-18T11:51:52.164887+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:52.164887+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9755, 'log_dups_size': 0, 'ondisk_log_size': 9755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28019180, 'num_objects': 3421, 'num_object_clones': 0, 'num_object_copies': 13684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 482, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9752, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1659, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2939}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:57.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:51:57.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9282", 'reported_seq': 9394, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:52.538475+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:52.538475+0000', 'last_peered': '2024-09-18T11:51:52.538475+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:52.538475+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9282, 'log_dups_size': 0, 'ondisk_log_size': 9282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27129078, 'num_objects': 3312, 'num_object_clones': 0, 'num_object_copies': 13248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9280, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1349}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:57.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:51:57.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9584", 'reported_seq': 9703, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:52.086279+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:52.086279+0000', 'last_peered': '2024-09-18T11:51:52.086279+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:52.086279+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9584, 'log_dups_size': 0, 'ondisk_log_size': 9584, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28249969, 'num_objects': 3449, 'num_object_clones': 0, 'num_object_copies': 13796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9581, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1415}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:57.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:51:57.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9660", 'reported_seq': 9748, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:52.181144+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:52.181144+0000', 'last_peered': '2024-09-18T11:51:52.181144+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:52.181144+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9660, 'log_dups_size': 0, 'ondisk_log_size': 9660, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27613819, 'num_objects': 3371, 'num_object_clones': 0, 'num_object_copies': 13484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9659, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:57.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:51:57.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9579", 'reported_seq': 9675, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:52.485668+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:52.485668+0000', 'last_peered': '2024-09-18T11:51:52.485668+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:52.485668+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9579, 'log_dups_size': 0, 'ondisk_log_size': 9579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27753083, 'num_objects': 3388, 'num_object_clones': 0, 'num_object_copies': 13552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9578, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1342}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:51:57.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:51:57.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9709", 'reported_seq': 9832, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:52.028793+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:52.028793+0000', 'last_peered': '2024-09-18T11:51:52.028793+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:52.028793+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9709, 'log_dups_size': 0, 'ondisk_log_size': 9709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27694326, 'num_objects': 3381, 'num_object_clones': 0, 'num_object_copies': 13524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9707, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1258}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:51:57.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:51:57.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9372", 'reported_seq': 9460, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:53.406953+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:53.406953+0000', 'last_peered': '2024-09-18T11:51:53.406953+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:53.406953+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9372, 'log_dups_size': 0, 'ondisk_log_size': 9372, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27317494, 'num_objects': 3335, 'num_object_clones': 0, 'num_object_copies': 13340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9370, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:51:57.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:51:58.263 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:58.263 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27259/211904 objects degraded (12.864%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":516,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52976,"data_bytes":433931660,"bytes_used":2120392704,"bytes_avail":384426663936,"bytes_total":386547056640,"degraded_objects":27259,"degraded_total":211904,"degraded_ratio":0.12863844004832378,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":243},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:51:58.282 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:51:58.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:51:58.872 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:51:58.872 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:51:58.898 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-09-18T11:51:58.016024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":433931660,"num_objects":52976,"num_object_clones":0,"num_object_copies":211904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154111,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154138,"ondisk_log_size":154138,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2070696,"kb_used_data":850740,"kb_used_omap":14,"kb_used_meta":1219889,"kb_avail":375416664,"statfs":{"total":386547056640,"available":384426663936,"internally_reserved":0,"allocated":871157760,"data_stored":870455896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249166363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,7,7,5,9,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23986176,"num_objects":-2928,"num_object_clones":0,"num_object_copies":-11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2928,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002812"},"pg_stats":[{"pgid":"1.f","version":"35'9891","reported_seq":9989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:56.942455+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:56.942455+0000","last_peered":"2024-09-18T11:51:56.942455+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:56.942455+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27516928,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9707","reported_seq":13997,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:57.863015+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:57.863015+0000","last_peered":"2024-09-18T11:51:57.863015+0000","last_clean":"2024-09-18T11:51:57.863015+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:57.863015+0000","last_undegraded":"2024-09-18T11:51:57.863015+0000","last_fullsized":"2024-09-18T11:51:57.863015+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432182,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9705,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9574","reported_seq":9689,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.527477+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:57.527477+0000","last_peered":"2024-09-18T11:51:57.527477+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:57.527477+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26491539,"num_objects":3235,"num_object_clones":0,"num_object_copies":12940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9573,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9589","reported_seq":9681,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.385765+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:53.385765+0000","last_peered":"2024-09-18T11:51:53.385765+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:53.385765+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9589,"log_dups_size":0,"ondisk_log_size":9589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27333878,"num_objects":3337,"num_object_clones":0,"num_object_copies":13348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9587,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9620","reported_seq":9739,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.121369+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:57.121369+0000","last_peered":"2024-09-18T11:51:57.121369+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:57.121369+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9620,"log_dups_size":0,"ondisk_log_size":9620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26825974,"num_objects":3275,"num_object_clones":0,"num_object_copies":13100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9618,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9588","reported_seq":9705,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.028259+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:57.028259+0000","last_peered":"2024-09-18T11:51:57.028259+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:57.028259+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9588,"log_dups_size":0,"ondisk_log_size":9588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26198016,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9588,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9696","reported_seq":9801,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.276500+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:53.276500+0000","last_peered":"2024-09-18T11:51:53.276500+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:53.276500+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9696,"log_dups_size":0,"ondisk_log_size":9696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27194614,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9694,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1204}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9490","reported_seq":9597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.291153+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:53.291153+0000","last_peered":"2024-09-18T11:51:53.291153+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:53.291153+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9490,"log_dups_size":0,"ondisk_log_size":9490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27727094,"num_objects":3385,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9488,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9833","reported_seq":13286,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.123478+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:57.123478+0000","last_peered":"2024-09-18T11:51:57.123478+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:57.123478+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9833,"log_dups_size":0,"ondisk_log_size":9833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27380204,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1699,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2901}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9344","reported_seq":9456,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.451987+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:57.451987+0000","last_peered":"2024-09-18T11:51:57.451987+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:57.451987+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9344,"log_dups_size":0,"ondisk_log_size":9344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26621174,"num_objects":3250,"num_object_clones":0,"num_object_copies":13000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9342,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9652","reported_seq":9771,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.240380+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:57.240380+0000","last_peered":"2024-09-18T11:51:57.240380+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:57.240380+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27692913,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9649,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9726","reported_seq":9817,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.025421+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:57.025421+0000","last_peered":"2024-09-18T11:51:57.025421+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:57.025421+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9726,"log_dups_size":0,"ondisk_log_size":9726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27073147,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9725,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1221}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9633","reported_seq":9729,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.050194+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:57.050194+0000","last_peered":"2024-09-18T11:51:57.050194+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:57.050194+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9633,"log_dups_size":0,"ondisk_log_size":9633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27310715,"num_objects":3334,"num_object_clones":0,"num_object_copies":13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9632,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1288}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9779","reported_seq":9902,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.207996+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:57.207996+0000","last_peered":"2024-09-18T11:51:57.207996+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:57.207996+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9779,"log_dups_size":0,"ondisk_log_size":9779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27120886,"num_objects":3311,"num_object_clones":0,"num_object_copies":13244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9777,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9644","reported_seq":13931,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:57.692088+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:57.692088+0000","last_peered":"2024-09-18T11:51:57.692088+0000","last_clean":"2024-09-18T11:51:57.692088+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:57.692088+0000","last_undegraded":"2024-09-18T11:51:57.692088+0000","last_fullsized":"2024-09-18T11:51:57.692088+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26694902,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9372","reported_seq":9460,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:53.406953+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:53.406953+0000","last_peered":"2024-09-18T11:51:53.406953+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:53.406953+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9372,"log_dups_size":0,"ondisk_log_size":9372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27317494,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9370,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":433931660,"num_objects":52976,"num_object_clones":0,"num_object_copies":211904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154111,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":867991552,"data_stored":867991552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154138,"ondisk_log_size":154138,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513596,"kb_used_data":211572,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93858244,"statfs":{"total":96636764160,"available":96110841856,"internally_reserved":0,"allocated":216649728,"data_stored":216474262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,5,3,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.7480000000000002},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":2.613}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.7599999999999998},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.2850000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":16.600999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.28000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525500,"kb_used_data":211508,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93846340,"statfs":{"total":96636764160,"available":96098652160,"internally_reserved":0,"allocated":216584192,"data_stored":216408726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":24.965},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":24.995000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":18.707000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":15.026}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":15.058},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.73199999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509952,"kb_used_data":212136,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93861888,"statfs":{"total":96636764160,"available":96114573312,"internally_reserved":0,"allocated":217227264,"data_stored":217051798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":32.371000000000002},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":3.863}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":3.7570000000000001},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":25.722999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":7.1219999999999999},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":3.79}]}]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521648,"kb_used_data":215524,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93850192,"statfs":{"total":96636764160,"available":96102596608,"internally_reserved":0,"allocated":220696576,"data_stored":220521110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:50:56 2024","interfaces":[{"interface":"back","average":{"1min":4.2960000000000003,"5min":4.6920000000000002,"15min":4.0179999999999998},"min":{"1min":0.20300000000000001,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":7.7519999999999998,"5min":26.189,"15min":26.189},"last":24.771000000000001},{"interface":"front","average":{"1min":4.4400000000000004,"5min":4.7670000000000003,"15min":4.0970000000000004},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":12.75,"5min":19.077999999999999,"15min":19.077999999999999},"last":5.4880000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.7729999999999997},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":5.5229999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":16.911000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":5.548}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":219635712,"data_stored":219635712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":216150016,"data_stored":216150016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215928832,"data_stored":215928832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216276992,"data_stored":216276992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:01.901 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:02.506 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:02.506 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:02.527 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-18T11:52:02.016972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":431949196,"num_objects":52734,"num_object_clones":0,"num_object_copies":210936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154353,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154380,"ondisk_log_size":154380,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2070572,"kb_used_data":846460,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375416788,"statfs":{"total":386547056640,"available":384426790912,"internally_reserved":0,"allocated":866775040,"data_stored":866073176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,7,9,4,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17350656,"num_objects":-2118,"num_object_clones":0,"num_object_copies":-8472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2118,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002828"},"pg_stats":[{"pgid":"1.f","version":"35'9891","reported_seq":9989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:56.942455+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:56.942455+0000","last_peered":"2024-09-18T11:51:56.942455+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:56.942455+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27516928,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9707","reported_seq":13997,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:57.863015+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:57.863015+0000","last_peered":"2024-09-18T11:51:57.863015+0000","last_clean":"2024-09-18T11:51:57.863015+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:57.863015+0000","last_undegraded":"2024-09-18T11:51:57.863015+0000","last_fullsized":"2024-09-18T11:51:57.863015+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432182,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9705,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9574","reported_seq":9689,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.527477+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:57.527477+0000","last_peered":"2024-09-18T11:51:57.527477+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:57.527477+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26491539,"num_objects":3235,"num_object_clones":0,"num_object_copies":12940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9573,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9646","reported_seq":9738,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:58.181772+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:58.181772+0000","last_peered":"2024-09-18T11:51:58.181772+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:58.181772+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9646,"log_dups_size":0,"ondisk_log_size":9646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26866934,"num_objects":3280,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9644,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9620","reported_seq":9739,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.121369+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:57.121369+0000","last_peered":"2024-09-18T11:51:57.121369+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:57.121369+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9620,"log_dups_size":0,"ondisk_log_size":9620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26825974,"num_objects":3275,"num_object_clones":0,"num_object_copies":13100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9618,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9588","reported_seq":9705,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.028259+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:57.028259+0000","last_peered":"2024-09-18T11:51:57.028259+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:57.028259+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9588,"log_dups_size":0,"ondisk_log_size":9588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26198016,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9588,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9765","reported_seq":9870,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:58.335217+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:58.335217+0000","last_peered":"2024-09-18T11:51:58.335217+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:58.335217+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26629366,"num_objects":3251,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1135}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9543","reported_seq":9649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:58.288396+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:58.288396+0000","last_peered":"2024-09-18T11:51:58.288396+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:58.288396+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9543,"log_dups_size":0,"ondisk_log_size":9543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27292918,"num_objects":3332,"num_object_clones":0,"num_object_copies":13328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9541,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9833","reported_seq":13286,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.123478+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:57.123478+0000","last_peered":"2024-09-18T11:51:57.123478+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:57.123478+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9833,"log_dups_size":0,"ondisk_log_size":9833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27380204,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1699,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2901}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9344","reported_seq":9456,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.451987+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:57.451987+0000","last_peered":"2024-09-18T11:51:57.451987+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:57.451987+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9344,"log_dups_size":0,"ondisk_log_size":9344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26621174,"num_objects":3250,"num_object_clones":0,"num_object_copies":13000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9342,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9652","reported_seq":9771,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.240380+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:57.240380+0000","last_peered":"2024-09-18T11:51:57.240380+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:57.240380+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27692913,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9649,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9726","reported_seq":9817,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.025421+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:57.025421+0000","last_peered":"2024-09-18T11:51:57.025421+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:57.025421+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9726,"log_dups_size":0,"ondisk_log_size":9726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27073147,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9725,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1221}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9633","reported_seq":9729,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.050194+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:57.050194+0000","last_peered":"2024-09-18T11:51:57.050194+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:57.050194+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9633,"log_dups_size":0,"ondisk_log_size":9633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27310715,"num_objects":3334,"num_object_clones":0,"num_object_copies":13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9632,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1288}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9779","reported_seq":9902,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.207996+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:57.207996+0000","last_peered":"2024-09-18T11:51:57.207996+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:57.207996+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9779,"log_dups_size":0,"ondisk_log_size":9779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27120886,"num_objects":3311,"num_object_clones":0,"num_object_copies":13244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9777,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9644","reported_seq":13931,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:57.692088+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:57.692088+0000","last_peered":"2024-09-18T11:51:57.692088+0000","last_clean":"2024-09-18T11:51:57.692088+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:57.692088+0000","last_undegraded":"2024-09-18T11:51:57.692088+0000","last_fullsized":"2024-09-18T11:51:57.692088+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26694902,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9435","reported_seq":9523,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:58.373516+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:58.373516+0000","last_peered":"2024-09-18T11:51:58.373516+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:58.373516+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9435,"log_dups_size":0,"ondisk_log_size":9435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26801398,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":431949196,"num_objects":52734,"num_object_clones":0,"num_object_copies":210936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154353,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":863629312,"data_stored":863629312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154380,"ondisk_log_size":154380,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513596,"kb_used_data":211572,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93858244,"statfs":{"total":96636764160,"available":96110841856,"internally_reserved":0,"allocated":216649728,"data_stored":216474262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,5,3,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.7480000000000002},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":2.613}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.7599999999999998},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.2850000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":16.600999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.28000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525500,"kb_used_data":211508,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93846340,"statfs":{"total":96636764160,"available":96098652160,"internally_reserved":0,"allocated":216584192,"data_stored":216408726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":24.965},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":24.995000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":18.707000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":15.026}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":15.058},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.73199999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509952,"kb_used_data":212136,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93861888,"statfs":{"total":96636764160,"available":96114573312,"internally_reserved":0,"allocated":217227264,"data_stored":217051798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":32.371000000000002},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":3.863}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":3.7570000000000001},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":25.722999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":7.1219999999999999},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":3.79}]}]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521524,"kb_used_data":211244,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93850316,"statfs":{"total":96636764160,"available":96102723584,"internally_reserved":0,"allocated":216313856,"data_stored":216138390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":3.0899999999999999},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.6539999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":13.534000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":15.731}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":15.692}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":215273472,"data_stored":215273472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":216150016,"data_stored":216150016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215928832,"data_stored":215928832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216276992,"data_stored":216276992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:02.529 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:03.046 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:03.046 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:03.072 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-18T11:52:02.016972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":431949196,"num_objects":52734,"num_object_clones":0,"num_object_copies":210936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154353,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154380,"ondisk_log_size":154380,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2070572,"kb_used_data":846460,"kb_used_omap":14,"kb_used_meta":1224049,"kb_avail":375416788,"statfs":{"total":386547056640,"available":384426790912,"internally_reserved":0,"allocated":866775040,"data_stored":866073176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253426203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,7,9,4,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17350656,"num_objects":-2118,"num_object_clones":0,"num_object_copies":-8472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2118,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002828"},"pg_stats":[{"pgid":"1.f","version":"35'9891","reported_seq":9989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:56.942455+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:51:56.942455+0000","last_peered":"2024-09-18T11:51:56.942455+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:51:56.942455+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27516928,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9707","reported_seq":13997,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:57.863015+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:51:57.863015+0000","last_peered":"2024-09-18T11:51:57.863015+0000","last_clean":"2024-09-18T11:51:57.863015+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:51:57.863015+0000","last_undegraded":"2024-09-18T11:51:57.863015+0000","last_fullsized":"2024-09-18T11:51:57.863015+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432182,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9705,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9574","reported_seq":9689,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.527477+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:51:57.527477+0000","last_peered":"2024-09-18T11:51:57.527477+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:51:57.527477+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26491539,"num_objects":3235,"num_object_clones":0,"num_object_copies":12940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9573,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9646","reported_seq":9738,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:58.181772+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:51:58.181772+0000","last_peered":"2024-09-18T11:51:58.181772+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:51:58.181772+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9646,"log_dups_size":0,"ondisk_log_size":9646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26866934,"num_objects":3280,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9644,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9620","reported_seq":9739,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.121369+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:51:57.121369+0000","last_peered":"2024-09-18T11:51:57.121369+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:51:57.121369+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9620,"log_dups_size":0,"ondisk_log_size":9620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26825974,"num_objects":3275,"num_object_clones":0,"num_object_copies":13100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9618,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9588","reported_seq":9705,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.028259+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:51:57.028259+0000","last_peered":"2024-09-18T11:51:57.028259+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:51:57.028259+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9588,"log_dups_size":0,"ondisk_log_size":9588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26198016,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9588,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9765","reported_seq":9870,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:58.335217+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:51:58.335217+0000","last_peered":"2024-09-18T11:51:58.335217+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:51:58.335217+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26629366,"num_objects":3251,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1135}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9543","reported_seq":9649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:58.288396+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:51:58.288396+0000","last_peered":"2024-09-18T11:51:58.288396+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:51:58.288396+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9543,"log_dups_size":0,"ondisk_log_size":9543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27292918,"num_objects":3332,"num_object_clones":0,"num_object_copies":13328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9541,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9833","reported_seq":13286,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.123478+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:51:57.123478+0000","last_peered":"2024-09-18T11:51:57.123478+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:51:57.123478+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9833,"log_dups_size":0,"ondisk_log_size":9833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27380204,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1699,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2901}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9344","reported_seq":9456,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.451987+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:51:57.451987+0000","last_peered":"2024-09-18T11:51:57.451987+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:51:57.451987+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9344,"log_dups_size":0,"ondisk_log_size":9344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26621174,"num_objects":3250,"num_object_clones":0,"num_object_copies":13000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9342,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9652","reported_seq":9771,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.240380+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:51:57.240380+0000","last_peered":"2024-09-18T11:51:57.240380+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:51:57.240380+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27692913,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9649,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9726","reported_seq":9817,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.025421+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:51:57.025421+0000","last_peered":"2024-09-18T11:51:57.025421+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:51:57.025421+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9726,"log_dups_size":0,"ondisk_log_size":9726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27073147,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9725,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1221}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9633","reported_seq":9729,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.050194+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:51:57.050194+0000","last_peered":"2024-09-18T11:51:57.050194+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:51:57.050194+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9633,"log_dups_size":0,"ondisk_log_size":9633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27310715,"num_objects":3334,"num_object_clones":0,"num_object_copies":13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9632,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1288}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9779","reported_seq":9902,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:57.207996+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:51:57.207996+0000","last_peered":"2024-09-18T11:51:57.207996+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:51:57.207996+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9779,"log_dups_size":0,"ondisk_log_size":9779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27120886,"num_objects":3311,"num_object_clones":0,"num_object_copies":13244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9777,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9644","reported_seq":13931,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:51:57.692088+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:51:57.692088+0000","last_peered":"2024-09-18T11:51:57.692088+0000","last_clean":"2024-09-18T11:51:57.692088+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:51:57.692088+0000","last_undegraded":"2024-09-18T11:51:57.692088+0000","last_fullsized":"2024-09-18T11:51:57.692088+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26694902,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9435","reported_seq":9523,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:51:58.373516+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:51:58.373516+0000","last_peered":"2024-09-18T11:51:58.373516+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:51:58.373516+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9435,"log_dups_size":0,"ondisk_log_size":9435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26801398,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":431949196,"num_objects":52734,"num_object_clones":0,"num_object_copies":210936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154353,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":863629312,"data_stored":863629312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154380,"ondisk_log_size":154380,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513596,"kb_used_data":211572,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93858244,"statfs":{"total":96636764160,"available":96110841856,"internally_reserved":0,"allocated":216649728,"data_stored":216474262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,5,3,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.7480000000000002},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":2.613}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.7599999999999998},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.2850000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":16.600999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.28000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525500,"kb_used_data":211508,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93846340,"statfs":{"total":96636764160,"available":96098652160,"internally_reserved":0,"allocated":216584192,"data_stored":216408726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":24.965},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":24.995000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:50:55 2024","interfaces":[{"interface":"back","average":{"1min":8.2799999999999994,"5min":6.4610000000000003,"15min":6.1070000000000002},"min":{"1min":0.93700000000000006,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":26.530000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":18.707000000000001},{"interface":"front","average":{"1min":4.0359999999999996,"5min":4.5490000000000004,"15min":3.9529999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":11.954000000000001,"5min":19.789999999999999,"15min":19.789999999999999},"last":15.026}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":15.058},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.73199999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509952,"kb_used_data":212136,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93861888,"statfs":{"total":96636764160,"available":96114573312,"internally_reserved":0,"allocated":217227264,"data_stored":217051798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":32.371000000000002},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":3.863}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":3.7570000000000001},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":25.722999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":7.1219999999999999},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":3.79}]}]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521524,"kb_used_data":211244,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93850316,"statfs":{"total":96636764160,"available":96102723584,"internally_reserved":0,"allocated":216313856,"data_stored":216138390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":3.0899999999999999},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.6539999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":13.534000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":15.731}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":15.692}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":215273472,"data_stored":215273472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":216150016,"data_stored":216150016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215928832,"data_stored":215928832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216276992,"data_stored":216276992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:03.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:03.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9891", 'reported_seq': 9989, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:56.942455+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:51:56.942455+0000', 'last_peered': '2024-09-18T11:51:56.942455+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:51:56.942455+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9891, 'log_dups_size': 0, 'ondisk_log_size': 9891, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27516928, 'num_objects': 3359, 'num_object_clones': 0, 'num_object_copies': 13436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9891, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1256}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:03.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:03.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9574", 'reported_seq': 9689, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:57.527477+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:51:57.527477+0000', 'last_peered': '2024-09-18T11:51:57.527477+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:51:57.527477+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9574, 'log_dups_size': 0, 'ondisk_log_size': 9574, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26491539, 'num_objects': 3235, 'num_object_clones': 0, 'num_object_copies': 12940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3235, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9573, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1195}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:03.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:03.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9646", 'reported_seq': 9738, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:58.181772+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:51:58.181772+0000', 'last_peered': '2024-09-18T11:51:58.181772+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:51:58.181772+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9646, 'log_dups_size': 0, 'ondisk_log_size': 9646, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26866934, 'num_objects': 3280, 'num_object_clones': 0, 'num_object_copies': 13120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9644, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1179}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:03.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:03.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9620", 'reported_seq': 9739, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:57.121369+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:51:57.121369+0000', 'last_peered': '2024-09-18T11:51:57.121369+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:51:57.121369+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9620, 'log_dups_size': 0, 'ondisk_log_size': 9620, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26825974, 'num_objects': 3275, 'num_object_clones': 0, 'num_object_copies': 13100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9618, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1223}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:03.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:03.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9588", 'reported_seq': 9705, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:57.028259+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:51:57.028259+0000', 'last_peered': '2024-09-18T11:51:57.028259+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:51:57.028259+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9588, 'log_dups_size': 0, 'ondisk_log_size': 9588, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26198016, 'num_objects': 3198, 'num_object_clones': 0, 'num_object_copies': 12792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9588, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1079}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:03.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:03.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9765", 'reported_seq': 9870, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:58.335217+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:51:58.335217+0000', 'last_peered': '2024-09-18T11:51:58.335217+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:51:58.335217+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9765, 'log_dups_size': 0, 'ondisk_log_size': 9765, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26629366, 'num_objects': 3251, 'num_object_clones': 0, 'num_object_copies': 13004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9763, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1135}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:03.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:03.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9543", 'reported_seq': 9649, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:58.288396+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:51:58.288396+0000', 'last_peered': '2024-09-18T11:51:58.288396+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:51:58.288396+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9543, 'log_dups_size': 0, 'ondisk_log_size': 9543, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27292918, 'num_objects': 3332, 'num_object_clones': 0, 'num_object_copies': 13328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9541, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:03.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:52:03.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9833", 'reported_seq': 13286, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:57.123478+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:51:57.123478+0000', 'last_peered': '2024-09-18T11:51:57.123478+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:51:57.123478+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9833, 'log_dups_size': 0, 'ondisk_log_size': 9833, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27380204, 'num_objects': 3343, 'num_object_clones': 0, 'num_object_copies': 13372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 442, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9830, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1699, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2901}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9344", 'reported_seq': 9456, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:57.451987+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:51:57.451987+0000', 'last_peered': '2024-09-18T11:51:57.451987+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:51:57.451987+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9344, 'log_dups_size': 0, 'ondisk_log_size': 9344, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26621174, 'num_objects': 3250, 'num_object_clones': 0, 'num_object_copies': 13000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9342, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1287}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9652", 'reported_seq': 9771, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:57.240380+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:51:57.240380+0000', 'last_peered': '2024-09-18T11:51:57.240380+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:51:57.240380+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9652, 'log_dups_size': 0, 'ondisk_log_size': 9652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27692913, 'num_objects': 3381, 'num_object_clones': 0, 'num_object_copies': 13524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9649, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1347}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9726", 'reported_seq': 9817, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:57.025421+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:51:57.025421+0000', 'last_peered': '2024-09-18T11:51:57.025421+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:51:57.025421+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9726, 'log_dups_size': 0, 'ondisk_log_size': 9726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27073147, 'num_objects': 3305, 'num_object_clones': 0, 'num_object_copies': 13220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9725, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1221}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9633", 'reported_seq': 9729, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:57.050194+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:51:57.050194+0000', 'last_peered': '2024-09-18T11:51:57.050194+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:51:57.050194+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9633, 'log_dups_size': 0, 'ondisk_log_size': 9633, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27310715, 'num_objects': 3334, 'num_object_clones': 0, 'num_object_copies': 13336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9632, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1288}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9779", 'reported_seq': 9902, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:57.207996+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:51:57.207996+0000', 'last_peered': '2024-09-18T11:51:57.207996+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:51:57.207996+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9779, 'log_dups_size': 0, 'ondisk_log_size': 9779, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27120886, 'num_objects': 3311, 'num_object_clones': 0, 'num_object_copies': 13244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9777, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:03.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:03.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9435", 'reported_seq': 9523, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:51:58.373516+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:51:58.373516+0000', 'last_peered': '2024-09-18T11:51:58.373516+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:51:58.373516+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9435, 'log_dups_size': 0, 'ondisk_log_size': 9435, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26801398, 'num_objects': 3272, 'num_object_clones': 0, 'num_object_copies': 13088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9433, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1270}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:03.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:03.792 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:03.792 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27259/210936 objects degraded (12.923%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":521,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52734,"data_bytes":431949196,"bytes_used":2120265728,"bytes_avail":384426790912,"bytes_total":386547056640,"degraded_objects":27259,"degraded_total":210936,"degraded_ratio":0.12922877081199985,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":176},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:03.820 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:03.821 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:04.455 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:04.455 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:04.478 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-18T11:52:04.017562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":423183756,"num_objects":51664,"num_object_clones":0,"num_object_copies":206656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155423,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6259,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155450,"ondisk_log_size":155450,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2057316,"kb_used_data":829016,"kb_used_omap":14,"kb_used_meta":1228209,"kb_avail":375430044,"statfs":{"total":386547056640,"available":384440365056,"internally_reserved":0,"allocated":848912384,"data_stored":848210520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257686043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,4,5,12,12,15,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26116096,"num_objects":-3188,"num_object_clones":0,"num_object_copies":-12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002955"},"pg_stats":[{"pgid":"1.f","version":"35'9960","reported_seq":10059,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:02.283651+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:02.283651+0000","last_peered":"2024-09-18T11:52:02.283651+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:02.283651+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9960,"log_dups_size":0,"ondisk_log_size":9960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26951680,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9960,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9782","reported_seq":14071,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:02.890922+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:02.890922+0000","last_peered":"2024-09-18T11:52:02.890922+0000","last_clean":"2024-09-18T11:52:02.890922+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:02.890922+0000","last_undegraded":"2024-09-18T11:52:02.890922+0000","last_fullsized":"2024-09-18T11:52:02.890922+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26817782,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9634","reported_seq":9749,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:02.508423+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:02.508423+0000","last_peered":"2024-09-18T11:52:02.508423+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:02.508423+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9634,"log_dups_size":0,"ondisk_log_size":9634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26000019,"num_objects":3175,"num_object_clones":0,"num_object_copies":12700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9633,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1135}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9710","reported_seq":9802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.438651+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:03.438651+0000","last_peered":"2024-09-18T11:52:03.438651+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:03.438651+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342646,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9708,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1115}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9682","reported_seq":9801,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:02.203771+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:02.203771+0000","last_peered":"2024-09-18T11:52:02.203771+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:02.203771+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9682,"log_dups_size":0,"ondisk_log_size":9682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26318070,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9680,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9643","reported_seq":9760,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:02.349279+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:02.349279+0000","last_peered":"2024-09-18T11:52:02.349279+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:02.349279+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9643,"log_dups_size":0,"ondisk_log_size":9643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25747456,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9643,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9832","reported_seq":9937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.345696+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:03.345696+0000","last_peered":"2024-09-18T11:52:03.345696+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:03.345696+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26080502,"num_objects":3184,"num_object_clones":0,"num_object_copies":12736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1068}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9607","reported_seq":9714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.457598+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:03.457598+0000","last_peered":"2024-09-18T11:52:03.457598+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:03.457598+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9607,"log_dups_size":0,"ondisk_log_size":9607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26768630,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9605,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9922","reported_seq":13501,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:02.336216+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:02.336216+0000","last_peered":"2024-09-18T11:52:02.336216+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:02.336216+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26651116,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2874}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9415","reported_seq":9527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:02.342150+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:02.342150+0000","last_peered":"2024-09-18T11:52:02.342150+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:02.342150+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9415,"log_dups_size":0,"ondisk_log_size":9415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26039542,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9413,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1216}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9704","reported_seq":9823,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:02.217567+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:02.217567+0000","last_peered":"2024-09-18T11:52:02.217567+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:02.217567+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9704,"log_dups_size":0,"ondisk_log_size":9704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27266929,"num_objects":3329,"num_object_clones":0,"num_object_copies":13316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9701,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9801","reported_seq":9892,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:02.324615+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:02.324615+0000","last_peered":"2024-09-18T11:52:02.324615+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:02.324615+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9801,"log_dups_size":0,"ondisk_log_size":9801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26458747,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9800,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1146}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9701","reported_seq":9797,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:02.517608+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:02.517608+0000","last_peered":"2024-09-18T11:52:02.517608+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:02.517608+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26753659,"num_objects":3266,"num_object_clones":0,"num_object_copies":13064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9700,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9848","reported_seq":9971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:02.199438+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:02.199438+0000","last_peered":"2024-09-18T11:52:02.199438+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:02.199438+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26555638,"num_objects":3242,"num_object_clones":0,"num_object_copies":12968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1119}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9709","reported_seq":13996,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:02.724390+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:02.724390+0000","last_peered":"2024-09-18T11:52:02.724390+0000","last_clean":"2024-09-18T11:52:02.724390+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:02.724390+0000","last_undegraded":"2024-09-18T11:52:02.724390+0000","last_fullsized":"2024-09-18T11:52:02.724390+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26162422,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9500","reported_seq":9588,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.345038+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:03.345038+0000","last_peered":"2024-09-18T11:52:03.345038+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:03.345038+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26268918,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9498,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":423183756,"num_objects":51664,"num_object_clones":0,"num_object_copies":206656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155423,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6259,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":846356480,"data_stored":846356480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155450,"ondisk_log_size":155450,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513624,"kb_used_data":207440,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93858216,"statfs":{"total":96636764160,"available":96110813184,"internally_reserved":0,"allocated":212418560,"data_stored":212243094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,2,2,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":3.4620000000000002},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":3.4910000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":3.5190000000000001},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":8.3049999999999997},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.613}]}]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508824,"kb_used_data":207268,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93863016,"statfs":{"total":96636764160,"available":96115728384,"internally_reserved":0,"allocated":212242432,"data_stored":212066966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":6.3499999999999996},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":6.3869999999999996}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":10.656000000000001},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":26.545000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":1.391},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":1.347}]}]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513324,"kb_used_data":207204,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93858516,"statfs":{"total":96636764160,"available":96111120384,"internally_reserved":0,"allocated":212176896,"data_stored":212001430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.5880000000000001},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.573}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.69899999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521544,"kb_used_data":207104,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93850296,"statfs":{"total":96636764160,"available":96102703104,"internally_reserved":0,"allocated":212074496,"data_stored":211899030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,1,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":20.116},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.802},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.751}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":18.015000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":8.4290000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211001344,"data_stored":211001344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211836928,"data_stored":211836928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211644416,"data_stored":211644416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":211873792,"data_stored":211873792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:07.481 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:08.049 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:08.050 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:08.069 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-18T11:52:08.019279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":417736076,"num_objects":50999,"num_object_clones":0,"num_object_copies":203996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156088,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6314,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156115,"ondisk_log_size":156115,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2634128,"kb_used_data":820776,"kb_used_omap":14,"kb_used_meta":1813297,"kb_avail":374853232,"statfs":{"total":386547056640,"available":383849709568,"internally_reserved":0,"allocated":840474624,"data_stored":839772760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1856816155},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,8,7,11,11,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22945792,"num_objects":-2801,"num_object_clones":0,"num_object_copies":-11204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003696"},"pg_stats":[{"pgid":"1.f","version":"35'10023","reported_seq":10122,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.285241+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:07.285241+0000","last_peered":"2024-09-18T11:52:07.285241+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:07.285241+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26435584,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10023,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9782","reported_seq":14071,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:02.890922+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:02.890922+0000","last_peered":"2024-09-18T11:52:02.890922+0000","last_clean":"2024-09-18T11:52:02.890922+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:02.890922+0000","last_undegraded":"2024-09-18T11:52:02.890922+0000","last_fullsized":"2024-09-18T11:52:02.890922+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26817782,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9702","reported_seq":9817,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.500119+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:07.500119+0000","last_peered":"2024-09-18T11:52:07.500119+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:07.500119+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25442963,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9701,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9710","reported_seq":9802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.438651+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:03.438651+0000","last_peered":"2024-09-18T11:52:03.438651+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:03.438651+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342646,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9708,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1115}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9755","reported_seq":9874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.229349+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:07.229349+0000","last_peered":"2024-09-18T11:52:07.229349+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:07.229349+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25720054,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9709","reported_seq":9825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.368844+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:07.368844+0000","last_peered":"2024-09-18T11:52:07.368844+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:07.368844+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25206784,"num_objects":3077,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9832","reported_seq":9937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.345696+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:03.345696+0000","last_peered":"2024-09-18T11:52:03.345696+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:03.345696+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26080502,"num_objects":3184,"num_object_clones":0,"num_object_copies":12736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1068}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9607","reported_seq":9714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.457598+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:03.457598+0000","last_peered":"2024-09-18T11:52:03.457598+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:03.457598+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9607,"log_dups_size":0,"ondisk_log_size":9607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26768630,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9605,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9982","reported_seq":13671,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.377556+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:07.377556+0000","last_peered":"2024-09-18T11:52:07.377556+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:07.377556+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9982,"log_dups_size":0,"ondisk_log_size":9982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26159596,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1816,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9485","reported_seq":9595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.669458+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:07.669458+0000","last_peered":"2024-09-18T11:52:07.669458+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:07.669458+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9485,"log_dups_size":0,"ondisk_log_size":9485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25466102,"num_objects":3109,"num_object_clones":0,"num_object_copies":12436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9769","reported_seq":9885,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.406521+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:07.406521+0000","last_peered":"2024-09-18T11:52:07.406521+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:07.406521+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26734449,"num_objects":3264,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9885","reported_seq":9976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.286380+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:07.286380+0000","last_peered":"2024-09-18T11:52:07.286380+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:07.286380+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25770619,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9884,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1062}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9762","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.698380+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:07.698380+0000","last_peered":"2024-09-18T11:52:07.698380+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:07.698380+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26253947,"num_objects":3205,"num_object_clones":0,"num_object_copies":12820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9761,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1159}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9903","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.298610+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:07.298610+0000","last_peered":"2024-09-18T11:52:07.298610+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:07.298610+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9903,"log_dups_size":0,"ondisk_log_size":9903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26105078,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1064}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9709","reported_seq":13996,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:02.724390+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:02.724390+0000","last_peered":"2024-09-18T11:52:02.724390+0000","last_clean":"2024-09-18T11:52:02.724390+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:02.724390+0000","last_undegraded":"2024-09-18T11:52:02.724390+0000","last_fullsized":"2024-09-18T11:52:02.724390+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26162422,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9500","reported_seq":9588,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.345038+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:03.345038+0000","last_peered":"2024-09-18T11:52:03.345038+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:03.345038+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26268918,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9498,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":417736076,"num_objects":50999,"num_object_clones":0,"num_object_copies":203996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156088,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6314,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":837783552,"data_stored":837783552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156115,"ondisk_log_size":156115,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802100,"kb_used_data":203364,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93569740,"statfs":{"total":96636764160,"available":95815413760,"internally_reserved":0,"allocated":208244736,"data_stored":208069270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,3,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.359},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":2.387}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.178000000000001},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":1.968}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.9409999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":4.0369999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797160,"kb_used_data":203104,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93574680,"statfs":{"total":96636764160,"available":95820472320,"internally_reserved":0,"allocated":207978496,"data_stored":207803030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":1.333}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.3439999999999999},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.286}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.5720000000000001},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.67900000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513324,"kb_used_data":207204,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93858516,"statfs":{"total":96636764160,"available":96111120384,"internally_reserved":0,"allocated":212176896,"data_stored":212001430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.5880000000000001},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.573}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.69899999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521544,"kb_used_data":207104,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93850296,"statfs":{"total":96636764160,"available":96102703104,"internally_reserved":0,"allocated":212074496,"data_stored":211899030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,1,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":20.116},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.802},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.751}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":18.015000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":8.4290000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211001344,"data_stored":211001344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211836928,"data_stored":211836928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207319040,"data_stored":207319040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207626240,"data_stored":207626240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:08.071 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:08.686 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:08.687 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:08.714 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-18T11:52:08.019279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":417736076,"num_objects":50999,"num_object_clones":0,"num_object_copies":203996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156088,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6314,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156115,"ondisk_log_size":156115,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2634128,"kb_used_data":820776,"kb_used_omap":14,"kb_used_meta":1813297,"kb_avail":374853232,"statfs":{"total":386547056640,"available":383849709568,"internally_reserved":0,"allocated":840474624,"data_stored":839772760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1856816155},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,8,7,11,11,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22945792,"num_objects":-2801,"num_object_clones":0,"num_object_copies":-11204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003696"},"pg_stats":[{"pgid":"1.f","version":"35'10023","reported_seq":10122,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.285241+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:07.285241+0000","last_peered":"2024-09-18T11:52:07.285241+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:07.285241+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26435584,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10023,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9782","reported_seq":14071,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:02.890922+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:02.890922+0000","last_peered":"2024-09-18T11:52:02.890922+0000","last_clean":"2024-09-18T11:52:02.890922+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:02.890922+0000","last_undegraded":"2024-09-18T11:52:02.890922+0000","last_fullsized":"2024-09-18T11:52:02.890922+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26817782,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9702","reported_seq":9817,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.500119+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:07.500119+0000","last_peered":"2024-09-18T11:52:07.500119+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:07.500119+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25442963,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9701,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9710","reported_seq":9802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.438651+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:03.438651+0000","last_peered":"2024-09-18T11:52:03.438651+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:03.438651+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342646,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9708,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1115}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9755","reported_seq":9874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.229349+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:07.229349+0000","last_peered":"2024-09-18T11:52:07.229349+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:07.229349+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25720054,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9709","reported_seq":9825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.368844+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:07.368844+0000","last_peered":"2024-09-18T11:52:07.368844+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:07.368844+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25206784,"num_objects":3077,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9832","reported_seq":9937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.345696+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:03.345696+0000","last_peered":"2024-09-18T11:52:03.345696+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:03.345696+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26080502,"num_objects":3184,"num_object_clones":0,"num_object_copies":12736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1068}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9607","reported_seq":9714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.457598+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:03.457598+0000","last_peered":"2024-09-18T11:52:03.457598+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:03.457598+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9607,"log_dups_size":0,"ondisk_log_size":9607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26768630,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9605,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9982","reported_seq":13671,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.377556+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:07.377556+0000","last_peered":"2024-09-18T11:52:07.377556+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:07.377556+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9982,"log_dups_size":0,"ondisk_log_size":9982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26159596,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1816,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9485","reported_seq":9595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.669458+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:07.669458+0000","last_peered":"2024-09-18T11:52:07.669458+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:07.669458+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9485,"log_dups_size":0,"ondisk_log_size":9485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25466102,"num_objects":3109,"num_object_clones":0,"num_object_copies":12436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9769","reported_seq":9885,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.406521+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:07.406521+0000","last_peered":"2024-09-18T11:52:07.406521+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:07.406521+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26734449,"num_objects":3264,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9885","reported_seq":9976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.286380+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:07.286380+0000","last_peered":"2024-09-18T11:52:07.286380+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:07.286380+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25770619,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9884,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1062}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9762","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.698380+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:07.698380+0000","last_peered":"2024-09-18T11:52:07.698380+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:07.698380+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26253947,"num_objects":3205,"num_object_clones":0,"num_object_copies":12820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9761,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1159}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9903","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.298610+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:07.298610+0000","last_peered":"2024-09-18T11:52:07.298610+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:07.298610+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9903,"log_dups_size":0,"ondisk_log_size":9903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26105078,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1064}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9709","reported_seq":13996,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:02.724390+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:02.724390+0000","last_peered":"2024-09-18T11:52:02.724390+0000","last_clean":"2024-09-18T11:52:02.724390+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:02.724390+0000","last_undegraded":"2024-09-18T11:52:02.724390+0000","last_fullsized":"2024-09-18T11:52:02.724390+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26162422,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9500","reported_seq":9588,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.345038+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:03.345038+0000","last_peered":"2024-09-18T11:52:03.345038+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:03.345038+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26268918,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9498,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":417736076,"num_objects":50999,"num_object_clones":0,"num_object_copies":203996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156088,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6314,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":837783552,"data_stored":837783552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156115,"ondisk_log_size":156115,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802100,"kb_used_data":203364,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93569740,"statfs":{"total":96636764160,"available":95815413760,"internally_reserved":0,"allocated":208244736,"data_stored":208069270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,3,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.359},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":2.387}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.178000000000001},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":1.968}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.9409999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":4.0369999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797160,"kb_used_data":203104,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93574680,"statfs":{"total":96636764160,"available":95820472320,"internally_reserved":0,"allocated":207978496,"data_stored":207803030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":1.333}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.3439999999999999},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.286}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.5720000000000001},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.67900000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513324,"kb_used_data":207204,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93858516,"statfs":{"total":96636764160,"available":96111120384,"internally_reserved":0,"allocated":212176896,"data_stored":212001430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.5880000000000001},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.573}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.69899999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521544,"kb_used_data":207104,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93850296,"statfs":{"total":96636764160,"available":96102703104,"internally_reserved":0,"allocated":212074496,"data_stored":211899030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,1,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":20.116},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.802},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.751}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":18.015000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":8.4290000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211001344,"data_stored":211001344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211836928,"data_stored":211836928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207319040,"data_stored":207319040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207626240,"data_stored":207626240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:08.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:08.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10023", 'reported_seq': 10122, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:07.285241+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:07.285241+0000', 'last_peered': '2024-09-18T11:52:07.285241+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:07.285241+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26435584, 'num_objects': 3227, 'num_object_clones': 0, 'num_object_copies': 12908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10023, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1124}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:08.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:08.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9702", 'reported_seq': 9817, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:07.500119+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:07.500119+0000', 'last_peered': '2024-09-18T11:52:07.500119+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:07.500119+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9702, 'log_dups_size': 0, 'ondisk_log_size': 9702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25442963, 'num_objects': 3107, 'num_object_clones': 0, 'num_object_copies': 12428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3107, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9701, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1067}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:08.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:08.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9710", 'reported_seq': 9802, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:03.438651+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:03.438651+0000', 'last_peered': '2024-09-18T11:52:03.438651+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:03.438651+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9710, 'log_dups_size': 0, 'ondisk_log_size': 9710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26342646, 'num_objects': 3216, 'num_object_clones': 0, 'num_object_copies': 12864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9708, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1115}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:08.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:08.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9755", 'reported_seq': 9874, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:07.229349+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:07.229349+0000', 'last_peered': '2024-09-18T11:52:07.229349+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:07.229349+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9755, 'log_dups_size': 0, 'ondisk_log_size': 9755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25720054, 'num_objects': 3140, 'num_object_clones': 0, 'num_object_copies': 12560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9753, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1088}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:08.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:08.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9709", 'reported_seq': 9825, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:07.368844+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:07.368844+0000', 'last_peered': '2024-09-18T11:52:07.368844+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:07.368844+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9709, 'log_dups_size': 0, 'ondisk_log_size': 9709, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25206784, 'num_objects': 3077, 'num_object_clones': 0, 'num_object_copies': 12308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9709, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 958}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:08.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:08.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9832", 'reported_seq': 9937, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:03.345696+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:03.345696+0000', 'last_peered': '2024-09-18T11:52:03.345696+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:03.345696+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9832, 'log_dups_size': 0, 'ondisk_log_size': 9832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26080502, 'num_objects': 3184, 'num_object_clones': 0, 'num_object_copies': 12736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9830, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1068}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:08.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:08.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9607", 'reported_seq': 9714, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:03.457598+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:52:03.457598+0000', 'last_peered': '2024-09-18T11:52:03.457598+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:03.457598+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9607, 'log_dups_size': 0, 'ondisk_log_size': 9607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26768630, 'num_objects': 3268, 'num_object_clones': 0, 'num_object_copies': 13072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3268, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9605, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1254}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:08.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:52:08.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9982", 'reported_seq': 13671, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:07.377556+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:52:07.377556+0000', 'last_peered': '2024-09-18T11:52:07.377556+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:52:07.377556+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9982, 'log_dups_size': 0, 'ondisk_log_size': 9982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26159596, 'num_objects': 3194, 'num_object_clones': 0, 'num_object_copies': 12776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9979, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1816, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2869}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:08.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:08.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9485", 'reported_seq': 9595, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:07.669458+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:07.669458+0000', 'last_peered': '2024-09-18T11:52:07.669458+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:07.669458+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9485, 'log_dups_size': 0, 'ondisk_log_size': 9485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25466102, 'num_objects': 3109, 'num_object_clones': 0, 'num_object_copies': 12436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9483, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1146}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:08.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:08.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9769", 'reported_seq': 9885, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:07.406521+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:07.406521+0000', 'last_peered': '2024-09-18T11:52:07.406521+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:07.406521+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9769, 'log_dups_size': 0, 'ondisk_log_size': 9769, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26734449, 'num_objects': 3264, 'num_object_clones': 0, 'num_object_copies': 13056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9766, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1230}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:08.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:08.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9885", 'reported_seq': 9976, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:07.286380+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:07.286380+0000', 'last_peered': '2024-09-18T11:52:07.286380+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:07.286380+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9885, 'log_dups_size': 0, 'ondisk_log_size': 9885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25770619, 'num_objects': 3146, 'num_object_clones': 0, 'num_object_copies': 12584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9884, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1062}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:08.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:08.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9762", 'reported_seq': 9858, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:07.698380+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:07.698380+0000', 'last_peered': '2024-09-18T11:52:07.698380+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:07.698380+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9762, 'log_dups_size': 0, 'ondisk_log_size': 9762, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26253947, 'num_objects': 3205, 'num_object_clones': 0, 'num_object_copies': 12820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9761, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:08.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:08.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9903", 'reported_seq': 10026, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:07.298610+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:07.298610+0000', 'last_peered': '2024-09-18T11:52:07.298610+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:07.298610+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9903, 'log_dups_size': 0, 'ondisk_log_size': 9903, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26105078, 'num_objects': 3187, 'num_object_clones': 0, 'num_object_copies': 12748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3187, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9901, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1064}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:08.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:08.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9500", 'reported_seq': 9588, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:03.345038+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:03.345038+0000', 'last_peered': '2024-09-18T11:52:03.345038+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:03.345038+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9500, 'log_dups_size': 0, 'ondisk_log_size': 9500, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26268918, 'num_objects': 3207, 'num_object_clones': 0, 'num_object_copies': 12828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9498, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1205}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:08.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:09.418 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27142/203996 objects degraded (13.305%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":527,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":50999,"data_bytes":417736076,"bytes_used":2697347072,"bytes_avail":383849709568,"bytes_total":386547056640,"degraded_objects":27142,"degraded_total":203996,"degraded_ratio":0.13305162846330321,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":233},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:09.469 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:09.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:10.004 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:10.006 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:10.032 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-18T11:52:08.019279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":417736076,"num_objects":50999,"num_object_clones":0,"num_object_copies":203996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156088,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6314,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156115,"ondisk_log_size":156115,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2634128,"kb_used_data":820776,"kb_used_omap":14,"kb_used_meta":1813297,"kb_avail":374853232,"statfs":{"total":386547056640,"available":383849709568,"internally_reserved":0,"allocated":840474624,"data_stored":839772760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1856816155},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,8,7,11,11,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22945792,"num_objects":-2801,"num_object_clones":0,"num_object_copies":-11204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003696"},"pg_stats":[{"pgid":"1.f","version":"35'10023","reported_seq":10122,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.285241+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:07.285241+0000","last_peered":"2024-09-18T11:52:07.285241+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:07.285241+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26435584,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10023,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9782","reported_seq":14071,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:02.890922+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:02.890922+0000","last_peered":"2024-09-18T11:52:02.890922+0000","last_clean":"2024-09-18T11:52:02.890922+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:02.890922+0000","last_undegraded":"2024-09-18T11:52:02.890922+0000","last_fullsized":"2024-09-18T11:52:02.890922+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26817782,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9702","reported_seq":9817,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.500119+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:07.500119+0000","last_peered":"2024-09-18T11:52:07.500119+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:07.500119+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25442963,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9701,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9710","reported_seq":9802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.438651+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:03.438651+0000","last_peered":"2024-09-18T11:52:03.438651+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:03.438651+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342646,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9708,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1115}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9755","reported_seq":9874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.229349+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:07.229349+0000","last_peered":"2024-09-18T11:52:07.229349+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:07.229349+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25720054,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9709","reported_seq":9825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.368844+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:07.368844+0000","last_peered":"2024-09-18T11:52:07.368844+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:07.368844+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25206784,"num_objects":3077,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9832","reported_seq":9937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.345696+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:03.345696+0000","last_peered":"2024-09-18T11:52:03.345696+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:03.345696+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26080502,"num_objects":3184,"num_object_clones":0,"num_object_copies":12736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1068}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9607","reported_seq":9714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.457598+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:03.457598+0000","last_peered":"2024-09-18T11:52:03.457598+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:03.457598+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9607,"log_dups_size":0,"ondisk_log_size":9607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26768630,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9605,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9982","reported_seq":13671,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.377556+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:07.377556+0000","last_peered":"2024-09-18T11:52:07.377556+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:07.377556+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9982,"log_dups_size":0,"ondisk_log_size":9982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26159596,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1816,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9485","reported_seq":9595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.669458+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:07.669458+0000","last_peered":"2024-09-18T11:52:07.669458+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:07.669458+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9485,"log_dups_size":0,"ondisk_log_size":9485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25466102,"num_objects":3109,"num_object_clones":0,"num_object_copies":12436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9769","reported_seq":9885,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.406521+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:07.406521+0000","last_peered":"2024-09-18T11:52:07.406521+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:07.406521+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26734449,"num_objects":3264,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9885","reported_seq":9976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.286380+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:07.286380+0000","last_peered":"2024-09-18T11:52:07.286380+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:07.286380+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25770619,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9884,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1062}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9762","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.698380+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:07.698380+0000","last_peered":"2024-09-18T11:52:07.698380+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:07.698380+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26253947,"num_objects":3205,"num_object_clones":0,"num_object_copies":12820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9761,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1159}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9903","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.298610+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:07.298610+0000","last_peered":"2024-09-18T11:52:07.298610+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:07.298610+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9903,"log_dups_size":0,"ondisk_log_size":9903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26105078,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1064}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9709","reported_seq":13996,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:02.724390+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:02.724390+0000","last_peered":"2024-09-18T11:52:02.724390+0000","last_clean":"2024-09-18T11:52:02.724390+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:02.724390+0000","last_undegraded":"2024-09-18T11:52:02.724390+0000","last_fullsized":"2024-09-18T11:52:02.724390+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26162422,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9500","reported_seq":9588,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:03.345038+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:03.345038+0000","last_peered":"2024-09-18T11:52:03.345038+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:03.345038+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26268918,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9498,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":417736076,"num_objects":50999,"num_object_clones":0,"num_object_copies":203996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156088,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6314,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":837783552,"data_stored":837783552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156115,"ondisk_log_size":156115,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802100,"kb_used_data":203364,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93569740,"statfs":{"total":96636764160,"available":95815413760,"internally_reserved":0,"allocated":208244736,"data_stored":208069270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,3,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.359},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":2.387}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.178000000000001},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":1.968}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.9409999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":4.0369999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797160,"kb_used_data":203104,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93574680,"statfs":{"total":96636764160,"available":95820472320,"internally_reserved":0,"allocated":207978496,"data_stored":207803030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":1.333}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.3439999999999999},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.286}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.5720000000000001},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.67900000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513324,"kb_used_data":207204,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93858516,"statfs":{"total":96636764160,"available":96111120384,"internally_reserved":0,"allocated":212176896,"data_stored":212001430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":6.0119999999999996,"5min":5.1429999999999998,"15min":3.907},"min":{"1min":0.78900000000000003,"5min":0.224,"15min":0.159},"max":{"1min":15.170999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.5880000000000001},{"interface":"front","average":{"1min":4.452,"5min":4.8390000000000004,"15min":5.0149999999999997},"min":{"1min":0.30199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.741,"5min":25.187000000000001,"15min":25.187000000000001},"last":1.573}]},{"osd":2,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":3.9209999999999998,"5min":5.5780000000000003,"15min":4.4800000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":8.8640000000000008,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":4.5759999999999996,"5min":4.5739999999999998,"15min":4.0700000000000003},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":12.833,"5min":26.719000000000001,"15min":26.719000000000001},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:51:02 2024","interfaces":[{"interface":"back","average":{"1min":5.8700000000000001,"5min":4.8600000000000003,"15min":4.4080000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":16.259,"5min":17.390999999999998,"15min":17.390999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":6.3860000000000001,"5min":6.1849999999999996,"15min":5.1369999999999996},"min":{"1min":0.65000000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.515999999999998,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.69899999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521544,"kb_used_data":207104,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93850296,"statfs":{"total":96636764160,"available":96102703104,"internally_reserved":0,"allocated":212074496,"data_stored":211899030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,1,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":20.116},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.802},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":0.751}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":18.015000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":8.4290000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211001344,"data_stored":211001344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211836928,"data_stored":211836928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207319040,"data_stored":207319040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207626240,"data_stored":207626240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:13.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:13.636 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:13.637 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:13.653 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-18T11:52:12.021018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":414623116,"num_objects":50619,"num_object_clones":0,"num_object_copies":202476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50619,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156468,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6314,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156495,"ondisk_log_size":156495,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2905788,"kb_used_data":812352,"kb_used_omap":14,"kb_used_meta":2093361,"kb_avail":374581572,"statfs":{"total":386547056640,"available":383571529728,"internally_reserved":0,"allocated":831848448,"data_stored":831146584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143601691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,15,8,11,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17326080,"num_objects":-2115,"num_object_clones":0,"num_object_copies":-8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2115,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004502"},"pg_stats":[{"pgid":"1.f","version":"35'10023","reported_seq":10122,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.285241+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:07.285241+0000","last_peered":"2024-09-18T11:52:07.285241+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:07.285241+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26435584,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10023,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9850","reported_seq":14140,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:08.070813+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:08.070813+0000","last_peered":"2024-09-18T11:52:08.070813+0000","last_clean":"2024-09-18T11:52:08.070813+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:08.070813+0000","last_undegraded":"2024-09-18T11:52:08.070813+0000","last_fullsized":"2024-09-18T11:52:08.070813+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26260726,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9702","reported_seq":9817,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.500119+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:07.500119+0000","last_peered":"2024-09-18T11:52:07.500119+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:07.500119+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25442963,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9701,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9779","reported_seq":9870,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:08.322742+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:08.322742+0000","last_peered":"2024-09-18T11:52:08.322742+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:08.322742+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9779,"log_dups_size":0,"ondisk_log_size":9779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25777398,"num_objects":3147,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9777,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1046}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9755","reported_seq":9874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.229349+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:07.229349+0000","last_peered":"2024-09-18T11:52:07.229349+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:07.229349+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25720054,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9709","reported_seq":9825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.368844+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:07.368844+0000","last_peered":"2024-09-18T11:52:07.368844+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:07.368844+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25206784,"num_objects":3077,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9884","reported_seq":9989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:08.204571+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:08.204571+0000","last_peered":"2024-09-18T11:52:08.204571+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:08.204571+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9884,"log_dups_size":0,"ondisk_log_size":9884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25654518,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9882,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1016}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9677","reported_seq":9784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:08.242234+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:08.242234+0000","last_peered":"2024-09-18T11:52:08.242234+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:08.242234+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9677,"log_dups_size":0,"ondisk_log_size":9677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195190,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9675,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9982","reported_seq":13671,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.377556+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:07.377556+0000","last_peered":"2024-09-18T11:52:07.377556+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:07.377556+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9982,"log_dups_size":0,"ondisk_log_size":9982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26159596,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1816,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9485","reported_seq":9595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.669458+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:07.669458+0000","last_peered":"2024-09-18T11:52:07.669458+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:07.669458+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9485,"log_dups_size":0,"ondisk_log_size":9485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25466102,"num_objects":3109,"num_object_clones":0,"num_object_copies":12436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9769","reported_seq":9885,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.406521+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:07.406521+0000","last_peered":"2024-09-18T11:52:07.406521+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:07.406521+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26734449,"num_objects":3264,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9885","reported_seq":9976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.286380+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:07.286380+0000","last_peered":"2024-09-18T11:52:07.286380+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:07.286380+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25770619,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9884,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1062}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9762","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.698380+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:07.698380+0000","last_peered":"2024-09-18T11:52:07.698380+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:07.698380+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26253947,"num_objects":3205,"num_object_clones":0,"num_object_copies":12820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9761,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1159}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9903","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:07.298610+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:07.298610+0000","last_peered":"2024-09-18T11:52:07.298610+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:07.298610+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9903,"log_dups_size":0,"ondisk_log_size":9903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26105078,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1064}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9767","reported_seq":14054,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:07.817407+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:07.817407+0000","last_peered":"2024-09-18T11:52:07.817407+0000","last_clean":"2024-09-18T11:52:07.817407+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:07.817407+0000","last_undegraded":"2024-09-18T11:52:07.817407+0000","last_fullsized":"2024-09-18T11:52:07.817407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9767,"log_dups_size":0,"ondisk_log_size":9767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25687286,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9765,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9563","reported_seq":9651,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:08.284192+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:08.284192+0000","last_peered":"2024-09-18T11:52:08.284192+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:08.284192+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25752822,"num_objects":3144,"num_object_clones":0,"num_object_copies":12576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9561,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":414623116,"num_objects":50619,"num_object_clones":0,"num_object_copies":202476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50619,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156468,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6314,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":829091840,"data_stored":829091840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156495,"ondisk_log_size":156495,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802100,"kb_used_data":203364,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93569740,"statfs":{"total":96636764160,"available":95815413760,"internally_reserved":0,"allocated":208244736,"data_stored":208069270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,3,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.359},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":2.387}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.178000000000001},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":1.968}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.9409999999999998},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":4.0369999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797160,"kb_used_data":203104,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93574680,"statfs":{"total":96636764160,"available":95820472320,"internally_reserved":0,"allocated":207978496,"data_stored":207803030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0670000000000002,"5min":5.1900000000000004,"15min":2.919},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":12.281000000000001,"5min":23.731999999999999,"15min":23.731999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":4.1120000000000001,"5min":4.2919999999999998,"15min":2.4820000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":12.311999999999999,"5min":17.285,"15min":18.047000000000001},"last":1.333}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.3439999999999999},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.286}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.5720000000000001},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.67900000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513196,"kb_used_data":202904,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93858644,"statfs":{"total":96636764160,"available":96111251456,"internally_reserved":0,"allocated":207773696,"data_stored":207598230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.0679999999999996},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.95099999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":6.1050000000000004},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":8.3040000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.20399999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":793332,"kb_used_data":202980,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93578508,"statfs":{"total":96636764160,"available":95824392192,"internally_reserved":0,"allocated":207851520,"data_stored":207676054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":14.499000000000001},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":14.390000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:51:11 2024","interfaces":[{"interface":"back","average":{"1min":5.9409999999999998,"5min":6.2409999999999997,"15min":2.8839999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":17.706,"5min":32.795000000000002,"15min":32.795000000000002},"last":14.653},{"interface":"front","average":{"1min":3.7290000000000001,"5min":4.6799999999999997,"15min":2.5379999999999998},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":21.885000000000002,"5min":22.739999999999998,"15min":22.739999999999998},"last":2.7240000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":14.433},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":2.286}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":206733312,"data_stored":206733312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":207413248,"data_stored":207413248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207319040,"data_stored":207319040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207626240,"data_stored":207626240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:13.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:14.187 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:14.188 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:14.216 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-18T11:52:14.021857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":406193548,"num_objects":49590,"num_object_clones":0,"num_object_copies":198360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49590,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157497,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6374,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157524,"ondisk_log_size":157524,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2885148,"kb_used_data":795884,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374602212,"statfs":{"total":386547056640,"available":383592665088,"internally_reserved":0,"allocated":814985216,"data_stored":814283352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,12,5,20,19,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25755648,"num_objects":-3144,"num_object_clones":0,"num_object_copies":-12576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3144,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004884"},"pg_stats":[{"pgid":"1.f","version":"35'10080","reported_seq":10178,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.349522+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:12.349522+0000","last_peered":"2024-09-18T11:52:12.349522+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:12.349522+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25968640,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10080,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9915","reported_seq":14204,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:13.181427+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:13.181427+0000","last_peered":"2024-09-18T11:52:13.181427+0000","last_clean":"2024-09-18T11:52:13.181427+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:13.181427+0000","last_undegraded":"2024-09-18T11:52:13.181427+0000","last_fullsized":"2024-09-18T11:52:13.181427+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9915,"log_dups_size":0,"ondisk_log_size":9915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25728246,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9913,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9768","reported_seq":9883,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.576742+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:12.576742+0000","last_peered":"2024-09-18T11:52:12.576742+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:12.576742+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9768,"log_dups_size":0,"ondisk_log_size":9768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24902291,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9767,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1001}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9846","reported_seq":9937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.349350+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:13.349350+0000","last_peered":"2024-09-18T11:52:13.349350+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:13.349350+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9846,"log_dups_size":0,"ondisk_log_size":9846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228534,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9844,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9811","reported_seq":9930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.209250+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:12.209250+0000","last_peered":"2024-09-18T11:52:12.209250+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:12.209250+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25261302,"num_objects":3084,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9809,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9770","reported_seq":9887,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.189300+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:12.189300+0000","last_peered":"2024-09-18T11:52:12.189300+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:12.189300+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9770,"log_dups_size":0,"ondisk_log_size":9770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":3016,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9770,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":897}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9949","reported_seq":10054,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.114750+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:13.114750+0000","last_peered":"2024-09-18T11:52:13.114750+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:13.114750+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25122038,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9947,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9729","reported_seq":9836,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.309973+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:13.309973+0000","last_peered":"2024-09-18T11:52:13.309973+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:13.309973+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25769206,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10040","reported_seq":13848,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.317030+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:12.317030+0000","last_peered":"2024-09-18T11:52:12.317030+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:12.317030+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25684460,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1876,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9564","reported_seq":9676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.543502+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:12.543502+0000","last_peered":"2024-09-18T11:52:12.543502+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:12.543502+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9564,"log_dups_size":0,"ondisk_log_size":9564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24818934,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9822","reported_seq":9940,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.422364+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:12.422364+0000","last_peered":"2024-09-18T11:52:12.422364+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:12.422364+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9822,"log_dups_size":0,"ondisk_log_size":9822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26300273,"num_objects":3211,"num_object_clones":0,"num_object_copies":12844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9819,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9956","reported_seq":10047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.270621+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:12.270621+0000","last_peered":"2024-09-18T11:52:12.270621+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:12.270621+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9956,"log_dups_size":0,"ondisk_log_size":9956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25188987,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9955,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":991}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9834","reported_seq":9930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.733566+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:12.733566+0000","last_peered":"2024-09-18T11:52:12.733566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:12.733566+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9834,"log_dups_size":0,"ondisk_log_size":9834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25664123,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9833,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1087}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9969","reported_seq":10090,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.158883+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:12.158883+0000","last_peered":"2024-09-18T11:52:12.158883+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:12.158883+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9969,"log_dups_size":0,"ondisk_log_size":9969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25564406,"num_objects":3121,"num_object_clones":0,"num_object_copies":12484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":998}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9827","reported_seq":14114,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:12.866953+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:12.866953+0000","last_peered":"2024-09-18T11:52:12.866953+0000","last_clean":"2024-09-18T11:52:12.866953+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:12.866953+0000","last_undegraded":"2024-09-18T11:52:12.866953+0000","last_fullsized":"2024-09-18T11:52:12.866953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9827,"log_dups_size":0,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25195766,"num_objects":3076,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9644","reported_seq":9732,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.500446+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:13.500446+0000","last_peered":"2024-09-18T11:52:13.500446+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:13.500446+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25089270,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":406193548,"num_objects":49590,"num_object_clones":0,"num_object_copies":198360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49590,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157497,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6374,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":812236800,"data_stored":812236800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157524,"ondisk_log_size":157524,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785568,"kb_used_data":199316,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93586272,"statfs":{"total":96636764160,"available":95832342528,"internally_reserved":0,"allocated":204099584,"data_stored":203924118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,12,0,2,7,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":9.8680000000000003},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":9.8409999999999993}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":9.891},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":9.2789999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":8.968},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.53100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797220,"kb_used_data":199000,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93574620,"statfs":{"total":96636764160,"available":95820410880,"internally_reserved":0,"allocated":203776000,"data_stored":203600534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,7,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":1.157},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.1819999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":5.0810000000000004},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.4510000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.8650000000000002},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.81499999999999995}]}]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509780,"kb_used_data":199492,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93862060,"statfs":{"total":96636764160,"available":96114749440,"internally_reserved":0,"allocated":204279808,"data_stored":204104342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.0679999999999996},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.95099999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":6.1050000000000004},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":8.3040000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.20399999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792580,"kb_used_data":198076,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93579260,"statfs":{"total":96636764160,"available":95825162240,"internally_reserved":0,"allocated":202829824,"data_stored":202654358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,7,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":3.02},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.9199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.1360000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":2.702}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":2.7450000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":2.7320000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202518528,"data_stored":202518528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":203157504,"data_stored":203157504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203128832,"data_stored":203128832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":203431936,"data_stored":203431936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:14.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:14.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10080", 'reported_seq': 10178, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:12.349522+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:12.349522+0000', 'last_peered': '2024-09-18T11:52:12.349522+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:12.349522+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25968640, 'num_objects': 3170, 'num_object_clones': 0, 'num_object_copies': 12680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10080, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1067}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:14.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:14.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9768", 'reported_seq': 9883, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:12.576742+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:12.576742+0000', 'last_peered': '2024-09-18T11:52:12.576742+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:12.576742+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9768, 'log_dups_size': 0, 'ondisk_log_size': 9768, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24902291, 'num_objects': 3041, 'num_object_clones': 0, 'num_object_copies': 12164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3041, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9767, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1001}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:14.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:14.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9846", 'reported_seq': 9937, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:13.349350+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:13.349350+0000', 'last_peered': '2024-09-18T11:52:13.349350+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:13.349350+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9846, 'log_dups_size': 0, 'ondisk_log_size': 9846, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25228534, 'num_objects': 3080, 'num_object_clones': 0, 'num_object_copies': 12320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9844, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 979}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:14.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:14.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9811", 'reported_seq': 9930, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:12.209250+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:12.209250+0000', 'last_peered': '2024-09-18T11:52:12.209250+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:12.209250+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9811, 'log_dups_size': 0, 'ondisk_log_size': 9811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25261302, 'num_objects': 3084, 'num_object_clones': 0, 'num_object_copies': 12336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9809, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1032}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9770", 'reported_seq': 9887, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:12.189300+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:12.189300+0000', 'last_peered': '2024-09-18T11:52:12.189300+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:12.189300+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9770, 'log_dups_size': 0, 'ondisk_log_size': 9770, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24707072, 'num_objects': 3016, 'num_object_clones': 0, 'num_object_copies': 12064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9770, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 897}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9949", 'reported_seq': 10054, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:13.114750+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:13.114750+0000', 'last_peered': '2024-09-18T11:52:13.114750+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:13.114750+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9949, 'log_dups_size': 0, 'ondisk_log_size': 9949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25122038, 'num_objects': 3067, 'num_object_clones': 0, 'num_object_copies': 12268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9947, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 951}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9729", 'reported_seq': 9836, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:13.309973+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:52:13.309973+0000', 'last_peered': '2024-09-18T11:52:13.309973+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:13.309973+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9729, 'log_dups_size': 0, 'ondisk_log_size': 9729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25769206, 'num_objects': 3146, 'num_object_clones': 0, 'num_object_copies': 12584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9727, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1132}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10040", 'reported_seq': 13848, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:12.317030+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:52:12.317030+0000', 'last_peered': '2024-09-18T11:52:12.317030+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:52:12.317030+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25684460, 'num_objects': 3136, 'num_object_clones': 0, 'num_object_copies': 12544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10037, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1876, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2871}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9564", 'reported_seq': 9676, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:12.543502+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:12.543502+0000', 'last_peered': '2024-09-18T11:52:12.543502+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:12.543502+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9564, 'log_dups_size': 0, 'ondisk_log_size': 9564, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24818934, 'num_objects': 3030, 'num_object_clones': 0, 'num_object_copies': 12120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9562, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1067}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:14.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:14.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9822", 'reported_seq': 9940, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:12.422364+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:12.422364+0000', 'last_peered': '2024-09-18T11:52:12.422364+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:12.422364+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9822, 'log_dups_size': 0, 'ondisk_log_size': 9822, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26300273, 'num_objects': 3211, 'num_object_clones': 0, 'num_object_copies': 12844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3211, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9819, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1177}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:14.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:14.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9956", 'reported_seq': 10047, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:12.270621+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:12.270621+0000', 'last_peered': '2024-09-18T11:52:12.270621+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:12.270621+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9956, 'log_dups_size': 0, 'ondisk_log_size': 9956, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25188987, 'num_objects': 3075, 'num_object_clones': 0, 'num_object_copies': 12300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9955, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 991}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:14.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:14.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9834", 'reported_seq': 9930, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:12.733566+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:12.733566+0000', 'last_peered': '2024-09-18T11:52:12.733566+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:12.733566+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9834, 'log_dups_size': 0, 'ondisk_log_size': 9834, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25664123, 'num_objects': 3133, 'num_object_clones': 0, 'num_object_copies': 12532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9833, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1087}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:14.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:14.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9969", 'reported_seq': 10090, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:12.158883+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:12.158883+0000', 'last_peered': '2024-09-18T11:52:12.158883+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:12.158883+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9969, 'log_dups_size': 0, 'ondisk_log_size': 9969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25564406, 'num_objects': 3121, 'num_object_clones': 0, 'num_object_copies': 12484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9967, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:14.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:14.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9644", 'reported_seq': 9732, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:13.500446+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:13.500446+0000', 'last_peered': '2024-09-18T11:52:13.500446+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:13.500446+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9644, 'log_dups_size': 0, 'ondisk_log_size': 9644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25089270, 'num_objects': 3063, 'num_object_clones': 0, 'num_object_copies': 12252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9642, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1061}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:14.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:14.938 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:14.939 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27082/198360 objects degraded (13.653%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":533,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49590,"data_bytes":406193548,"bytes_used":2954391552,"bytes_avail":383592665088,"bytes_total":386547056640,"degraded_objects":27082,"degraded_total":198360,"degraded_ratio":0.13652954224642064,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":261},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:14.962 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:14.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:15.521 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:15.521 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:15.552 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-18T11:52:14.021857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":406193548,"num_objects":49590,"num_object_clones":0,"num_object_copies":198360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49590,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157497,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6374,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157524,"ondisk_log_size":157524,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2885148,"kb_used_data":795884,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374602212,"statfs":{"total":386547056640,"available":383592665088,"internally_reserved":0,"allocated":814985216,"data_stored":814283352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,12,5,20,19,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25755648,"num_objects":-3144,"num_object_clones":0,"num_object_copies":-12576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3144,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004884"},"pg_stats":[{"pgid":"1.f","version":"35'10080","reported_seq":10178,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.349522+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:12.349522+0000","last_peered":"2024-09-18T11:52:12.349522+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:12.349522+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25968640,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10080,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9915","reported_seq":14204,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:13.181427+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:13.181427+0000","last_peered":"2024-09-18T11:52:13.181427+0000","last_clean":"2024-09-18T11:52:13.181427+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:13.181427+0000","last_undegraded":"2024-09-18T11:52:13.181427+0000","last_fullsized":"2024-09-18T11:52:13.181427+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9915,"log_dups_size":0,"ondisk_log_size":9915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25728246,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9913,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9768","reported_seq":9883,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.576742+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:12.576742+0000","last_peered":"2024-09-18T11:52:12.576742+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:12.576742+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9768,"log_dups_size":0,"ondisk_log_size":9768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24902291,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9767,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1001}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9846","reported_seq":9937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.349350+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:13.349350+0000","last_peered":"2024-09-18T11:52:13.349350+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:13.349350+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9846,"log_dups_size":0,"ondisk_log_size":9846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228534,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9844,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9811","reported_seq":9930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.209250+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:12.209250+0000","last_peered":"2024-09-18T11:52:12.209250+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:12.209250+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25261302,"num_objects":3084,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9809,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9770","reported_seq":9887,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.189300+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:12.189300+0000","last_peered":"2024-09-18T11:52:12.189300+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:12.189300+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9770,"log_dups_size":0,"ondisk_log_size":9770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":3016,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9770,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":897}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9949","reported_seq":10054,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.114750+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:13.114750+0000","last_peered":"2024-09-18T11:52:13.114750+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:13.114750+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25122038,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9947,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9729","reported_seq":9836,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.309973+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:13.309973+0000","last_peered":"2024-09-18T11:52:13.309973+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:13.309973+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25769206,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10040","reported_seq":13848,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.317030+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:12.317030+0000","last_peered":"2024-09-18T11:52:12.317030+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:12.317030+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25684460,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1876,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9564","reported_seq":9676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.543502+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:12.543502+0000","last_peered":"2024-09-18T11:52:12.543502+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:12.543502+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9564,"log_dups_size":0,"ondisk_log_size":9564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24818934,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9822","reported_seq":9940,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.422364+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:12.422364+0000","last_peered":"2024-09-18T11:52:12.422364+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:12.422364+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9822,"log_dups_size":0,"ondisk_log_size":9822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26300273,"num_objects":3211,"num_object_clones":0,"num_object_copies":12844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9819,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9956","reported_seq":10047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.270621+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:12.270621+0000","last_peered":"2024-09-18T11:52:12.270621+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:12.270621+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9956,"log_dups_size":0,"ondisk_log_size":9956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25188987,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9955,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":991}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9834","reported_seq":9930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.733566+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:12.733566+0000","last_peered":"2024-09-18T11:52:12.733566+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:12.733566+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9834,"log_dups_size":0,"ondisk_log_size":9834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25664123,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9833,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1087}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9969","reported_seq":10090,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:12.158883+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:12.158883+0000","last_peered":"2024-09-18T11:52:12.158883+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:12.158883+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9969,"log_dups_size":0,"ondisk_log_size":9969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25564406,"num_objects":3121,"num_object_clones":0,"num_object_copies":12484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":998}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9827","reported_seq":14114,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:12.866953+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:12.866953+0000","last_peered":"2024-09-18T11:52:12.866953+0000","last_clean":"2024-09-18T11:52:12.866953+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:12.866953+0000","last_undegraded":"2024-09-18T11:52:12.866953+0000","last_fullsized":"2024-09-18T11:52:12.866953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9827,"log_dups_size":0,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25195766,"num_objects":3076,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9644","reported_seq":9732,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.500446+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:13.500446+0000","last_peered":"2024-09-18T11:52:13.500446+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:13.500446+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25089270,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":406193548,"num_objects":49590,"num_object_clones":0,"num_object_copies":198360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49590,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157497,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6374,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":812236800,"data_stored":812236800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157524,"ondisk_log_size":157524,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785568,"kb_used_data":199316,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93586272,"statfs":{"total":96636764160,"available":95832342528,"internally_reserved":0,"allocated":204099584,"data_stored":203924118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,12,0,2,7,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":9.8680000000000003},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":9.8409999999999993}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":9.891},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":9.2789999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":8.968},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.53100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797220,"kb_used_data":199000,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93574620,"statfs":{"total":96636764160,"available":95820410880,"internally_reserved":0,"allocated":203776000,"data_stored":203600534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,7,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":1.157},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.1819999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":5.0810000000000004},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.4510000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.8650000000000002},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.81499999999999995}]}]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509780,"kb_used_data":199492,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93862060,"statfs":{"total":96636764160,"available":96114749440,"internally_reserved":0,"allocated":204279808,"data_stored":204104342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.0679999999999996},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.95099999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":6.1050000000000004},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":8.3040000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.20399999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792580,"kb_used_data":198076,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93579260,"statfs":{"total":96636764160,"available":95825162240,"internally_reserved":0,"allocated":202829824,"data_stored":202654358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,7,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":3.02},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.9199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.1360000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":2.702}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":2.7450000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":2.7320000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202518528,"data_stored":202518528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":203157504,"data_stored":203157504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203128832,"data_stored":203128832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":203431936,"data_stored":203431936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:18.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:19.166 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:19.166 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:19.191 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-18T11:52:18.023497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":401032588,"num_objects":48960,"num_object_clones":0,"num_object_copies":195840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158127,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6433,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157954,"ondisk_log_size":157954,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2885312,"kb_used_data":787712,"kb_used_omap":14,"kb_used_meta":2097521,"kb_avail":374602048,"statfs":{"total":386547056640,"available":383592497152,"internally_reserved":0,"allocated":806617088,"data_stored":805915224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2147861531},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,5,13,13,11,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22151168,"num_objects":-2704,"num_object_clones":0,"num_object_copies":-10816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2704,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005025"},"pg_stats":[{"pgid":"1.f","version":"35'10130","reported_seq":10229,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.236771+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:17.236771+0000","last_peered":"2024-09-18T11:52:17.236771+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:17.236771+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25559040,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10130,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9915","reported_seq":14204,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:13.181427+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:13.181427+0000","last_peered":"2024-09-18T11:52:13.181427+0000","last_clean":"2024-09-18T11:52:13.181427+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:13.181427+0000","last_undegraded":"2024-09-18T11:52:13.181427+0000","last_fullsized":"2024-09-18T11:52:13.181427+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9915,"log_dups_size":0,"ondisk_log_size":9915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25728246,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9913,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9824","reported_seq":9939,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.561239+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:17.561239+0000","last_peered":"2024-09-18T11:52:17.561239+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:17.561239+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9824,"log_dups_size":0,"ondisk_log_size":9824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24443539,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9823,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":945}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9846","reported_seq":9937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.349350+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:13.349350+0000","last_peered":"2024-09-18T11:52:13.349350+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:13.349350+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9846,"log_dups_size":0,"ondisk_log_size":9846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228534,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9844,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9864","reported_seq":9983,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.348343+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:17.348343+0000","last_peered":"2024-09-18T11:52:17.348343+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:17.348343+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24827126,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9828","reported_seq":9945,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.325843+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:17.325843+0000","last_peered":"2024-09-18T11:52:17.325843+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:17.325843+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9828,"log_dups_size":0,"ondisk_log_size":9828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24231936,"num_objects":2958,"num_object_clones":0,"num_object_copies":11832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9828,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9949","reported_seq":10054,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.114750+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:13.114750+0000","last_peered":"2024-09-18T11:52:13.114750+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:13.114750+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25122038,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9947,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9729","reported_seq":9836,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.309973+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:13.309973+0000","last_peered":"2024-09-18T11:52:13.309973+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:13.309973+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25769206,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10109","reported_seq":14033,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.409780+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:17.409780+0000","last_peered":"2024-09-18T11:52:17.409780+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:17.409780+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25119212,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10106,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1935,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9638","reported_seq":9749,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.644398+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:17.644398+0000","last_peered":"2024-09-18T11:52:17.644398+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:17.644398+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24212726,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9636,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9897","reported_seq":10014,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.436851+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:17.436851+0000","last_peered":"2024-09-18T11:52:17.436851+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:17.436851+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25685873,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9894,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10014","reported_seq":10105,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.256059+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:17.256059+0000","last_peered":"2024-09-18T11:52:17.256059+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:17.256059+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24713851,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":933}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9904","reported_seq":9999,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.701446+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:17.701446+0000","last_peered":"2024-09-18T11:52:17.701446+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:17.701446+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25090683,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9903,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10036","reported_seq":10159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.191083+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:17.191083+0000","last_peered":"2024-09-18T11:52:17.191083+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:17.191083+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25015542,"num_objects":3054,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10034,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":931}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9827","reported_seq":14114,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:12.866953+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:12.866953+0000","last_peered":"2024-09-18T11:52:12.866953+0000","last_clean":"2024-09-18T11:52:12.866953+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:12.866953+0000","last_undegraded":"2024-09-18T11:52:12.866953+0000","last_fullsized":"2024-09-18T11:52:12.866953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9827,"log_dups_size":0,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25195766,"num_objects":3076,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9644","reported_seq":9732,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.500446+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:13.500446+0000","last_peered":"2024-09-18T11:52:13.500446+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:13.500446+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25089270,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":401032588,"num_objects":48960,"num_object_clones":0,"num_object_copies":195840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158127,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6433,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":803909632,"data_stored":803909632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157954,"ondisk_log_size":157954,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785724,"kb_used_data":195296,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93586116,"statfs":{"total":96636764160,"available":95832182784,"internally_reserved":0,"allocated":199983104,"data_stored":199807638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":21.420000000000002},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":21.510999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":3.4039999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":3.5550000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":27.411999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.55400000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797228,"kb_used_data":194848,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":93574612,"statfs":{"total":96636764160,"available":95820402688,"internally_reserved":0,"allocated":199524352,"data_stored":199348886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":2.617},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":2.6389999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.6749999999999998},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.585}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":6.681},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.55200000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509780,"kb_used_data":199492,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93862060,"statfs":{"total":96636764160,"available":96114749440,"internally_reserved":0,"allocated":204279808,"data_stored":204104342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.0679999999999996},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.95099999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":6.1050000000000004},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":8.3040000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.20399999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792580,"kb_used_data":198076,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93579260,"statfs":{"total":96636764160,"available":95825162240,"internally_reserved":0,"allocated":202829824,"data_stored":202654358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,7,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":3.02},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.9199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.1360000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":2.702}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":2.7450000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":2.7320000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202518528,"data_stored":202518528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":203157504,"data_stored":203157504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":198950912,"data_stored":198950912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":199282688,"data_stored":199282688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:19.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:19.745 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:19.746 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:19.771 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-18T11:52:18.023497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":401032588,"num_objects":48960,"num_object_clones":0,"num_object_copies":195840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158127,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6433,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157954,"ondisk_log_size":157954,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2885312,"kb_used_data":787712,"kb_used_omap":14,"kb_used_meta":2097521,"kb_avail":374602048,"statfs":{"total":386547056640,"available":383592497152,"internally_reserved":0,"allocated":806617088,"data_stored":805915224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2147861531},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,5,13,13,11,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22151168,"num_objects":-2704,"num_object_clones":0,"num_object_copies":-10816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2704,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005025"},"pg_stats":[{"pgid":"1.f","version":"35'10130","reported_seq":10229,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.236771+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:17.236771+0000","last_peered":"2024-09-18T11:52:17.236771+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:17.236771+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25559040,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10130,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9915","reported_seq":14204,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:13.181427+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:13.181427+0000","last_peered":"2024-09-18T11:52:13.181427+0000","last_clean":"2024-09-18T11:52:13.181427+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:13.181427+0000","last_undegraded":"2024-09-18T11:52:13.181427+0000","last_fullsized":"2024-09-18T11:52:13.181427+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9915,"log_dups_size":0,"ondisk_log_size":9915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25728246,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9913,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9824","reported_seq":9939,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.561239+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:17.561239+0000","last_peered":"2024-09-18T11:52:17.561239+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:17.561239+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9824,"log_dups_size":0,"ondisk_log_size":9824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24443539,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9823,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":945}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9846","reported_seq":9937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.349350+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:13.349350+0000","last_peered":"2024-09-18T11:52:13.349350+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:13.349350+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9846,"log_dups_size":0,"ondisk_log_size":9846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228534,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9844,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9864","reported_seq":9983,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.348343+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:17.348343+0000","last_peered":"2024-09-18T11:52:17.348343+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:17.348343+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24827126,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9828","reported_seq":9945,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.325843+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:17.325843+0000","last_peered":"2024-09-18T11:52:17.325843+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:17.325843+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9828,"log_dups_size":0,"ondisk_log_size":9828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24231936,"num_objects":2958,"num_object_clones":0,"num_object_copies":11832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9828,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9949","reported_seq":10054,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.114750+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:13.114750+0000","last_peered":"2024-09-18T11:52:13.114750+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:13.114750+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25122038,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9947,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9729","reported_seq":9836,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.309973+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:13.309973+0000","last_peered":"2024-09-18T11:52:13.309973+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:13.309973+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25769206,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10109","reported_seq":14033,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.409780+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:17.409780+0000","last_peered":"2024-09-18T11:52:17.409780+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:17.409780+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25119212,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10106,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1935,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9638","reported_seq":9749,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.644398+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:17.644398+0000","last_peered":"2024-09-18T11:52:17.644398+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:17.644398+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24212726,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9636,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9897","reported_seq":10014,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.436851+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:17.436851+0000","last_peered":"2024-09-18T11:52:17.436851+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:17.436851+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25685873,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9894,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10014","reported_seq":10105,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.256059+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:17.256059+0000","last_peered":"2024-09-18T11:52:17.256059+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:17.256059+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24713851,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":933}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9904","reported_seq":9999,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.701446+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:17.701446+0000","last_peered":"2024-09-18T11:52:17.701446+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:17.701446+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25090683,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9903,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10036","reported_seq":10159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.191083+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:17.191083+0000","last_peered":"2024-09-18T11:52:17.191083+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:17.191083+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25015542,"num_objects":3054,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10034,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":931}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9827","reported_seq":14114,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:12.866953+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:12.866953+0000","last_peered":"2024-09-18T11:52:12.866953+0000","last_clean":"2024-09-18T11:52:12.866953+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:12.866953+0000","last_undegraded":"2024-09-18T11:52:12.866953+0000","last_fullsized":"2024-09-18T11:52:12.866953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9827,"log_dups_size":0,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25195766,"num_objects":3076,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9644","reported_seq":9732,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:13.500446+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:13.500446+0000","last_peered":"2024-09-18T11:52:13.500446+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:13.500446+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25089270,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":401032588,"num_objects":48960,"num_object_clones":0,"num_object_copies":195840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158127,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6433,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":803909632,"data_stored":803909632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157954,"ondisk_log_size":157954,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785724,"kb_used_data":195296,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93586116,"statfs":{"total":96636764160,"available":95832182784,"internally_reserved":0,"allocated":199983104,"data_stored":199807638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":21.420000000000002},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":21.510999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":3.4039999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":3.5550000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":27.411999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.55400000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797228,"kb_used_data":194848,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":93574612,"statfs":{"total":96636764160,"available":95820402688,"internally_reserved":0,"allocated":199524352,"data_stored":199348886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":2.617},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":2.6389999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.6749999999999998},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.585}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":6.681},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.55200000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509780,"kb_used_data":199492,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93862060,"statfs":{"total":96636764160,"available":96114749440,"internally_reserved":0,"allocated":204279808,"data_stored":204104342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.0679999999999996},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":0.95099999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":6.1050000000000004},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":8.3040000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.20399999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792580,"kb_used_data":198076,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93579260,"statfs":{"total":96636764160,"available":95825162240,"internally_reserved":0,"allocated":202829824,"data_stored":202654358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,7,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":3.02},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.9199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":3.1360000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":2.702}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":2.7450000000000001},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":2.7320000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202518528,"data_stored":202518528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":203157504,"data_stored":203157504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":198950912,"data_stored":198950912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":199282688,"data_stored":199282688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:19.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:19.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10130", 'reported_seq': 10229, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:17.236771+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:17.236771+0000', 'last_peered': '2024-09-18T11:52:17.236771+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:17.236771+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25559040, 'num_objects': 3120, 'num_object_clones': 0, 'num_object_copies': 12480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10130, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1017}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:19.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:19.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9824", 'reported_seq': 9939, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:17.561239+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:17.561239+0000', 'last_peered': '2024-09-18T11:52:17.561239+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:17.561239+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9824, 'log_dups_size': 0, 'ondisk_log_size': 9824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24443539, 'num_objects': 2985, 'num_object_clones': 0, 'num_object_copies': 11940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2985, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9823, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 945}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:19.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:19.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9846", 'reported_seq': 9937, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:13.349350+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:13.349350+0000', 'last_peered': '2024-09-18T11:52:13.349350+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:13.349350+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9846, 'log_dups_size': 0, 'ondisk_log_size': 9846, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25228534, 'num_objects': 3080, 'num_object_clones': 0, 'num_object_copies': 12320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9844, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 979}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:19.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:19.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9864", 'reported_seq': 9983, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:17.348343+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:17.348343+0000', 'last_peered': '2024-09-18T11:52:17.348343+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:17.348343+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9864, 'log_dups_size': 0, 'ondisk_log_size': 9864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24827126, 'num_objects': 3031, 'num_object_clones': 0, 'num_object_copies': 12124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9862, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 979}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:19.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:19.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9828", 'reported_seq': 9945, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:17.325843+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:17.325843+0000', 'last_peered': '2024-09-18T11:52:17.325843+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:17.325843+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9828, 'log_dups_size': 0, 'ondisk_log_size': 9828, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24231936, 'num_objects': 2958, 'num_object_clones': 0, 'num_object_copies': 11832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2958, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9828, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 839}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:19.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:19.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9949", 'reported_seq': 10054, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:13.114750+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:13.114750+0000', 'last_peered': '2024-09-18T11:52:13.114750+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:13.114750+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9949, 'log_dups_size': 0, 'ondisk_log_size': 9949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25122038, 'num_objects': 3067, 'num_object_clones': 0, 'num_object_copies': 12268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9947, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 951}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:19.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:19.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9729", 'reported_seq': 9836, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:13.309973+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:52:13.309973+0000', 'last_peered': '2024-09-18T11:52:13.309973+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:13.309973+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9729, 'log_dups_size': 0, 'ondisk_log_size': 9729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25769206, 'num_objects': 3146, 'num_object_clones': 0, 'num_object_copies': 12584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9727, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1132}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:19.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:52:19.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10109", 'reported_seq': 14033, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:17.409780+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:52:17.409780+0000', 'last_peered': '2024-09-18T11:52:17.409780+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:52:17.409780+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25119212, 'num_objects': 3067, 'num_object_clones': 0, 'num_object_copies': 12268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10106, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1935, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2861}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:19.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:19.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9638", 'reported_seq': 9749, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:17.644398+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:17.644398+0000', 'last_peered': '2024-09-18T11:52:17.644398+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:17.644398+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9638, 'log_dups_size': 0, 'ondisk_log_size': 9638, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24212726, 'num_objects': 2956, 'num_object_clones': 0, 'num_object_copies': 11824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9636, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:19.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:19.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9897", 'reported_seq': 10014, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:17.436851+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:17.436851+0000', 'last_peered': '2024-09-18T11:52:17.436851+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:17.436851+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9897, 'log_dups_size': 0, 'ondisk_log_size': 9897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25685873, 'num_objects': 3136, 'num_object_clones': 0, 'num_object_copies': 12544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9894, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1102}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:19.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:19.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10014", 'reported_seq': 10105, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:17.256059+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:17.256059+0000', 'last_peered': '2024-09-18T11:52:17.256059+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:17.256059+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24713851, 'num_objects': 3017, 'num_object_clones': 0, 'num_object_copies': 12068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10013, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 933}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:19.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:19.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9904", 'reported_seq': 9999, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:17.701446+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:17.701446+0000', 'last_peered': '2024-09-18T11:52:17.701446+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:17.701446+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9904, 'log_dups_size': 0, 'ondisk_log_size': 9904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25090683, 'num_objects': 3063, 'num_object_clones': 0, 'num_object_copies': 12252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9903, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:19.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:19.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10036", 'reported_seq': 10159, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:17.191083+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:17.191083+0000', 'last_peered': '2024-09-18T11:52:17.191083+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:17.191083+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25015542, 'num_objects': 3054, 'num_object_clones': 0, 'num_object_copies': 12216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10034, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 931}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:19.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:19.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9644", 'reported_seq': 9732, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:13.500446+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:13.500446+0000', 'last_peered': '2024-09-18T11:52:13.500446+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:13.500446+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9644, 'log_dups_size': 0, 'ondisk_log_size': 9644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25089270, 'num_objects': 3063, 'num_object_clones': 0, 'num_object_copies': 12252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9642, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1061}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:19.776 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:20.431 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:20.432 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27023/194252 objects degraded (13.911%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":538,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":48563,"data_bytes":397780364,"bytes_used":2941218816,"bytes_avail":383605837824,"bytes_total":386547056640,"degraded_objects":27023,"degraded_total":194252,"degraded_ratio":0.13911311080452196,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":202},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:20.453 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:20.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:21.023 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:21.024 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:21.044 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-18T11:52:20.024361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":397780364,"num_objects":48563,"num_object_clones":0,"num_object_copies":194252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48563,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158524,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6433,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158351,"ondisk_log_size":158351,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2872284,"kb_used_data":778836,"kb_used_omap":14,"kb_used_meta":2093361,"kb_avail":374615076,"statfs":{"total":386547056640,"available":383605837824,"internally_reserved":0,"allocated":797528064,"data_stored":796826200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143601691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,9,11,15,11,17,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19955712,"num_objects":-2436,"num_object_clones":0,"num_object_copies":-9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005081"},"pg_stats":[{"pgid":"1.f","version":"35'10130","reported_seq":10229,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.236771+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:17.236771+0000","last_peered":"2024-09-18T11:52:17.236771+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:17.236771+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25559040,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10130,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9977","reported_seq":14264,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:18.128673+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:18.128673+0000","last_peered":"2024-09-18T11:52:18.128673+0000","last_clean":"2024-09-18T11:52:18.128673+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:18.128673+0000","last_undegraded":"2024-09-18T11:52:18.128673+0000","last_fullsized":"2024-09-18T11:52:18.128673+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9977,"log_dups_size":0,"ondisk_log_size":9977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25220342,"num_objects":3079,"num_object_clones":0,"num_object_copies":12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9975,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9824","reported_seq":9939,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.561239+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:17.561239+0000","last_peered":"2024-09-18T11:52:17.561239+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:17.561239+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9824,"log_dups_size":0,"ondisk_log_size":9824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24443539,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9823,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":945}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9910","reported_seq":10002,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:18.536572+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:18.536572+0000","last_peered":"2024-09-18T11:52:18.536572+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:18.536572+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24704246,"num_objects":3016,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9908,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":915}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9864","reported_seq":9983,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.348343+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:17.348343+0000","last_peered":"2024-09-18T11:52:17.348343+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:17.348343+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24827126,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9828","reported_seq":9945,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.325843+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:17.325843+0000","last_peered":"2024-09-18T11:52:17.325843+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:17.325843+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9828,"log_dups_size":0,"ondisk_log_size":9828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24231936,"num_objects":2958,"num_object_clones":0,"num_object_copies":11832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9828,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10023","reported_seq":10128,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:18.532669+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:18.532669+0000","last_peered":"2024-09-18T11:52:18.532669+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:18.532669+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24515830,"num_objects":2993,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10021,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":877}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9799","reported_seq":9906,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:18.456690+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:18.456690+0000","last_peered":"2024-09-18T11:52:18.456690+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:18.456690+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9799,"log_dups_size":0,"ondisk_log_size":9799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25195766,"num_objects":3076,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9797,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1062}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10109","reported_seq":14033,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.409780+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:17.409780+0000","last_peered":"2024-09-18T11:52:17.409780+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:17.409780+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25119212,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10106,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1935,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9638","reported_seq":9749,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.644398+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:17.644398+0000","last_peered":"2024-09-18T11:52:17.644398+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:17.644398+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24212726,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9636,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9897","reported_seq":10014,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.436851+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:17.436851+0000","last_peered":"2024-09-18T11:52:17.436851+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:17.436851+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25685873,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9894,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10014","reported_seq":10105,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.256059+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:17.256059+0000","last_peered":"2024-09-18T11:52:17.256059+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:17.256059+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24713851,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":933}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9904","reported_seq":9999,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.701446+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:17.701446+0000","last_peered":"2024-09-18T11:52:17.701446+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:17.701446+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25090683,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9903,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10036","reported_seq":10159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:17.191083+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:17.191083+0000","last_peered":"2024-09-18T11:52:17.191083+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:17.191083+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25015542,"num_objects":3054,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10034,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":931}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9896","reported_seq":14183,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:17.961167+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:17.961167+0000","last_peered":"2024-09-18T11:52:17.961167+0000","last_clean":"2024-09-18T11:52:17.961167+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:17.961167+0000","last_undegraded":"2024-09-18T11:52:17.961167+0000","last_fullsized":"2024-09-18T11:52:17.961167+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9896,"log_dups_size":0,"ondisk_log_size":9896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24630518,"num_objects":3007,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9894,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9702","reported_seq":9790,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:18.442698+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:18.442698+0000","last_peered":"2024-09-18T11:52:18.442698+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:18.442698+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24614134,"num_objects":3005,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9700,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":397780364,"num_objects":48563,"num_object_clones":0,"num_object_copies":194252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48563,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158524,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6433,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":795381760,"data_stored":795381760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158351,"ondisk_log_size":158351,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785724,"kb_used_data":195296,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93586116,"statfs":{"total":96636764160,"available":95832182784,"internally_reserved":0,"allocated":199983104,"data_stored":199807638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":5.6200000000000001,"5min":5.6139999999999999,"15min":3.8069999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.155},"max":{"1min":18.882000000000001,"5min":21.539999999999999,"15min":21.539999999999999},"last":21.420000000000002},{"interface":"front","average":{"1min":3.7639999999999998,"5min":5.1630000000000003,"15min":3.71},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.782,"5min":21.562000000000001,"15min":31.675000000000001},"last":21.510999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":3.4039999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":3.5550000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":6.1500000000000004,"15min":2.8839999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.752000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":27.411999999999999},{"interface":"front","average":{"1min":5.5549999999999997,"5min":4.6749999999999998,"15min":3.4420000000000002},"min":{"1min":0.35699999999999998,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":21.431999999999999,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.55400000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797228,"kb_used_data":194848,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":93574612,"statfs":{"total":96636764160,"available":95820402688,"internally_reserved":0,"allocated":199524352,"data_stored":199348886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":2.617},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":2.6389999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":2.6749999999999998},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":2.585}]},{"osd":3,"last update":"Wed Sep 18 11:51:20 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":4.2709999999999999,"15min":3.1320000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":14.108000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":6.681},{"interface":"front","average":{"1min":4.8869999999999996,"5min":3.2160000000000002,"15min":2.419},"min":{"1min":0.35499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.884,"5min":18.884,"15min":18.884},"last":0.55200000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496616,"kb_used_data":194640,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93875224,"statfs":{"total":96636764160,"available":96128229376,"internally_reserved":0,"allocated":199311360,"data_stored":199135894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,3,4,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":1.3140000000000001},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":11.387}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":11.499000000000001},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":6.1429999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.3149999999999999},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.248}]}]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792716,"kb_used_data":194052,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93579124,"statfs":{"total":96636764160,"available":95825022976,"internally_reserved":0,"allocated":198709248,"data_stored":198533782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":27.419},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":13.497}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.3179999999999996},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":4.8879999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:51:19 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":4.2629999999999999,"15min":3.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.8109999999999999,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.3810000000000002},{"interface":"front","average":{"1min":3.6779999999999999,"5min":5.6970000000000001,"15min":3.0870000000000002},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":18.57,"5min":19.129999999999999,"15min":19.129999999999999},"last":13.587999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198225920,"data_stored":198225920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198922240,"data_stored":198922240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":198950912,"data_stored":198950912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":199282688,"data_stored":199282688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:24.046 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:24.596 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:24.614 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-18T11:52:24.025969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389612940,"num_objects":47566,"num_object_clones":0,"num_object_copies":190264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6501,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159348,"ondisk_log_size":159348,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2856224,"kb_used_data":762792,"kb_used_omap":14,"kb_used_meta":2093361,"kb_avail":374631136,"statfs":{"total":386547056640,"available":383622283264,"internally_reserved":0,"allocated":781099008,"data_stored":780397144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143601691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,7,8,26,18,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25010176,"num_objects":-3053,"num_object_clones":0,"num_object_copies":-12212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3053,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004950"},"pg_stats":[{"pgid":"1.f","version":"35'10193","reported_seq":10291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.283449+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:22.283449+0000","last_peered":"2024-09-18T11:52:22.283449+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:22.283449+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25042944,"num_objects":3057,"num_object_clones":0,"num_object_copies":12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10193,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":954}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10039","reported_seq":14329,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:23.189369+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:23.189369+0000","last_peered":"2024-09-18T11:52:23.189369+0000","last_clean":"2024-09-18T11:52:23.189369+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:23.189369+0000","last_undegraded":"2024-09-18T11:52:23.189369+0000","last_fullsized":"2024-09-18T11:52:23.189369+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24712438,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9892","reported_seq":10007,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.780828+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:22.780828+0000","last_peered":"2024-09-18T11:52:22.780828+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:22.780828+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23886483,"num_objects":2917,"num_object_clones":0,"num_object_copies":11668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9891,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":877}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9981","reported_seq":10073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.598275+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:23.598275+0000","last_peered":"2024-09-18T11:52:23.598275+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:23.598275+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24122614,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9923","reported_seq":10041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.284346+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:22.284346+0000","last_peered":"2024-09-18T11:52:22.284346+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:22.284346+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9923,"log_dups_size":0,"ondisk_log_size":9923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24343798,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9921,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9891","reported_seq":10008,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.270812+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:22.270812+0000","last_peered":"2024-09-18T11:52:22.270812+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:22.270812+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23715840,"num_objects":2895,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10084","reported_seq":10189,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.465262+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:23.465262+0000","last_peered":"2024-09-18T11:52:23.465262+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:23.465262+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016118,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10082,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":816}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9865","reported_seq":9971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.572449+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:23.572449+0000","last_peered":"2024-09-18T11:52:23.572449+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:23.572449+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24655094,"num_objects":3010,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9863,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":996}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10155","reported_seq":14218,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.473244+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:22.473244+0000","last_peered":"2024-09-18T11:52:22.473244+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:22.473244+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24742380,"num_objects":3021,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2003,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9693","reported_seq":9805,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.699260+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:22.699260+0000","last_peered":"2024-09-18T11:52:22.699260+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:22.699260+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9693,"log_dups_size":0,"ondisk_log_size":9693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762166,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9691,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":938}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9979","reported_seq":10098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.406448+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:22.406448+0000","last_peered":"2024-09-18T11:52:22.406448+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:22.406448+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9979,"log_dups_size":0,"ondisk_log_size":9979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25014129,"num_objects":3054,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10075","reported_seq":10164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.420184+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:22.420184+0000","last_peered":"2024-09-18T11:52:22.420184+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:22.420184+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24214139,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":872}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9967","reported_seq":10063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.635134+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:22.635134+0000","last_peered":"2024-09-18T11:52:22.635134+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:22.635134+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24574587,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9966,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10099","reported_seq":10222,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.391592+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:22.391592+0000","last_peered":"2024-09-18T11:52:22.391592+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:22.391592+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24499446,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10097,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":868}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9946","reported_seq":14233,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:23.083499+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:23.083499+0000","last_peered":"2024-09-18T11:52:23.083499+0000","last_clean":"2024-09-18T11:52:23.083499+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:23.083499+0000","last_undegraded":"2024-09-18T11:52:23.083499+0000","last_fullsized":"2024-09-18T11:52:23.083499+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24220918,"num_objects":2957,"num_object_clones":0,"num_object_copies":11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9766","reported_seq":9854,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.574794+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:23.574794+0000","last_peered":"2024-09-18T11:52:23.574794+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:23.574794+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9766,"log_dups_size":0,"ondisk_log_size":9766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24089846,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9764,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389612940,"num_objects":47566,"num_object_clones":0,"num_object_copies":190264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6501,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":778862592,"data_stored":778862592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159348,"ondisk_log_size":159348,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785912,"kb_used_data":191336,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93585928,"statfs":{"total":96636764160,"available":95831990272,"internally_reserved":0,"allocated":195928064,"data_stored":195752598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,4,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.638999999999999},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.839}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":10.404}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":9.1519999999999992},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.2270000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":780684,"kb_used_data":190788,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93591156,"statfs":{"total":96636764160,"available":95837343744,"internally_reserved":0,"allocated":195366912,"data_stored":195191446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":5.9939999999999998},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":6.0209999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":24},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":4.1600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.9580000000000002},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":8.4730000000000008}]}]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496696,"kb_used_data":190572,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93875144,"statfs":{"total":96636764160,"available":96128147456,"internally_reserved":0,"allocated":195145728,"data_stored":194970262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":10.249000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":0.40899999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.375},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.436}]}]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792932,"kb_used_data":190096,"kb_used_omap":4,"kb_used_meta":602811,"kb_avail":93578908,"statfs":{"total":96636764160,"available":95824801792,"internally_reserved":0,"allocated":194658304,"data_stored":194482838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617279041},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,12,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":16.521000000000001},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.9640000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.0270000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":6.9349999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":3.9329999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":6.9859999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":194101248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":194772992,"data_stored":194772992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194850816,"data_stored":194850816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":195137536,"data_stored":195137536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:24.616 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:25.222 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:25.223 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:25.248 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-18T11:52:24.025969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389612940,"num_objects":47566,"num_object_clones":0,"num_object_copies":190264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6501,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159348,"ondisk_log_size":159348,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2856224,"kb_used_data":762792,"kb_used_omap":14,"kb_used_meta":2093361,"kb_avail":374631136,"statfs":{"total":386547056640,"available":383622283264,"internally_reserved":0,"allocated":781099008,"data_stored":780397144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143601691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,7,8,26,18,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25010176,"num_objects":-3053,"num_object_clones":0,"num_object_copies":-12212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3053,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004950"},"pg_stats":[{"pgid":"1.f","version":"35'10193","reported_seq":10291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.283449+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:22.283449+0000","last_peered":"2024-09-18T11:52:22.283449+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:22.283449+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25042944,"num_objects":3057,"num_object_clones":0,"num_object_copies":12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10193,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":954}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10039","reported_seq":14329,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:23.189369+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:23.189369+0000","last_peered":"2024-09-18T11:52:23.189369+0000","last_clean":"2024-09-18T11:52:23.189369+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:23.189369+0000","last_undegraded":"2024-09-18T11:52:23.189369+0000","last_fullsized":"2024-09-18T11:52:23.189369+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24712438,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9892","reported_seq":10007,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.780828+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:22.780828+0000","last_peered":"2024-09-18T11:52:22.780828+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:22.780828+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23886483,"num_objects":2917,"num_object_clones":0,"num_object_copies":11668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9891,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":877}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9981","reported_seq":10073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.598275+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:23.598275+0000","last_peered":"2024-09-18T11:52:23.598275+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:23.598275+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24122614,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9923","reported_seq":10041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.284346+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:22.284346+0000","last_peered":"2024-09-18T11:52:22.284346+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:22.284346+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9923,"log_dups_size":0,"ondisk_log_size":9923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24343798,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9921,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9891","reported_seq":10008,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.270812+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:22.270812+0000","last_peered":"2024-09-18T11:52:22.270812+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:22.270812+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23715840,"num_objects":2895,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10084","reported_seq":10189,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.465262+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:23.465262+0000","last_peered":"2024-09-18T11:52:23.465262+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:23.465262+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016118,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10082,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":816}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9865","reported_seq":9971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.572449+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:23.572449+0000","last_peered":"2024-09-18T11:52:23.572449+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:23.572449+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24655094,"num_objects":3010,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9863,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":996}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10155","reported_seq":14218,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.473244+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:22.473244+0000","last_peered":"2024-09-18T11:52:22.473244+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:22.473244+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24742380,"num_objects":3021,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2003,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9693","reported_seq":9805,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.699260+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:22.699260+0000","last_peered":"2024-09-18T11:52:22.699260+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:22.699260+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9693,"log_dups_size":0,"ondisk_log_size":9693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762166,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9691,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":938}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9979","reported_seq":10098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.406448+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:22.406448+0000","last_peered":"2024-09-18T11:52:22.406448+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:22.406448+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9979,"log_dups_size":0,"ondisk_log_size":9979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25014129,"num_objects":3054,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10075","reported_seq":10164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.420184+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:22.420184+0000","last_peered":"2024-09-18T11:52:22.420184+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:22.420184+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24214139,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":872}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9967","reported_seq":10063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.635134+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:22.635134+0000","last_peered":"2024-09-18T11:52:22.635134+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:22.635134+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24574587,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9966,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10099","reported_seq":10222,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.391592+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:22.391592+0000","last_peered":"2024-09-18T11:52:22.391592+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:22.391592+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24499446,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10097,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":868}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9946","reported_seq":14233,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:23.083499+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:23.083499+0000","last_peered":"2024-09-18T11:52:23.083499+0000","last_clean":"2024-09-18T11:52:23.083499+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:23.083499+0000","last_undegraded":"2024-09-18T11:52:23.083499+0000","last_fullsized":"2024-09-18T11:52:23.083499+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24220918,"num_objects":2957,"num_object_clones":0,"num_object_copies":11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9766","reported_seq":9854,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.574794+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:23.574794+0000","last_peered":"2024-09-18T11:52:23.574794+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:23.574794+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9766,"log_dups_size":0,"ondisk_log_size":9766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24089846,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9764,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389612940,"num_objects":47566,"num_object_clones":0,"num_object_copies":190264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6501,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":778862592,"data_stored":778862592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159348,"ondisk_log_size":159348,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785912,"kb_used_data":191336,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93585928,"statfs":{"total":96636764160,"available":95831990272,"internally_reserved":0,"allocated":195928064,"data_stored":195752598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,4,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.638999999999999},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.839}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":10.404}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":9.1519999999999992},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.2270000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":780684,"kb_used_data":190788,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93591156,"statfs":{"total":96636764160,"available":95837343744,"internally_reserved":0,"allocated":195366912,"data_stored":195191446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":5.9939999999999998},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":6.0209999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":24},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":4.1600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.9580000000000002},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":8.4730000000000008}]}]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496696,"kb_used_data":190572,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93875144,"statfs":{"total":96636764160,"available":96128147456,"internally_reserved":0,"allocated":195145728,"data_stored":194970262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":10.249000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":0.40899999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.375},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.436}]}]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792932,"kb_used_data":190096,"kb_used_omap":4,"kb_used_meta":602811,"kb_avail":93578908,"statfs":{"total":96636764160,"available":95824801792,"internally_reserved":0,"allocated":194658304,"data_stored":194482838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617279041},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,12,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":16.521000000000001},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.9640000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.0270000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":6.9349999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":3.9329999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":6.9859999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":194101248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":194772992,"data_stored":194772992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194850816,"data_stored":194850816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":195137536,"data_stored":195137536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:25.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:25.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10193", 'reported_seq': 10291, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:22.283449+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:22.283449+0000', 'last_peered': '2024-09-18T11:52:22.283449+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:22.283449+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25042944, 'num_objects': 3057, 'num_object_clones': 0, 'num_object_copies': 12228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10193, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 954}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:25.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:25.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9892", 'reported_seq': 10007, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:22.780828+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:22.780828+0000', 'last_peered': '2024-09-18T11:52:22.780828+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:22.780828+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9892, 'log_dups_size': 0, 'ondisk_log_size': 9892, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23886483, 'num_objects': 2917, 'num_object_clones': 0, 'num_object_copies': 11668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2917, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9891, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 877}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:25.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:25.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9981", 'reported_seq': 10073, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:23.598275+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:23.598275+0000', 'last_peered': '2024-09-18T11:52:23.598275+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:23.598275+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9981, 'log_dups_size': 0, 'ondisk_log_size': 9981, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24122614, 'num_objects': 2945, 'num_object_clones': 0, 'num_object_copies': 11780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9979, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 844}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:25.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:25.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9923", 'reported_seq': 10041, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:22.284346+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:22.284346+0000', 'last_peered': '2024-09-18T11:52:22.284346+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:22.284346+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9923, 'log_dups_size': 0, 'ondisk_log_size': 9923, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24343798, 'num_objects': 2972, 'num_object_clones': 0, 'num_object_copies': 11888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9921, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 920}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:25.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:25.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9891", 'reported_seq': 10008, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:22.270812+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:22.270812+0000', 'last_peered': '2024-09-18T11:52:22.270812+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:22.270812+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9891, 'log_dups_size': 0, 'ondisk_log_size': 9891, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23715840, 'num_objects': 2895, 'num_object_clones': 0, 'num_object_copies': 11580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2895, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9891, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 776}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:25.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:25.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10084", 'reported_seq': 10189, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:23.465262+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:23.465262+0000', 'last_peered': '2024-09-18T11:52:23.465262+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:23.465262+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24016118, 'num_objects': 2932, 'num_object_clones': 0, 'num_object_copies': 11728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10082, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 816}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:25.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:25.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9865", 'reported_seq': 9971, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:23.572449+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:52:23.572449+0000', 'last_peered': '2024-09-18T11:52:23.572449+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:23.572449+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9865, 'log_dups_size': 0, 'ondisk_log_size': 9865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24655094, 'num_objects': 3010, 'num_object_clones': 0, 'num_object_copies': 12040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9863, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 996}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:25.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10155", 'reported_seq': 14218, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:22.473244+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:52:22.473244+0000', 'last_peered': '2024-09-18T11:52:22.473244+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:52:22.473244+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24742380, 'num_objects': 3021, 'num_object_clones': 0, 'num_object_copies': 12084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10152, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2003, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2883}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9693", 'reported_seq': 9805, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:22.699260+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:22.699260+0000', 'last_peered': '2024-09-18T11:52:22.699260+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:22.699260+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9693, 'log_dups_size': 0, 'ondisk_log_size': 9693, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23762166, 'num_objects': 2901, 'num_object_clones': 0, 'num_object_copies': 11604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9691, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 938}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9979", 'reported_seq': 10098, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:22.406448+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:22.406448+0000', 'last_peered': '2024-09-18T11:52:22.406448+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:22.406448+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9979, 'log_dups_size': 0, 'ondisk_log_size': 9979, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25014129, 'num_objects': 3054, 'num_object_clones': 0, 'num_object_copies': 12216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9976, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1020}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10075", 'reported_seq': 10164, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:22.420184+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:22.420184+0000', 'last_peered': '2024-09-18T11:52:22.420184+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:22.420184+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24214139, 'num_objects': 2956, 'num_object_clones': 0, 'num_object_copies': 11824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10074, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 872}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9967", 'reported_seq': 10063, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:22.635134+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:22.635134+0000', 'last_peered': '2024-09-18T11:52:22.635134+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:22.635134+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9967, 'log_dups_size': 0, 'ondisk_log_size': 9967, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24574587, 'num_objects': 3000, 'num_object_clones': 0, 'num_object_copies': 12000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9966, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:25.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10099", 'reported_seq': 10222, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:22.391592+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:22.391592+0000', 'last_peered': '2024-09-18T11:52:22.391592+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:22.391592+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24499446, 'num_objects': 2991, 'num_object_clones': 0, 'num_object_copies': 11964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10097, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 868}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:25.252 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:25.252 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9766", 'reported_seq': 9854, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:23.574794+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:23.574794+0000', 'last_peered': '2024-09-18T11:52:23.574794+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:23.574794+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9766, 'log_dups_size': 0, 'ondisk_log_size': 9766, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24089846, 'num_objects': 2941, 'num_object_clones': 0, 'num_object_copies': 11764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9764, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 939}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:25.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:25.943 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26955/190264 objects degraded (14.167%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":544,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47566,"data_bytes":389612940,"bytes_used":2924773376,"bytes_avail":383622283264,"bytes_total":386547056640,"degraded_objects":26955,"degraded_total":190264,"degraded_ratio":0.14167157213135434,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":187,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":254},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:25.964 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:25.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:26.468 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:26.468 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:26.487 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-18T11:52:26.026709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389612940,"num_objects":47566,"num_object_clones":0,"num_object_copies":190264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6501,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159348,"ondisk_log_size":159348,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2856224,"kb_used_data":762792,"kb_used_omap":14,"kb_used_meta":2093361,"kb_avail":374631136,"statfs":{"total":386547056640,"available":383622283264,"internally_reserved":0,"allocated":781099008,"data_stored":780397144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143601691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,7,8,26,18,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16580608,"num_objects":-2024,"num_object_clones":0,"num_object_copies":-8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2024,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004851"},"pg_stats":[{"pgid":"1.f","version":"35'10193","reported_seq":10291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.283449+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:22.283449+0000","last_peered":"2024-09-18T11:52:22.283449+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:22.283449+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25042944,"num_objects":3057,"num_object_clones":0,"num_object_copies":12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10193,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":954}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10039","reported_seq":14329,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:23.189369+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:23.189369+0000","last_peered":"2024-09-18T11:52:23.189369+0000","last_clean":"2024-09-18T11:52:23.189369+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:23.189369+0000","last_undegraded":"2024-09-18T11:52:23.189369+0000","last_fullsized":"2024-09-18T11:52:23.189369+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24712438,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9892","reported_seq":10007,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.780828+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:22.780828+0000","last_peered":"2024-09-18T11:52:22.780828+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:22.780828+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23886483,"num_objects":2917,"num_object_clones":0,"num_object_copies":11668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9891,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":877}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9981","reported_seq":10073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.598275+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:23.598275+0000","last_peered":"2024-09-18T11:52:23.598275+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:23.598275+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24122614,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9923","reported_seq":10041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.284346+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:22.284346+0000","last_peered":"2024-09-18T11:52:22.284346+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:22.284346+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9923,"log_dups_size":0,"ondisk_log_size":9923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24343798,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9921,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9891","reported_seq":10008,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.270812+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:22.270812+0000","last_peered":"2024-09-18T11:52:22.270812+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:22.270812+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23715840,"num_objects":2895,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10084","reported_seq":10189,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.465262+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:23.465262+0000","last_peered":"2024-09-18T11:52:23.465262+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:23.465262+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016118,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10082,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":816}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9865","reported_seq":9971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.572449+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:23.572449+0000","last_peered":"2024-09-18T11:52:23.572449+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:23.572449+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24655094,"num_objects":3010,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9863,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":996}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10155","reported_seq":14218,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.473244+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:22.473244+0000","last_peered":"2024-09-18T11:52:22.473244+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:22.473244+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24742380,"num_objects":3021,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2003,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9693","reported_seq":9805,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.699260+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:22.699260+0000","last_peered":"2024-09-18T11:52:22.699260+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:22.699260+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9693,"log_dups_size":0,"ondisk_log_size":9693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762166,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9691,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":938}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9979","reported_seq":10098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.406448+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:22.406448+0000","last_peered":"2024-09-18T11:52:22.406448+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:22.406448+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9979,"log_dups_size":0,"ondisk_log_size":9979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25014129,"num_objects":3054,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10075","reported_seq":10164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.420184+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:22.420184+0000","last_peered":"2024-09-18T11:52:22.420184+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:22.420184+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24214139,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":872}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9967","reported_seq":10063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.635134+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:22.635134+0000","last_peered":"2024-09-18T11:52:22.635134+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:22.635134+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24574587,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9966,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10099","reported_seq":10222,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:22.391592+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:22.391592+0000","last_peered":"2024-09-18T11:52:22.391592+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:22.391592+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24499446,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10097,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":868}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9946","reported_seq":14233,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:23.083499+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:23.083499+0000","last_peered":"2024-09-18T11:52:23.083499+0000","last_clean":"2024-09-18T11:52:23.083499+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:23.083499+0000","last_undegraded":"2024-09-18T11:52:23.083499+0000","last_fullsized":"2024-09-18T11:52:23.083499+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24220918,"num_objects":2957,"num_object_clones":0,"num_object_copies":11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9766","reported_seq":9854,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:23.574794+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:23.574794+0000","last_peered":"2024-09-18T11:52:23.574794+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:23.574794+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9766,"log_dups_size":0,"ondisk_log_size":9766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24089846,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9764,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389612940,"num_objects":47566,"num_object_clones":0,"num_object_copies":190264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6501,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":778862592,"data_stored":778862592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159348,"ondisk_log_size":159348,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785912,"kb_used_data":191336,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93585928,"statfs":{"total":96636764160,"available":95831990272,"internally_reserved":0,"allocated":195928064,"data_stored":195752598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,4,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.638999999999999},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.839}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":10.404}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":9.1519999999999992},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.2270000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":780684,"kb_used_data":190788,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93591156,"statfs":{"total":96636764160,"available":95837343744,"internally_reserved":0,"allocated":195366912,"data_stored":195191446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":5.9939999999999998},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":6.0209999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":24},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":4.1600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.9580000000000002},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":8.4730000000000008}]}]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496696,"kb_used_data":190572,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93875144,"statfs":{"total":96636764160,"available":96128147456,"internally_reserved":0,"allocated":195145728,"data_stored":194970262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":10.249000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":0.40899999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.375},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.436}]}]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792932,"kb_used_data":190096,"kb_used_omap":4,"kb_used_meta":602811,"kb_avail":93578908,"statfs":{"total":96636764160,"available":95824801792,"internally_reserved":0,"allocated":194658304,"data_stored":194482838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617279041},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,12,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":16.521000000000001},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.9640000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.0270000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":6.9349999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":3.9329999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":6.9859999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":194101248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":194772992,"data_stored":194772992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194850816,"data_stored":194850816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":195137536,"data_stored":195137536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:29.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:30.116 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:30.116 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:30.149 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-18T11:52:30.027626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":381117836,"num_objects":46529,"num_object_clones":0,"num_object_copies":186116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160558,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6557,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159885,"ondisk_log_size":159885,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2835800,"kb_used_data":746528,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374651560,"statfs":{"total":386547056640,"available":383643197440,"internally_reserved":0,"allocated":764444672,"data_stored":763742808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,2,4,5,5,7,13,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19914752,"num_objects":-2431,"num_object_clones":0,"num_object_copies":-9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004128"},"pg_stats":[{"pgid":"1.f","version":"35'10266","reported_seq":10364,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.489569+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:27.489569+0000","last_peered":"2024-09-18T11:52:27.489569+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:27.489569+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24444928,"num_objects":2984,"num_object_clones":0,"num_object_copies":11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10266,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10100","reported_seq":14390,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:28.371958+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:28.371958+0000","last_peered":"2024-09-18T11:52:28.371958+0000","last_clean":"2024-09-18T11:52:28.371958+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:28.371958+0000","last_undegraded":"2024-09-18T11:52:28.371958+0000","last_fullsized":"2024-09-18T11:52:28.371958+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24212726,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10098,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9944","reported_seq":10059,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.354348+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:27.354348+0000","last_peered":"2024-09-18T11:52:27.354348+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:27.354348+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9944,"log_dups_size":0,"ondisk_log_size":9944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23460499,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9943,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":825}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10054","reported_seq":10146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.528419+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:28.528419+0000","last_peered":"2024-09-18T11:52:28.528419+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:28.528419+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23524598,"num_objects":2872,"num_object_clones":0,"num_object_copies":11488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10052,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":771}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9973","reported_seq":10092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.401998+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:27.401998+0000","last_peered":"2024-09-18T11:52:27.401998+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:27.401998+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23934198,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9949","reported_seq":10066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.464788+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:27.464788+0000","last_peered":"2024-09-18T11:52:27.464788+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:27.464788+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23240704,"num_objects":2837,"num_object_clones":0,"num_object_copies":11348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10144","reported_seq":10249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.600694+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:28.600694+0000","last_peered":"2024-09-18T11:52:28.600694+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:28.600694+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23524598,"num_objects":2872,"num_object_clones":0,"num_object_copies":11488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":756}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9936","reported_seq":10043,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.621049+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:28.621049+0000","last_peered":"2024-09-18T11:52:28.621049+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:28.621049+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9936,"log_dups_size":0,"ondisk_log_size":9936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24073462,"num_objects":2939,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9934,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":925}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10221","reported_seq":14395,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.538238+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:27.538238+0000","last_peered":"2024-09-18T11:52:27.538238+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:27.538238+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24201708,"num_objects":2955,"num_object_clones":0,"num_object_copies":11820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10218,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9760","reported_seq":9872,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.707361+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:27.707361+0000","last_peered":"2024-09-18T11:52:27.707361+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:27.707361+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23213302,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":871}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10060","reported_seq":10177,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.433916+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:27.433916+0000","last_peered":"2024-09-18T11:52:27.433916+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:27.433916+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24350577,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10132","reported_seq":10223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.025050+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:27.025050+0000","last_peered":"2024-09-18T11:52:27.025050+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:27.025050+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23747195,"num_objects":2899,"num_object_clones":0,"num_object_copies":11596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10131,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10036","reported_seq":10132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.680033+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:27.680033+0000","last_peered":"2024-09-18T11:52:27.680033+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:27.680033+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24009339,"num_objects":2931,"num_object_clones":0,"num_object_copies":11724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10162","reported_seq":10284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.382311+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:27.382311+0000","last_peered":"2024-09-18T11:52:27.382311+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:27.382311+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23983350,"num_objects":2928,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10160,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10010","reported_seq":14297,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:28.352070+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:28.352070+0000","last_peered":"2024-09-18T11:52:28.352070+0000","last_clean":"2024-09-18T11:52:28.352070+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:28.352070+0000","last_undegraded":"2024-09-18T11:52:28.352070+0000","last_fullsized":"2024-09-18T11:52:28.352070+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23696630,"num_objects":2893,"num_object_clones":0,"num_object_copies":11572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9838","reported_seq":9926,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.519488+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:28.519488+0000","last_peered":"2024-09-18T11:52:28.519488+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:28.519488+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23500022,"num_objects":2869,"num_object_clones":0,"num_object_copies":11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":381117836,"num_objects":46529,"num_object_clones":0,"num_object_copies":186116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160558,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6557,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":761958400,"data_stored":761958400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159885,"ondisk_log_size":159885,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785948,"kb_used_data":187220,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93585892,"statfs":{"total":96636764160,"available":95831953408,"internally_reserved":0,"allocated":191713280,"data_stored":191537814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,0,0,0,2,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.1790000000000003},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.141}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":23.948},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.7670000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.0259999999999998},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":2.0609999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776580,"kb_used_data":186680,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93595260,"statfs":{"total":96636764160,"available":95841546240,"internally_reserved":0,"allocated":191160320,"data_stored":190984854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":13.747999999999999},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":13.785}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.2160000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.48299999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":13.816000000000001},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.853999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496800,"kb_used_data":186520,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93875040,"statfs":{"total":96636764160,"available":96128040960,"internally_reserved":0,"allocated":190996480,"data_stored":190821014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,3,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.92300000000000004},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.4630000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":15.874000000000001},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":15.805}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":6.5910000000000002},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.1179999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776472,"kb_used_data":186108,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93595368,"statfs":{"total":96636764160,"available":95841656832,"internally_reserved":0,"allocated":190574592,"data_stored":190399126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,0,0,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.159},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.686999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":8.5860000000000003},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":36.697000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":4.8680000000000003},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":14.292}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189890560,"data_stored":189890560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":190517248,"data_stored":190517248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190648320,"data_stored":190648320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":190902272,"data_stored":190902272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:30.151 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:30.752 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:30.753 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:30.783 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-18T11:52:30.027626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":381117836,"num_objects":46529,"num_object_clones":0,"num_object_copies":186116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160558,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6557,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159885,"ondisk_log_size":159885,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2835800,"kb_used_data":746528,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374651560,"statfs":{"total":386547056640,"available":383643197440,"internally_reserved":0,"allocated":764444672,"data_stored":763742808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,2,4,5,5,7,13,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19914752,"num_objects":-2431,"num_object_clones":0,"num_object_copies":-9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004128"},"pg_stats":[{"pgid":"1.f","version":"35'10266","reported_seq":10364,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.489569+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:27.489569+0000","last_peered":"2024-09-18T11:52:27.489569+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:27.489569+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24444928,"num_objects":2984,"num_object_clones":0,"num_object_copies":11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10266,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10100","reported_seq":14390,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:28.371958+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:28.371958+0000","last_peered":"2024-09-18T11:52:28.371958+0000","last_clean":"2024-09-18T11:52:28.371958+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:28.371958+0000","last_undegraded":"2024-09-18T11:52:28.371958+0000","last_fullsized":"2024-09-18T11:52:28.371958+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24212726,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10098,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9944","reported_seq":10059,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.354348+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:27.354348+0000","last_peered":"2024-09-18T11:52:27.354348+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:27.354348+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9944,"log_dups_size":0,"ondisk_log_size":9944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23460499,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9943,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":825}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10054","reported_seq":10146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.528419+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:28.528419+0000","last_peered":"2024-09-18T11:52:28.528419+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:28.528419+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23524598,"num_objects":2872,"num_object_clones":0,"num_object_copies":11488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10052,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":771}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9973","reported_seq":10092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.401998+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:27.401998+0000","last_peered":"2024-09-18T11:52:27.401998+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:27.401998+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23934198,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9949","reported_seq":10066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.464788+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:27.464788+0000","last_peered":"2024-09-18T11:52:27.464788+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:27.464788+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23240704,"num_objects":2837,"num_object_clones":0,"num_object_copies":11348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10144","reported_seq":10249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.600694+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:28.600694+0000","last_peered":"2024-09-18T11:52:28.600694+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:28.600694+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23524598,"num_objects":2872,"num_object_clones":0,"num_object_copies":11488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":756}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9936","reported_seq":10043,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.621049+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:28.621049+0000","last_peered":"2024-09-18T11:52:28.621049+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:28.621049+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9936,"log_dups_size":0,"ondisk_log_size":9936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24073462,"num_objects":2939,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9934,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":925}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10221","reported_seq":14395,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.538238+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:27.538238+0000","last_peered":"2024-09-18T11:52:27.538238+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:27.538238+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24201708,"num_objects":2955,"num_object_clones":0,"num_object_copies":11820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10218,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9760","reported_seq":9872,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.707361+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:27.707361+0000","last_peered":"2024-09-18T11:52:27.707361+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:27.707361+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23213302,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":871}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10060","reported_seq":10177,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.433916+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:27.433916+0000","last_peered":"2024-09-18T11:52:27.433916+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:27.433916+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24350577,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10132","reported_seq":10223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.025050+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:27.025050+0000","last_peered":"2024-09-18T11:52:27.025050+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:27.025050+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23747195,"num_objects":2899,"num_object_clones":0,"num_object_copies":11596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10131,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10036","reported_seq":10132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.680033+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:27.680033+0000","last_peered":"2024-09-18T11:52:27.680033+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:27.680033+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24009339,"num_objects":2931,"num_object_clones":0,"num_object_copies":11724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10162","reported_seq":10284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.382311+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:27.382311+0000","last_peered":"2024-09-18T11:52:27.382311+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:27.382311+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23983350,"num_objects":2928,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10160,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10010","reported_seq":14297,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:28.352070+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:28.352070+0000","last_peered":"2024-09-18T11:52:28.352070+0000","last_clean":"2024-09-18T11:52:28.352070+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:28.352070+0000","last_undegraded":"2024-09-18T11:52:28.352070+0000","last_fullsized":"2024-09-18T11:52:28.352070+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23696630,"num_objects":2893,"num_object_clones":0,"num_object_copies":11572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9838","reported_seq":9926,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.519488+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:28.519488+0000","last_peered":"2024-09-18T11:52:28.519488+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:28.519488+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23500022,"num_objects":2869,"num_object_clones":0,"num_object_copies":11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":381117836,"num_objects":46529,"num_object_clones":0,"num_object_copies":186116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160558,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6557,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":761958400,"data_stored":761958400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159885,"ondisk_log_size":159885,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785948,"kb_used_data":187220,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93585892,"statfs":{"total":96636764160,"available":95831953408,"internally_reserved":0,"allocated":191713280,"data_stored":191537814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,0,0,0,2,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.1790000000000003},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.141}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":23.948},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.7670000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.0259999999999998},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":2.0609999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776580,"kb_used_data":186680,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93595260,"statfs":{"total":96636764160,"available":95841546240,"internally_reserved":0,"allocated":191160320,"data_stored":190984854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":13.747999999999999},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":13.785}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.2160000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.48299999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":13.816000000000001},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.853999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496800,"kb_used_data":186520,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93875040,"statfs":{"total":96636764160,"available":96128040960,"internally_reserved":0,"allocated":190996480,"data_stored":190821014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,3,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.92300000000000004},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.4630000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":15.874000000000001},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":15.805}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":6.5910000000000002},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.1179999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776472,"kb_used_data":186108,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93595368,"statfs":{"total":96636764160,"available":95841656832,"internally_reserved":0,"allocated":190574592,"data_stored":190399126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,0,0,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.159},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.686999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":8.5860000000000003},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":36.697000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":4.8680000000000003},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":14.292}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189890560,"data_stored":189890560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":190517248,"data_stored":190517248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190648320,"data_stored":190648320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":190902272,"data_stored":190902272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:30.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:30.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10266", 'reported_seq': 10364, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:27.489569+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:27.489569+0000', 'last_peered': '2024-09-18T11:52:27.489569+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:27.489569+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24444928, 'num_objects': 2984, 'num_object_clones': 0, 'num_object_copies': 11936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10266, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 881}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:30.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:30.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9944", 'reported_seq': 10059, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:27.354348+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:27.354348+0000', 'last_peered': '2024-09-18T11:52:27.354348+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:27.354348+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9944, 'log_dups_size': 0, 'ondisk_log_size': 9944, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23460499, 'num_objects': 2865, 'num_object_clones': 0, 'num_object_copies': 11460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2865, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9943, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 825}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:30.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:30.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10054", 'reported_seq': 10146, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:28.528419+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:28.528419+0000', 'last_peered': '2024-09-18T11:52:28.528419+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:28.528419+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23524598, 'num_objects': 2872, 'num_object_clones': 0, 'num_object_copies': 11488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10052, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 771}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:30.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:30.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9973", 'reported_seq': 10092, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:27.401998+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:27.401998+0000', 'last_peered': '2024-09-18T11:52:27.401998+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:27.401998+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9973, 'log_dups_size': 0, 'ondisk_log_size': 9973, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23934198, 'num_objects': 2922, 'num_object_clones': 0, 'num_object_copies': 11688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9971, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 870}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:30.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:30.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9949", 'reported_seq': 10066, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:27.464788+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:27.464788+0000', 'last_peered': '2024-09-18T11:52:27.464788+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:27.464788+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9949, 'log_dups_size': 0, 'ondisk_log_size': 9949, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23240704, 'num_objects': 2837, 'num_object_clones': 0, 'num_object_copies': 11348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9949, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 718}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10144", 'reported_seq': 10249, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:28.600694+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:28.600694+0000', 'last_peered': '2024-09-18T11:52:28.600694+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:28.600694+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23524598, 'num_objects': 2872, 'num_object_clones': 0, 'num_object_copies': 11488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10142, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 756}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9936", 'reported_seq': 10043, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:28.621049+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:52:28.621049+0000', 'last_peered': '2024-09-18T11:52:28.621049+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:28.621049+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9936, 'log_dups_size': 0, 'ondisk_log_size': 9936, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24073462, 'num_objects': 2939, 'num_object_clones': 0, 'num_object_copies': 11756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9934, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 925}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10221", 'reported_seq': 14395, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:27.538238+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:52:27.538238+0000', 'last_peered': '2024-09-18T11:52:27.538238+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:52:27.538238+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24201708, 'num_objects': 2955, 'num_object_clones': 0, 'num_object_copies': 11820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 82, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10218, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2059, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2873}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9760", 'reported_seq': 9872, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:27.707361+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:27.707361+0000', 'last_peered': '2024-09-18T11:52:27.707361+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:27.707361+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9760, 'log_dups_size': 0, 'ondisk_log_size': 9760, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23213302, 'num_objects': 2834, 'num_object_clones': 0, 'num_object_copies': 11336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9758, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 871}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10060", 'reported_seq': 10177, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:27.433916+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:27.433916+0000', 'last_peered': '2024-09-18T11:52:27.433916+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:27.433916+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24350577, 'num_objects': 2973, 'num_object_clones': 0, 'num_object_copies': 11892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10057, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 939}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:30.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:30.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10132", 'reported_seq': 10223, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:27.025050+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:27.025050+0000', 'last_peered': '2024-09-18T11:52:27.025050+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:27.025050+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23747195, 'num_objects': 2899, 'num_object_clones': 0, 'num_object_copies': 11596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2899, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10131, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:30.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:30.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10036", 'reported_seq': 10132, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:27.680033+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:27.680033+0000', 'last_peered': '2024-09-18T11:52:27.680033+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:27.680033+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24009339, 'num_objects': 2931, 'num_object_clones': 0, 'num_object_copies': 11724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10035, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 885}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:30.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:30.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10162", 'reported_seq': 10284, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:27.382311+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:27.382311+0000', 'last_peered': '2024-09-18T11:52:27.382311+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:27.382311+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23983350, 'num_objects': 2928, 'num_object_clones': 0, 'num_object_copies': 11712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10160, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 805}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:30.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:30.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9838", 'reported_seq': 9926, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:28.519488+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:28.519488+0000', 'last_peered': '2024-09-18T11:52:28.519488+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:28.519488+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9838, 'log_dups_size': 0, 'ondisk_log_size': 9838, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23500022, 'num_objects': 2869, 'num_object_clones': 0, 'num_object_copies': 11476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9836, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 867}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:30.787 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:31.460 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:31.460 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26899/186116 objects degraded (14.453%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":549,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":46529,"data_bytes":381117836,"bytes_used":2903859200,"bytes_avail":383643197440,"bytes_total":386547056640,"degraded_objects":26899,"degraded_total":186116,"degraded_ratio":0.14452814373831374,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":202},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:31.485 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:31.486 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:31.944 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:31.945 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:31.981 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-18T11:52:30.027626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":381117836,"num_objects":46529,"num_object_clones":0,"num_object_copies":186116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160558,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6557,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159885,"ondisk_log_size":159885,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2835800,"kb_used_data":746528,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374651560,"statfs":{"total":386547056640,"available":383643197440,"internally_reserved":0,"allocated":764444672,"data_stored":763742808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,2,4,5,5,7,13,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19914752,"num_objects":-2431,"num_object_clones":0,"num_object_copies":-9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004128"},"pg_stats":[{"pgid":"1.f","version":"35'10266","reported_seq":10364,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.489569+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:27.489569+0000","last_peered":"2024-09-18T11:52:27.489569+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:27.489569+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24444928,"num_objects":2984,"num_object_clones":0,"num_object_copies":11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10266,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10100","reported_seq":14390,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:28.371958+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:28.371958+0000","last_peered":"2024-09-18T11:52:28.371958+0000","last_clean":"2024-09-18T11:52:28.371958+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:28.371958+0000","last_undegraded":"2024-09-18T11:52:28.371958+0000","last_fullsized":"2024-09-18T11:52:28.371958+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24212726,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10098,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9944","reported_seq":10059,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.354348+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:27.354348+0000","last_peered":"2024-09-18T11:52:27.354348+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:27.354348+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9944,"log_dups_size":0,"ondisk_log_size":9944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23460499,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9943,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":825}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10054","reported_seq":10146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.528419+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:28.528419+0000","last_peered":"2024-09-18T11:52:28.528419+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:28.528419+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23524598,"num_objects":2872,"num_object_clones":0,"num_object_copies":11488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10052,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":771}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9973","reported_seq":10092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.401998+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:27.401998+0000","last_peered":"2024-09-18T11:52:27.401998+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:27.401998+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23934198,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9949","reported_seq":10066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.464788+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:27.464788+0000","last_peered":"2024-09-18T11:52:27.464788+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:27.464788+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23240704,"num_objects":2837,"num_object_clones":0,"num_object_copies":11348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10144","reported_seq":10249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.600694+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:28.600694+0000","last_peered":"2024-09-18T11:52:28.600694+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:28.600694+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23524598,"num_objects":2872,"num_object_clones":0,"num_object_copies":11488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":756}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9936","reported_seq":10043,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.621049+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:28.621049+0000","last_peered":"2024-09-18T11:52:28.621049+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:28.621049+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9936,"log_dups_size":0,"ondisk_log_size":9936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24073462,"num_objects":2939,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9934,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":925}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10221","reported_seq":14395,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.538238+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:27.538238+0000","last_peered":"2024-09-18T11:52:27.538238+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:27.538238+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24201708,"num_objects":2955,"num_object_clones":0,"num_object_copies":11820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10218,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9760","reported_seq":9872,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.707361+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:27.707361+0000","last_peered":"2024-09-18T11:52:27.707361+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:27.707361+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23213302,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":871}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10060","reported_seq":10177,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.433916+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:27.433916+0000","last_peered":"2024-09-18T11:52:27.433916+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:27.433916+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24350577,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10132","reported_seq":10223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.025050+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:27.025050+0000","last_peered":"2024-09-18T11:52:27.025050+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:27.025050+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23747195,"num_objects":2899,"num_object_clones":0,"num_object_copies":11596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10131,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10036","reported_seq":10132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.680033+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:27.680033+0000","last_peered":"2024-09-18T11:52:27.680033+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:27.680033+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24009339,"num_objects":2931,"num_object_clones":0,"num_object_copies":11724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10162","reported_seq":10284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:27.382311+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:27.382311+0000","last_peered":"2024-09-18T11:52:27.382311+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:27.382311+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23983350,"num_objects":2928,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10160,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10010","reported_seq":14297,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:28.352070+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:28.352070+0000","last_peered":"2024-09-18T11:52:28.352070+0000","last_clean":"2024-09-18T11:52:28.352070+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:28.352070+0000","last_undegraded":"2024-09-18T11:52:28.352070+0000","last_fullsized":"2024-09-18T11:52:28.352070+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23696630,"num_objects":2893,"num_object_clones":0,"num_object_copies":11572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9838","reported_seq":9926,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:28.519488+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:28.519488+0000","last_peered":"2024-09-18T11:52:28.519488+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:28.519488+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23500022,"num_objects":2869,"num_object_clones":0,"num_object_copies":11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":381117836,"num_objects":46529,"num_object_clones":0,"num_object_copies":186116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160558,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6557,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":761958400,"data_stored":761958400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159885,"ondisk_log_size":159885,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785948,"kb_used_data":187220,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93585892,"statfs":{"total":96636764160,"available":95831953408,"internally_reserved":0,"allocated":191713280,"data_stored":191537814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,0,0,0,2,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.1790000000000003},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.141}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":23.948},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.7670000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.0259999999999998},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":2.0609999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776580,"kb_used_data":186680,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93595260,"statfs":{"total":96636764160,"available":95841546240,"internally_reserved":0,"allocated":191160320,"data_stored":190984854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":13.747999999999999},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":13.785}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.2160000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.48299999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":13.816000000000001},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.853999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496800,"kb_used_data":186520,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93875040,"statfs":{"total":96636764160,"available":96128040960,"internally_reserved":0,"allocated":190996480,"data_stored":190821014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,3,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.92300000000000004},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.4630000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":15.874000000000001},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":15.805}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":6.5910000000000002},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":2.1179999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776472,"kb_used_data":186108,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93595368,"statfs":{"total":96636764160,"available":95841656832,"internally_reserved":0,"allocated":190574592,"data_stored":190399126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,0,0,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.159},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.686999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":8.5860000000000003},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":36.697000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":4.8680000000000003},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":14.292}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189890560,"data_stored":189890560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":190517248,"data_stored":190517248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190648320,"data_stored":190648320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":190902272,"data_stored":190902272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:34.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:35.578 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:35.579 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:35.599 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-18T11:52:34.028521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":372901260,"num_objects":45526,"num_object_clones":0,"num_object_copies":182104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161561,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160188,"ondisk_log_size":160188,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815548,"kb_used_data":730432,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374671812,"statfs":{"total":386547056640,"available":383663935488,"internally_reserved":0,"allocated":747962368,"data_stored":747260504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,9,14,17,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24879104,"num_objects":-3037,"num_object_clones":0,"num_object_copies":-12148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3037,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003334"},"pg_stats":[{"pgid":"1.f","version":"35'10341","reported_seq":10440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.621322+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:32.621322+0000","last_peered":"2024-09-18T11:52:32.621322+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:32.621322+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23830528,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10155","reported_seq":14445,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:33.266246+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:33.266246+0000","last_peered":"2024-09-18T11:52:33.266246+0000","last_clean":"2024-09-18T11:52:33.266246+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:33.266246+0000","last_undegraded":"2024-09-18T11:52:33.266246+0000","last_fullsized":"2024-09-18T11:52:33.266246+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762166,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10000","reported_seq":10115,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.719621+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:32.719621+0000","last_peered":"2024-09-18T11:52:32.719621+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:32.719621+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10000,"log_dups_size":0,"ondisk_log_size":10000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23001747,"num_objects":2809,"num_object_clones":0,"num_object_copies":11236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9999,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":769}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10113","reported_seq":10205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.385082+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:33.385082+0000","last_peered":"2024-09-18T11:52:33.385082+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:33.385082+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041270,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10111,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10037","reported_seq":10154,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.348381+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:32.348381+0000","last_peered":"2024-09-18T11:52:32.348381+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:32.348381+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409910,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10017","reported_seq":10134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.492282+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:32.492282+0000","last_peered":"2024-09-18T11:52:32.492282+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:32.492282+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22683648,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10017,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10206","reported_seq":10311,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.562725+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:33.562725+0000","last_peered":"2024-09-18T11:52:33.562725+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:33.562725+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23016694,"num_objects":2810,"num_object_clones":0,"num_object_copies":11240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":694}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10001","reported_seq":10108,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.477995+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:33.477995+0000","last_peered":"2024-09-18T11:52:33.477995+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:33.477995+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23540982,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10282","reported_seq":14563,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.589447+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:32.589447+0000","last_peered":"2024-09-18T11:52:32.589447+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:32.589447+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23701996,"num_objects":2894,"num_object_clones":0,"num_object_copies":11576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10279,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9821","reported_seq":9933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.718480+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:32.718480+0000","last_peered":"2024-09-18T11:52:32.718480+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:32.718480+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22713590,"num_objects":2773,"num_object_clones":0,"num_object_copies":11092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9819,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":810}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10124","reported_seq":10243,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.500230+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:32.500230+0000","last_peered":"2024-09-18T11:52:32.500230+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:32.500230+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23826289,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10121,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10195","reported_seq":10286,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.454876+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:32.454876+0000","last_peered":"2024-09-18T11:52:32.454876+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:32.454876+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23231099,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10194,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10101","reported_seq":10197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.612722+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:32.612722+0000","last_peered":"2024-09-18T11:52:32.612722+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:32.612722+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23476859,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10100,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10219","reported_seq":10342,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.571735+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:32.571735+0000","last_peered":"2024-09-18T11:52:32.571735+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:32.571735+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23516406,"num_objects":2871,"num_object_clones":0,"num_object_copies":11484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10217,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10066","reported_seq":14353,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:33.451140+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:33.451140+0000","last_peered":"2024-09-18T11:52:33.451140+0000","last_clean":"2024-09-18T11:52:33.451140+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:33.451140+0000","last_undegraded":"2024-09-18T11:52:33.451140+0000","last_fullsized":"2024-09-18T11:52:33.451140+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23237878,"num_objects":2837,"num_object_clones":0,"num_object_copies":11348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10064,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9910","reported_seq":9998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.716319+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:33.716319+0000","last_peered":"2024-09-18T11:52:33.716319+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:33.716319+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22910198,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9908,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":372901260,"num_objects":45526,"num_object_clones":0,"num_object_copies":182104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161561,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":745369600,"data_stored":745369600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160188,"ondisk_log_size":160188,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769496,"kb_used_data":183232,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93602344,"statfs":{"total":96636764160,"available":95848800256,"internally_reserved":0,"allocated":187629568,"data_stored":187454102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.3150000000000004},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.22500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.2040000000000006},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":1.966}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.4809999999999999},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.95699999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776748,"kb_used_data":182692,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93595092,"statfs":{"total":96636764160,"available":95841374208,"internally_reserved":0,"allocated":187076608,"data_stored":186901142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":8.9779999999999998},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":9.0060000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":3.3660000000000001},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.27300000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.9459999999999997},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.776999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496912,"kb_used_data":182468,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93874928,"statfs":{"total":96636764160,"available":96127926272,"internally_reserved":0,"allocated":186847232,"data_stored":186671766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.2999999999999998},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.4909999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":9.6799999999999997},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":3.2669999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.4239999999999999},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":17.300999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772392,"kb_used_data":182040,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93599448,"statfs":{"total":96636764160,"available":95845834752,"internally_reserved":0,"allocated":186408960,"data_stored":186233494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":6.4740000000000002},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.7370000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.5069999999999997},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":6.7949999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":6.5030000000000001},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":6.7720000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185794560,"data_stored":185794560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186302464,"data_stored":186302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186535936,"data_stored":186535936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":186736640,"data_stored":186736640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:35.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:36.266 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:36.294 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-18T11:52:36.029006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":372901260,"num_objects":45526,"num_object_clones":0,"num_object_copies":182104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161561,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160188,"ondisk_log_size":160188,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815548,"kb_used_data":730432,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374671812,"statfs":{"total":386547056640,"available":383663935488,"internally_reserved":0,"allocated":747962368,"data_stored":747260504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,9,14,17,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16711680,"num_objects":-2040,"num_object_clones":0,"num_object_copies":-8160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2040,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003036"},"pg_stats":[{"pgid":"1.f","version":"35'10341","reported_seq":10440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.621322+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:32.621322+0000","last_peered":"2024-09-18T11:52:32.621322+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:32.621322+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23830528,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10155","reported_seq":14445,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:33.266246+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:33.266246+0000","last_peered":"2024-09-18T11:52:33.266246+0000","last_clean":"2024-09-18T11:52:33.266246+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:33.266246+0000","last_undegraded":"2024-09-18T11:52:33.266246+0000","last_fullsized":"2024-09-18T11:52:33.266246+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762166,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10000","reported_seq":10115,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.719621+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:32.719621+0000","last_peered":"2024-09-18T11:52:32.719621+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:32.719621+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10000,"log_dups_size":0,"ondisk_log_size":10000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23001747,"num_objects":2809,"num_object_clones":0,"num_object_copies":11236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9999,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":769}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10113","reported_seq":10205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.385082+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:33.385082+0000","last_peered":"2024-09-18T11:52:33.385082+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:33.385082+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041270,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10111,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10037","reported_seq":10154,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.348381+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:32.348381+0000","last_peered":"2024-09-18T11:52:32.348381+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:32.348381+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409910,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10017","reported_seq":10134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.492282+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:32.492282+0000","last_peered":"2024-09-18T11:52:32.492282+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:32.492282+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22683648,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10017,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10206","reported_seq":10311,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.562725+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:33.562725+0000","last_peered":"2024-09-18T11:52:33.562725+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:33.562725+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23016694,"num_objects":2810,"num_object_clones":0,"num_object_copies":11240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":694}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10001","reported_seq":10108,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.477995+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:33.477995+0000","last_peered":"2024-09-18T11:52:33.477995+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:33.477995+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23540982,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10282","reported_seq":14563,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.589447+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:32.589447+0000","last_peered":"2024-09-18T11:52:32.589447+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:32.589447+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23701996,"num_objects":2894,"num_object_clones":0,"num_object_copies":11576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10279,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9821","reported_seq":9933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.718480+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:32.718480+0000","last_peered":"2024-09-18T11:52:32.718480+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:32.718480+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22713590,"num_objects":2773,"num_object_clones":0,"num_object_copies":11092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9819,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":810}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10124","reported_seq":10243,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.500230+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:32.500230+0000","last_peered":"2024-09-18T11:52:32.500230+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:32.500230+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23826289,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10121,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10195","reported_seq":10286,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.454876+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:32.454876+0000","last_peered":"2024-09-18T11:52:32.454876+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:32.454876+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23231099,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10194,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10101","reported_seq":10197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.612722+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:32.612722+0000","last_peered":"2024-09-18T11:52:32.612722+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:32.612722+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23476859,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10100,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10219","reported_seq":10342,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.571735+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:32.571735+0000","last_peered":"2024-09-18T11:52:32.571735+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:32.571735+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23516406,"num_objects":2871,"num_object_clones":0,"num_object_copies":11484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10217,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10066","reported_seq":14353,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:33.451140+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:33.451140+0000","last_peered":"2024-09-18T11:52:33.451140+0000","last_clean":"2024-09-18T11:52:33.451140+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:33.451140+0000","last_undegraded":"2024-09-18T11:52:33.451140+0000","last_fullsized":"2024-09-18T11:52:33.451140+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23237878,"num_objects":2837,"num_object_clones":0,"num_object_copies":11348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10064,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9910","reported_seq":9998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.716319+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:33.716319+0000","last_peered":"2024-09-18T11:52:33.716319+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:33.716319+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22910198,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9908,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":372901260,"num_objects":45526,"num_object_clones":0,"num_object_copies":182104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161561,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":745369600,"data_stored":745369600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160188,"ondisk_log_size":160188,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769496,"kb_used_data":183232,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93602344,"statfs":{"total":96636764160,"available":95848800256,"internally_reserved":0,"allocated":187629568,"data_stored":187454102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.3150000000000004},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.22500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.2040000000000006},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":1.966}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.4809999999999999},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.95699999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776748,"kb_used_data":182692,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93595092,"statfs":{"total":96636764160,"available":95841374208,"internally_reserved":0,"allocated":187076608,"data_stored":186901142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":8.9779999999999998},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":9.0060000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":3.3660000000000001},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.27300000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.9459999999999997},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.776999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496912,"kb_used_data":182468,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93874928,"statfs":{"total":96636764160,"available":96127926272,"internally_reserved":0,"allocated":186847232,"data_stored":186671766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.2999999999999998},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.4909999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":9.6799999999999997},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":3.2669999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.4239999999999999},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":17.300999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772392,"kb_used_data":182040,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93599448,"statfs":{"total":96636764160,"available":95845834752,"internally_reserved":0,"allocated":186408960,"data_stored":186233494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":6.4740000000000002},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.7370000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.5069999999999997},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":6.7949999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":6.5030000000000001},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":6.7720000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185794560,"data_stored":185794560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186302464,"data_stored":186302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186535936,"data_stored":186535936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":186736640,"data_stored":186736640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10341", 'reported_seq': 10440, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:32.621322+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:32.621322+0000', 'last_peered': '2024-09-18T11:52:32.621322+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:32.621322+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23830528, 'num_objects': 2909, 'num_object_clones': 0, 'num_object_copies': 11636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10341, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 806}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10000", 'reported_seq': 10115, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:32.719621+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:32.719621+0000', 'last_peered': '2024-09-18T11:52:32.719621+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:32.719621+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10000, 'log_dups_size': 0, 'ondisk_log_size': 10000, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23001747, 'num_objects': 2809, 'num_object_clones': 0, 'num_object_copies': 11236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2809, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9999, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 769}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10113", 'reported_seq': 10205, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:33.385082+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:33.385082+0000', 'last_peered': '2024-09-18T11:52:33.385082+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:33.385082+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23041270, 'num_objects': 2813, 'num_object_clones': 0, 'num_object_copies': 11252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10111, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 712}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10037", 'reported_seq': 10154, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:32.348381+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:32.348381+0000', 'last_peered': '2024-09-18T11:52:32.348381+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:32.348381+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23409910, 'num_objects': 2858, 'num_object_clones': 0, 'num_object_copies': 11432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10035, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 806}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10017", 'reported_seq': 10134, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:32.492282+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:32.492282+0000', 'last_peered': '2024-09-18T11:52:32.492282+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:32.492282+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22683648, 'num_objects': 2769, 'num_object_clones': 0, 'num_object_copies': 11076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10017, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 650}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10206", 'reported_seq': 10311, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:33.562725+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:33.562725+0000', 'last_peered': '2024-09-18T11:52:33.562725+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:33.562725+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23016694, 'num_objects': 2810, 'num_object_clones': 0, 'num_object_copies': 11240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10204, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 694}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10001", 'reported_seq': 10108, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:33.477995+0000', 'last_change': '2024-09-18T11:44:47.413703+0000', 'last_active': '2024-09-18T11:52:33.477995+0000', 'last_peered': '2024-09-18T11:52:33.477995+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:33.477995+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23540982, 'num_objects': 2874, 'num_object_clones': 0, 'num_object_copies': 11496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9999, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:36.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-18T11:52:36.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10282", 'reported_seq': 14563, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:32.589447+0000', 'last_change': '2024-09-18T11:49:40.464642+0000', 'last_active': '2024-09-18T11:52:32.589447+0000', 'last_peered': '2024-09-18T11:52:32.589447+0000', 'last_clean': '2024-09-18T11:43:44.501034+0000', 'last_became_active': '2024-09-18T11:44:47.351354+0000', 'last_became_peered': '2024-09-18T11:44:47.351354+0000', 'last_unstale': '2024-09-18T11:52:32.589447+0000', 'last_undegraded': '2024-09-18T11:44:47.165795+0000', 'last_fullsized': '2024-09-18T11:44:47.117426+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23701996, 'num_objects': 2894, 'num_object_clones': 0, 'num_object_copies': 11576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 29, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2894, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10279, 'num_write_kb': 52689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2112, 'num_bytes_recovered': 5251072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2865}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:36.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:36.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9821", 'reported_seq': 9933, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:32.718480+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:32.718480+0000', 'last_peered': '2024-09-18T11:52:32.718480+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:32.718480+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9821, 'log_dups_size': 0, 'ondisk_log_size': 9821, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22713590, 'num_objects': 2773, 'num_object_clones': 0, 'num_object_copies': 11092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9819, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 810}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:36.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:36.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10124", 'reported_seq': 10243, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:32.500230+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:32.500230+0000', 'last_peered': '2024-09-18T11:52:32.500230+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:32.500230+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23826289, 'num_objects': 2909, 'num_object_clones': 0, 'num_object_copies': 11636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10121, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 875}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:36.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:36.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10195", 'reported_seq': 10286, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:32.454876+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:32.454876+0000', 'last_peered': '2024-09-18T11:52:32.454876+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:32.454876+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23231099, 'num_objects': 2836, 'num_object_clones': 0, 'num_object_copies': 11344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10194, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:36.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:36.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10101", 'reported_seq': 10197, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:32.612722+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:32.612722+0000', 'last_peered': '2024-09-18T11:52:32.612722+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:32.612722+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23476859, 'num_objects': 2866, 'num_object_clones': 0, 'num_object_copies': 11464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10100, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 820}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:36.298 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:36.298 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10219", 'reported_seq': 10342, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:32.571735+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:32.571735+0000', 'last_peered': '2024-09-18T11:52:32.571735+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:32.571735+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23516406, 'num_objects': 2871, 'num_object_clones': 0, 'num_object_copies': 11484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10217, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 748}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:36.298 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:36.298 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9910", 'reported_seq': 9998, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:33.716319+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:33.716319+0000', 'last_peered': '2024-09-18T11:52:33.716319+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:33.716319+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9910, 'log_dups_size': 0, 'ondisk_log_size': 9910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22910198, 'num_objects': 2797, 'num_object_clones': 0, 'num_object_copies': 11188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2797, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9908, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 795}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:36.298 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:36.985 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:36.986 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26846/182104 objects degraded (14.742%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":555,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45526,"data_bytes":372901260,"bytes_used":2883121152,"bytes_avail":383663935488,"bytes_total":386547056640,"degraded_objects":26846,"degraded_total":182104,"degraded_ratio":0.14742125378904361,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":169},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:37.020 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:37.021 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:37.606 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:37.627 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-18T11:52:37.271177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":372901260,"num_objects":45526,"num_object_clones":0,"num_object_copies":182104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161561,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160188,"ondisk_log_size":160188,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815548,"kb_used_data":730432,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374671812,"statfs":{"total":386547056640,"available":383663935488,"internally_reserved":0,"allocated":747962368,"data_stored":747260504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,9,14,17,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11501568,"num_objects":-1404,"num_object_clones":0,"num_object_copies":-5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-53,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1404,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.244034"},"pg_stats":[{"pgid":"1.f","version":"35'10341","reported_seq":10440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.621322+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:32.621322+0000","last_peered":"2024-09-18T11:52:32.621322+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:32.621322+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23830528,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10155","reported_seq":14445,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:33.266246+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:33.266246+0000","last_peered":"2024-09-18T11:52:33.266246+0000","last_clean":"2024-09-18T11:52:33.266246+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:33.266246+0000","last_undegraded":"2024-09-18T11:52:33.266246+0000","last_fullsized":"2024-09-18T11:52:33.266246+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762166,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10000","reported_seq":10115,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.719621+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:32.719621+0000","last_peered":"2024-09-18T11:52:32.719621+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:32.719621+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10000,"log_dups_size":0,"ondisk_log_size":10000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23001747,"num_objects":2809,"num_object_clones":0,"num_object_copies":11236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9999,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":769}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10113","reported_seq":10205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.385082+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:33.385082+0000","last_peered":"2024-09-18T11:52:33.385082+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:33.385082+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041270,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10111,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10037","reported_seq":10154,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.348381+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:32.348381+0000","last_peered":"2024-09-18T11:52:32.348381+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:32.348381+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409910,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10017","reported_seq":10134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.492282+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:32.492282+0000","last_peered":"2024-09-18T11:52:32.492282+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:32.492282+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22683648,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10017,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10206","reported_seq":10311,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.562725+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:33.562725+0000","last_peered":"2024-09-18T11:52:33.562725+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:33.562725+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23016694,"num_objects":2810,"num_object_clones":0,"num_object_copies":11240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":694}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10001","reported_seq":10108,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.477995+0000","last_change":"2024-09-18T11:44:47.413703+0000","last_active":"2024-09-18T11:52:33.477995+0000","last_peered":"2024-09-18T11:52:33.477995+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:33.477995+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23540982,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10282","reported_seq":14563,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.589447+0000","last_change":"2024-09-18T11:49:40.464642+0000","last_active":"2024-09-18T11:52:32.589447+0000","last_peered":"2024-09-18T11:52:32.589447+0000","last_clean":"2024-09-18T11:43:44.501034+0000","last_became_active":"2024-09-18T11:44:47.351354+0000","last_became_peered":"2024-09-18T11:44:47.351354+0000","last_unstale":"2024-09-18T11:52:32.589447+0000","last_undegraded":"2024-09-18T11:44:47.165795+0000","last_fullsized":"2024-09-18T11:44:47.117426+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23701996,"num_objects":2894,"num_object_clones":0,"num_object_copies":11576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10279,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9821","reported_seq":9933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.718480+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:32.718480+0000","last_peered":"2024-09-18T11:52:32.718480+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:32.718480+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22713590,"num_objects":2773,"num_object_clones":0,"num_object_copies":11092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9819,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":810}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10124","reported_seq":10243,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.500230+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:32.500230+0000","last_peered":"2024-09-18T11:52:32.500230+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:32.500230+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23826289,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10121,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10195","reported_seq":10286,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.454876+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:32.454876+0000","last_peered":"2024-09-18T11:52:32.454876+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:32.454876+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23231099,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10194,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10101","reported_seq":10197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.612722+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:32.612722+0000","last_peered":"2024-09-18T11:52:32.612722+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:32.612722+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23476859,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10100,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10219","reported_seq":10342,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:32.571735+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:32.571735+0000","last_peered":"2024-09-18T11:52:32.571735+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:32.571735+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23516406,"num_objects":2871,"num_object_clones":0,"num_object_copies":11484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10217,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10066","reported_seq":14353,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:52:33.451140+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:33.451140+0000","last_peered":"2024-09-18T11:52:33.451140+0000","last_clean":"2024-09-18T11:52:33.451140+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:33.451140+0000","last_undegraded":"2024-09-18T11:52:33.451140+0000","last_fullsized":"2024-09-18T11:52:33.451140+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23237878,"num_objects":2837,"num_object_clones":0,"num_object_copies":11348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10064,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"35'9910","reported_seq":9998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:33.716319+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:33.716319+0000","last_peered":"2024-09-18T11:52:33.716319+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:33.716319+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22910198,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9908,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":372901260,"num_objects":45526,"num_object_clones":0,"num_object_copies":182104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161561,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":745369600,"data_stored":745369600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160188,"ondisk_log_size":160188,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769496,"kb_used_data":183232,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93602344,"statfs":{"total":96636764160,"available":95848800256,"internally_reserved":0,"allocated":187629568,"data_stored":187454102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":6.3150000000000004},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.22500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.2040000000000006},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":1.966}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.4809999999999999},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.95699999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776748,"kb_used_data":182692,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93595092,"statfs":{"total":96636764160,"available":95841374208,"internally_reserved":0,"allocated":187076608,"data_stored":186901142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":8.9779999999999998},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":9.0060000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":3.3660000000000001},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":0.27300000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.9459999999999997},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.776999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496912,"kb_used_data":182468,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93874928,"statfs":{"total":96636764160,"available":96127926272,"internally_reserved":0,"allocated":186847232,"data_stored":186671766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":3.2999999999999998},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":4.4909999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":9.6799999999999997},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":3.2669999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.4239999999999999},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":17.300999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772392,"kb_used_data":182040,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93599448,"statfs":{"total":96636764160,"available":95845834752,"internally_reserved":0,"allocated":186408960,"data_stored":186233494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":6.4740000000000002},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.7370000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.5069999999999997},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":6.7949999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":6.5030000000000001},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":6.7720000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185794560,"data_stored":185794560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186302464,"data_stored":186302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186535936,"data_stored":186535936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":186736640,"data_stored":186736640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:40.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:41.178 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:41.179 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:41.197 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-18T11:52:40.030705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":364758412,"num_objects":44532,"num_object_clones":0,"num_object_copies":178128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162555,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6676,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160582,"ondisk_log_size":160582,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799600,"kb_used_data":714484,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374687760,"statfs":{"total":386547056640,"available":383680266240,"internally_reserved":0,"allocated":731631616,"data_stored":730900520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,6,11,14,6,8,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16359424,"num_objects":-1997,"num_object_clones":0,"num_object_copies":-7988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1997,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002599"},"pg_stats":[{"pgid":"1.f","version":"37'10403","reported_seq":10504,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.452808+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:37.452808+0000","last_peered":"2024-09-18T11:52:37.452808+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:37.452808+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23322624,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10222","reported_seq":14514,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:38.561291+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:38.561291+0000","last_peered":"2024-09-18T11:52:38.561291+0000","last_clean":"2024-09-18T11:52:38.561291+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:38.561291+0000","last_undegraded":"2024-09-18T11:52:38.561291+0000","last_fullsized":"2024-09-18T11:52:38.561291+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23213302,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10220,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10057","reported_seq":10174,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.737648+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:37.737648+0000","last_peered":"2024-09-18T11:52:37.737648+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:37.737648+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22534803,"num_objects":2752,"num_object_clones":0,"num_object_copies":11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10056,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":712}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10164","reported_seq":10258,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.248658+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:38.248658+0000","last_peered":"2024-09-18T11:52:38.248658+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:38.248658+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22623478,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10098","reported_seq":10219,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.536808+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:37.536808+0000","last_peered":"2024-09-18T11:52:37.536808+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:37.536808+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22910198,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10096,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10081","reported_seq":10200,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.659777+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:37.659777+0000","last_peered":"2024-09-18T11:52:37.659777+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:37.659777+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22159360,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10081,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10268","reported_seq":10375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.717986+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:38.717986+0000","last_peered":"2024-09-18T11:52:38.717986+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:38.717986+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22508790,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10266,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":632}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10059","reported_seq":10242,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.741429+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:38.741429+0000","last_peered":"2024-09-18T11:52:38.741429+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:38.741429+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23065846,"num_objects":2816,"num_object_clones":0,"num_object_copies":11264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10340","reported_seq":14692,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:37.659628+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:37.659628+0000","last_peered":"2024-09-18T11:52:37.659628+0000","last_clean":"2024-09-18T11:52:37.659628+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:37.659628+0000","last_undegraded":"2024-09-18T11:52:37.659628+0000","last_fullsized":"2024-09-18T11:52:37.659628+0000","mapping_epoch":36,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23226860,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10337,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9885","reported_seq":9998,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.826258+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:37.826258+0000","last_peered":"2024-09-18T11:52:37.826258+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:37.826258+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22189302,"num_objects":2709,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":746}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10207","reported_seq":10328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.640457+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:37.640457+0000","last_peered":"2024-09-18T11:52:37.640457+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:37.640457+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23146353,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10262","reported_seq":10355,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.557398+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:37.557398+0000","last_peered":"2024-09-18T11:52:37.557398+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:37.557398+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22682235,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10261,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10169","reported_seq":10267,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.742747+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:37.742747+0000","last_peered":"2024-09-18T11:52:37.742747+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:37.742747+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22919803,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10168,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10275","reported_seq":10400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.515304+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:37.515304+0000","last_peered":"2024-09-18T11:52:37.515304+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:37.515304+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23057654,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":692}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10122","reported_seq":14411,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:38.430602+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:38.430602+0000","last_peered":"2024-09-18T11:52:38.430602+0000","last_clean":"2024-09-18T11:52:38.430602+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:38.430602+0000","last_undegraded":"2024-09-18T11:52:38.430602+0000","last_fullsized":"2024-09-18T11:52:38.430602+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22779126,"num_objects":2781,"num_object_clones":0,"num_object_copies":11124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10120,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'9970","reported_seq":10060,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.632449+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:38.632449+0000","last_peered":"2024-09-18T11:52:38.632449+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:38.632449+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22418678,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9968,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":364758412,"num_objects":44532,"num_object_clones":0,"num_object_copies":178128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162555,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6676,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":729174016,"data_stored":729174016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160582,"ondisk_log_size":160582,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769560,"kb_used_data":179148,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93602280,"statfs":{"total":96636764160,"available":95848734720,"internally_reserved":0,"allocated":183447552,"data_stored":183267919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,2,3,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.9630000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":2.9390000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.749},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":2.7599999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.89500000000000002},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.036}]}]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777000,"kb_used_data":178772,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93594840,"statfs":{"total":96636764160,"available":95841116160,"internally_reserved":0,"allocated":183062528,"data_stored":182878707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":3.0249999999999999},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":3.0510000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.4420000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.8089999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":1.778},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.513}]}]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480484,"kb_used_data":178516,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93891356,"statfs":{"total":96636764160,"available":96144748544,"internally_reserved":0,"allocated":182800384,"data_stored":182616563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":15.079000000000001},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.8040000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":15.288},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":1.847}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":8.6500000000000004},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":14.835000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772556,"kb_used_data":178048,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93599284,"statfs":{"total":96636764160,"available":95845666816,"internally_reserved":0,"allocated":182321152,"data_stored":182137331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,10,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":23.052},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.0919999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":19.166},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":5.0570000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":26.114000000000001},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":5.1180000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":181760000,"data_stored":181760000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182194176,"data_stored":182194176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182538240,"data_stored":182538240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":182681600,"data_stored":182681600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:41.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:41.829 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:41.830 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:41.863 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-18T11:52:40.030705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":364758412,"num_objects":44532,"num_object_clones":0,"num_object_copies":178128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162555,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6676,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160582,"ondisk_log_size":160582,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799600,"kb_used_data":714484,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374687760,"statfs":{"total":386547056640,"available":383680266240,"internally_reserved":0,"allocated":731631616,"data_stored":730900520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,6,11,14,6,8,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16359424,"num_objects":-1997,"num_object_clones":0,"num_object_copies":-7988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1997,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002599"},"pg_stats":[{"pgid":"1.f","version":"37'10403","reported_seq":10504,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.452808+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:37.452808+0000","last_peered":"2024-09-18T11:52:37.452808+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:37.452808+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23322624,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10222","reported_seq":14514,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:38.561291+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:38.561291+0000","last_peered":"2024-09-18T11:52:38.561291+0000","last_clean":"2024-09-18T11:52:38.561291+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:38.561291+0000","last_undegraded":"2024-09-18T11:52:38.561291+0000","last_fullsized":"2024-09-18T11:52:38.561291+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23213302,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10220,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10057","reported_seq":10174,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.737648+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:37.737648+0000","last_peered":"2024-09-18T11:52:37.737648+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:37.737648+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22534803,"num_objects":2752,"num_object_clones":0,"num_object_copies":11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10056,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":712}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10164","reported_seq":10258,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.248658+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:38.248658+0000","last_peered":"2024-09-18T11:52:38.248658+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:38.248658+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22623478,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10098","reported_seq":10219,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.536808+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:37.536808+0000","last_peered":"2024-09-18T11:52:37.536808+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:37.536808+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22910198,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10096,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10081","reported_seq":10200,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.659777+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:37.659777+0000","last_peered":"2024-09-18T11:52:37.659777+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:37.659777+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22159360,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10081,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10268","reported_seq":10375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.717986+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:38.717986+0000","last_peered":"2024-09-18T11:52:38.717986+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:38.717986+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22508790,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10266,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":632}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10059","reported_seq":10242,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.741429+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:38.741429+0000","last_peered":"2024-09-18T11:52:38.741429+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:38.741429+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23065846,"num_objects":2816,"num_object_clones":0,"num_object_copies":11264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10340","reported_seq":14692,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:37.659628+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:37.659628+0000","last_peered":"2024-09-18T11:52:37.659628+0000","last_clean":"2024-09-18T11:52:37.659628+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:37.659628+0000","last_undegraded":"2024-09-18T11:52:37.659628+0000","last_fullsized":"2024-09-18T11:52:37.659628+0000","mapping_epoch":36,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23226860,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10337,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9885","reported_seq":9998,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.826258+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:37.826258+0000","last_peered":"2024-09-18T11:52:37.826258+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:37.826258+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22189302,"num_objects":2709,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":746}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10207","reported_seq":10328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.640457+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:37.640457+0000","last_peered":"2024-09-18T11:52:37.640457+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:37.640457+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23146353,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10262","reported_seq":10355,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.557398+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:37.557398+0000","last_peered":"2024-09-18T11:52:37.557398+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:37.557398+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22682235,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10261,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10169","reported_seq":10267,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.742747+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:37.742747+0000","last_peered":"2024-09-18T11:52:37.742747+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:37.742747+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22919803,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10168,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10275","reported_seq":10400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.515304+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:37.515304+0000","last_peered":"2024-09-18T11:52:37.515304+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:37.515304+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23057654,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":692}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10122","reported_seq":14411,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:38.430602+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:38.430602+0000","last_peered":"2024-09-18T11:52:38.430602+0000","last_clean":"2024-09-18T11:52:38.430602+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:38.430602+0000","last_undegraded":"2024-09-18T11:52:38.430602+0000","last_fullsized":"2024-09-18T11:52:38.430602+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22779126,"num_objects":2781,"num_object_clones":0,"num_object_copies":11124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10120,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'9970","reported_seq":10060,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.632449+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:38.632449+0000","last_peered":"2024-09-18T11:52:38.632449+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:38.632449+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22418678,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9968,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":364758412,"num_objects":44532,"num_object_clones":0,"num_object_copies":178128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162555,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6676,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":729174016,"data_stored":729174016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160582,"ondisk_log_size":160582,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769560,"kb_used_data":179148,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93602280,"statfs":{"total":96636764160,"available":95848734720,"internally_reserved":0,"allocated":183447552,"data_stored":183267919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,2,3,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.9630000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":2.9390000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.749},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":2.7599999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.89500000000000002},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.036}]}]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777000,"kb_used_data":178772,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93594840,"statfs":{"total":96636764160,"available":95841116160,"internally_reserved":0,"allocated":183062528,"data_stored":182878707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":3.0249999999999999},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":3.0510000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.4420000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.8089999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":1.778},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.513}]}]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480484,"kb_used_data":178516,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93891356,"statfs":{"total":96636764160,"available":96144748544,"internally_reserved":0,"allocated":182800384,"data_stored":182616563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":15.079000000000001},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.8040000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":15.288},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":1.847}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":8.6500000000000004},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":14.835000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772556,"kb_used_data":178048,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93599284,"statfs":{"total":96636764160,"available":95845666816,"internally_reserved":0,"allocated":182321152,"data_stored":182137331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,10,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":23.052},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.0919999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":19.166},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":5.0570000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":26.114000000000001},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":5.1180000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":181760000,"data_stored":181760000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182194176,"data_stored":182194176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182538240,"data_stored":182538240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":182681600,"data_stored":182681600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:41.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:41.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10403", 'reported_seq': 10504, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:37.452808+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:37.452808+0000', 'last_peered': '2024-09-18T11:52:37.452808+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:37.452808+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23322624, 'num_objects': 2847, 'num_object_clones': 0, 'num_object_copies': 11388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10403, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 744}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:41.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:41.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10057", 'reported_seq': 10174, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:37.737648+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:37.737648+0000', 'last_peered': '2024-09-18T11:52:37.737648+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:37.737648+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22534803, 'num_objects': 2752, 'num_object_clones': 0, 'num_object_copies': 11008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2752, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10056, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 712}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:41.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:41.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10164", 'reported_seq': 10258, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:38.248658+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:38.248658+0000', 'last_peered': '2024-09-18T11:52:38.248658+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:38.248658+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22623478, 'num_objects': 2762, 'num_object_clones': 0, 'num_object_copies': 11048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10162, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 661}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:41.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:41.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10098", 'reported_seq': 10219, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:37.536808+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:37.536808+0000', 'last_peered': '2024-09-18T11:52:37.536808+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:37.536808+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22910198, 'num_objects': 2797, 'num_object_clones': 0, 'num_object_copies': 11188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2797, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10096, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 745}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:41.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:41.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10081", 'reported_seq': 10200, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:37.659777+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:37.659777+0000', 'last_peered': '2024-09-18T11:52:37.659777+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:37.659777+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22159360, 'num_objects': 2705, 'num_object_clones': 0, 'num_object_copies': 10820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10081, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 586}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:41.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:41.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10268", 'reported_seq': 10375, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:38.717986+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:38.717986+0000', 'last_peered': '2024-09-18T11:52:38.717986+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:38.717986+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22508790, 'num_objects': 2748, 'num_object_clones': 0, 'num_object_copies': 10992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10266, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 632}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:41.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:41.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10059", 'reported_seq': 10242, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:38.741429+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:52:38.741429+0000', 'last_peered': '2024-09-18T11:52:38.741429+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:38.741429+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23065846, 'num_objects': 2816, 'num_object_clones': 0, 'num_object_copies': 11264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2816, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10057, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 59, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:41.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:41.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9885", 'reported_seq': 9998, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:37.826258+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:37.826258+0000', 'last_peered': '2024-09-18T11:52:37.826258+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:37.826258+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9885, 'log_dups_size': 0, 'ondisk_log_size': 9885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22189302, 'num_objects': 2709, 'num_object_clones': 0, 'num_object_copies': 10836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9883, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 746}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:41.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:41.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10207", 'reported_seq': 10328, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:37.640457+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:37.640457+0000', 'last_peered': '2024-09-18T11:52:37.640457+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:37.640457+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23146353, 'num_objects': 2826, 'num_object_clones': 0, 'num_object_copies': 11304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10204, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 792}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:41.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:41.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10262", 'reported_seq': 10355, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:37.557398+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:37.557398+0000', 'last_peered': '2024-09-18T11:52:37.557398+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:37.557398+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22682235, 'num_objects': 2769, 'num_object_clones': 0, 'num_object_copies': 11076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10261, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 685}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:41.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:41.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10169", 'reported_seq': 10267, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:37.742747+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:37.742747+0000', 'last_peered': '2024-09-18T11:52:37.742747+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:37.742747+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22919803, 'num_objects': 2798, 'num_object_clones': 0, 'num_object_copies': 11192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10168, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:41.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:41.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10275", 'reported_seq': 10400, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:37.515304+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:37.515304+0000', 'last_peered': '2024-09-18T11:52:37.515304+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:37.515304+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23057654, 'num_objects': 2815, 'num_object_clones': 0, 'num_object_copies': 11260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10273, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 692}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:41.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:41.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9970", 'reported_seq': 10060, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:38.632449+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:38.632449+0000', 'last_peered': '2024-09-18T11:52:38.632449+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:38.632449+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9970, 'log_dups_size': 0, 'ondisk_log_size': 9970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22418678, 'num_objects': 2737, 'num_object_clones': 0, 'num_object_copies': 10948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9968, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 735}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:41.867 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:42.609 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:42.609 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26780/178128 objects degraded (15.034%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":560,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44532,"data_bytes":364758412,"bytes_used":2866790400,"bytes_avail":383680266240,"bytes_total":386547056640,"degraded_objects":26780,"degraded_total":178128,"degraded_ratio":0.15034132758465824,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":124},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:42.634 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:42.634 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:43.152 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:43.153 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:43.180 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-18T11:52:42.031128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":364758412,"num_objects":44532,"num_object_clones":0,"num_object_copies":178128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162555,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6676,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160582,"ondisk_log_size":160582,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799600,"kb_used_data":714484,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374687760,"statfs":{"total":386547056640,"available":383680266240,"internally_reserved":0,"allocated":731631616,"data_stored":730900520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,6,11,14,6,8,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8142848,"num_objects":-994,"num_object_clones":0,"num_object_copies":-3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-66,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":994,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002606"},"pg_stats":[{"pgid":"1.f","version":"37'10403","reported_seq":10504,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.452808+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:37.452808+0000","last_peered":"2024-09-18T11:52:37.452808+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:37.452808+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23322624,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10222","reported_seq":14514,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:38.561291+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:38.561291+0000","last_peered":"2024-09-18T11:52:38.561291+0000","last_clean":"2024-09-18T11:52:38.561291+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:38.561291+0000","last_undegraded":"2024-09-18T11:52:38.561291+0000","last_fullsized":"2024-09-18T11:52:38.561291+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23213302,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10220,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10057","reported_seq":10174,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.737648+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:37.737648+0000","last_peered":"2024-09-18T11:52:37.737648+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:37.737648+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22534803,"num_objects":2752,"num_object_clones":0,"num_object_copies":11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10056,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":712}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10164","reported_seq":10258,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.248658+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:38.248658+0000","last_peered":"2024-09-18T11:52:38.248658+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:38.248658+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22623478,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10098","reported_seq":10219,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.536808+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:37.536808+0000","last_peered":"2024-09-18T11:52:37.536808+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:37.536808+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22910198,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10096,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10081","reported_seq":10200,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.659777+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:37.659777+0000","last_peered":"2024-09-18T11:52:37.659777+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:37.659777+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22159360,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10081,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10268","reported_seq":10375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.717986+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:38.717986+0000","last_peered":"2024-09-18T11:52:38.717986+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:38.717986+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22508790,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10266,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":632}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10059","reported_seq":10242,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.741429+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:38.741429+0000","last_peered":"2024-09-18T11:52:38.741429+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:38.741429+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23065846,"num_objects":2816,"num_object_clones":0,"num_object_copies":11264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10340","reported_seq":14692,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:37.659628+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:37.659628+0000","last_peered":"2024-09-18T11:52:37.659628+0000","last_clean":"2024-09-18T11:52:37.659628+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:37.659628+0000","last_undegraded":"2024-09-18T11:52:37.659628+0000","last_fullsized":"2024-09-18T11:52:37.659628+0000","mapping_epoch":36,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23226860,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10337,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9885","reported_seq":9998,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.826258+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:37.826258+0000","last_peered":"2024-09-18T11:52:37.826258+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:37.826258+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22189302,"num_objects":2709,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":746}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10207","reported_seq":10328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.640457+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:37.640457+0000","last_peered":"2024-09-18T11:52:37.640457+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:37.640457+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23146353,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10262","reported_seq":10355,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.557398+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:37.557398+0000","last_peered":"2024-09-18T11:52:37.557398+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:37.557398+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22682235,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10261,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10169","reported_seq":10267,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.742747+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:37.742747+0000","last_peered":"2024-09-18T11:52:37.742747+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:37.742747+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22919803,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10168,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10275","reported_seq":10400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:37.515304+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:37.515304+0000","last_peered":"2024-09-18T11:52:37.515304+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:37.515304+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23057654,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":692}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10122","reported_seq":14411,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:38.430602+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:38.430602+0000","last_peered":"2024-09-18T11:52:38.430602+0000","last_clean":"2024-09-18T11:52:38.430602+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:38.430602+0000","last_undegraded":"2024-09-18T11:52:38.430602+0000","last_fullsized":"2024-09-18T11:52:38.430602+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22779126,"num_objects":2781,"num_object_clones":0,"num_object_copies":11124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10120,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'9970","reported_seq":10060,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:38.632449+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:38.632449+0000","last_peered":"2024-09-18T11:52:38.632449+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:38.632449+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22418678,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9968,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":364758412,"num_objects":44532,"num_object_clones":0,"num_object_copies":178128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162555,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6676,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":729174016,"data_stored":729174016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160582,"ondisk_log_size":160582,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769560,"kb_used_data":179148,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93602280,"statfs":{"total":96636764160,"available":95848734720,"internally_reserved":0,"allocated":183447552,"data_stored":183267919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,2,3,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":2.9630000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":2.9390000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.749},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":2.7599999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.89500000000000002},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.036}]}]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777000,"kb_used_data":178772,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93594840,"statfs":{"total":96636764160,"available":95841116160,"internally_reserved":0,"allocated":183062528,"data_stored":182878707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":3.0249999999999999},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":3.0510000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.4420000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":1.8089999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":1.778},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.513}]}]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480484,"kb_used_data":178516,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93891356,"statfs":{"total":96636764160,"available":96144748544,"internally_reserved":0,"allocated":182800384,"data_stored":182616563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":15.079000000000001},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.8040000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":15.288},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":1.847}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":8.6500000000000004},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":14.835000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772556,"kb_used_data":178048,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93599284,"statfs":{"total":96636764160,"available":95845666816,"internally_reserved":0,"allocated":182321152,"data_stored":182137331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,10,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":23.052},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.0919999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":19.166},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":5.0570000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":26.114000000000001},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":5.1180000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":181760000,"data_stored":181760000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182194176,"data_stored":182194176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182538240,"data_stored":182538240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":182681600,"data_stored":182681600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:46.183 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:46.755 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:46.756 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:46.777 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-09-18T11:52:46.032032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356312460,"num_objects":43501,"num_object_clones":0,"num_object_copies":174004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43501,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163586,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6716,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160613,"ondisk_log_size":160613,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779064,"kb_used_data":698104,"kb_used_omap":14,"kb_used_meta":2080881,"kb_avail":374708296,"statfs":{"total":386547056640,"available":383701295104,"internally_reserved":0,"allocated":714858496,"data_stored":714123212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130822171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,11,2,19,28,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16588800,"num_objects":-2025,"num_object_clones":0,"num_object_copies":-8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2025,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.761189"},"pg_stats":[{"pgid":"1.f","version":"37'10473","reported_seq":10574,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.601542+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:42.601542+0000","last_peered":"2024-09-18T11:52:42.601542+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:42.601542+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22749184,"num_objects":2777,"num_object_clones":0,"num_object_copies":11108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10291","reported_seq":14583,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:43.657012+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:43.657012+0000","last_peered":"2024-09-18T11:52:43.657012+0000","last_clean":"2024-09-18T11:52:43.657012+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:43.657012+0000","last_undegraded":"2024-09-18T11:52:43.657012+0000","last_fullsized":"2024-09-18T11:52:43.657012+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648054,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10289,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10119","reported_seq":10235,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.696542+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:42.696542+0000","last_peered":"2024-09-18T11:52:42.696542+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:42.696542+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22026899,"num_objects":2690,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10118,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10234","reported_seq":10328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.657410+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:43.657410+0000","last_peered":"2024-09-18T11:52:43.657410+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:43.657410+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22050038,"num_objects":2692,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10232,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10152","reported_seq":10273,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.626220+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:42.626220+0000","last_peered":"2024-09-18T11:52:42.626220+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:42.626220+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22467830,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10150,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10146","reported_seq":10265,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.562346+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:42.562346+0000","last_peered":"2024-09-18T11:52:42.562346+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:42.562346+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21626880,"num_objects":2640,"num_object_clones":0,"num_object_copies":10560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10146,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":521}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10320","reported_seq":10426,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.746342+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:43.746342+0000","last_peered":"2024-09-18T11:52:43.746342+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:43.746342+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22082806,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10318,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":580}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10118","reported_seq":10382,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.719649+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:43.719649+0000","last_peered":"2024-09-18T11:52:43.719649+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:43.719649+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22582518,"num_objects":2757,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10116,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10411","reported_seq":14764,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:42.699615+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:42.699615+0000","last_peered":"2024-09-18T11:52:42.699615+0000","last_clean":"2024-09-18T11:52:42.699615+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:42.699615+0000","last_undegraded":"2024-09-18T11:52:42.699615+0000","last_fullsized":"2024-09-18T11:52:42.699615+0000","mapping_epoch":36,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22645228,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10408,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9949","reported_seq":10063,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.877929+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:42.877929+0000","last_peered":"2024-09-18T11:52:42.877929+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:42.877929+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21665014,"num_objects":2645,"num_object_clones":0,"num_object_copies":10580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9947,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":682}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10270","reported_seq":10391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.497248+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:42.497248+0000","last_peered":"2024-09-18T11:52:42.497248+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:42.497248+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22630257,"num_objects":2763,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10324","reported_seq":10417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.540823+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:42.540823+0000","last_peered":"2024-09-18T11:52:42.540823+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:42.540823+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22174331,"num_objects":2707,"num_object_clones":0,"num_object_copies":10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10323,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10237","reported_seq":10335,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.655543+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:42.655543+0000","last_peered":"2024-09-18T11:52:42.655543+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:42.655543+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22362747,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10236,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":684}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10340","reported_seq":10465,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.529712+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:42.529712+0000","last_peered":"2024-09-18T11:52:42.529712+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:42.529712+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22525174,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10338,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":627}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10176","reported_seq":14465,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:43.646753+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:43.646753+0000","last_peered":"2024-09-18T11:52:43.646753+0000","last_clean":"2024-09-18T11:52:43.646753+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:43.646753+0000","last_undegraded":"2024-09-18T11:52:43.646753+0000","last_fullsized":"2024-09-18T11:52:43.646753+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22336758,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10174,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10053","reported_seq":10142,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.718738+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:43.718738+0000","last_peered":"2024-09-18T11:52:43.718738+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:43.718738+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21738742,"num_objects":2654,"num_object_clones":0,"num_object_copies":10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10051,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":356312460,"num_objects":43501,"num_object_clones":0,"num_object_copies":174004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43501,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163586,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6716,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712429568,"data_stored":712429568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160613,"ondisk_log_size":160613,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769692,"kb_used_data":175112,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93602148,"statfs":{"total":96636764160,"available":95848599552,"internally_reserved":0,"allocated":179314688,"data_stored":179130867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,8,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.992000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":11.023999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":17.117999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.9880000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":1.99},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":6.0199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760332,"kb_used_data":174584,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93611508,"statfs":{"total":96636764160,"available":95858184192,"internally_reserved":0,"allocated":178774016,"data_stored":178590195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":3.4689999999999999},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":3.5009999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":15.564},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.536}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":3.556},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":13.574999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476368,"kb_used_data":174408,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93895472,"statfs":{"total":96636764160,"available":96148963328,"internally_reserved":0,"allocated":178593792,"data_stored":178409971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":12.708},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":12.577999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":9.9529999999999994},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":4.1379999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":9.9920000000000009}]}]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772672,"kb_used_data":174000,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93599168,"statfs":{"total":96636764160,"available":95845548032,"internally_reserved":0,"allocated":178176000,"data_stored":177992179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.5169999999999999},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.27800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.5960000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":11.715999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":1.55},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":11.673999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":177594368,"data_stored":177594368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":178032640,"data_stored":178032640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178335744,"data_stored":178335744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":178466816,"data_stored":178466816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:46.778 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:47.301 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:47.301 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:47.323 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-09-18T11:52:46.032032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356312460,"num_objects":43501,"num_object_clones":0,"num_object_copies":174004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43501,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163586,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6716,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160613,"ondisk_log_size":160613,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779064,"kb_used_data":698104,"kb_used_omap":14,"kb_used_meta":2080881,"kb_avail":374708296,"statfs":{"total":386547056640,"available":383701295104,"internally_reserved":0,"allocated":714858496,"data_stored":714123212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130822171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,11,2,19,28,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16588800,"num_objects":-2025,"num_object_clones":0,"num_object_copies":-8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2025,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.761189"},"pg_stats":[{"pgid":"1.f","version":"37'10473","reported_seq":10574,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.601542+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:42.601542+0000","last_peered":"2024-09-18T11:52:42.601542+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:42.601542+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22749184,"num_objects":2777,"num_object_clones":0,"num_object_copies":11108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10291","reported_seq":14583,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:43.657012+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:43.657012+0000","last_peered":"2024-09-18T11:52:43.657012+0000","last_clean":"2024-09-18T11:52:43.657012+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:43.657012+0000","last_undegraded":"2024-09-18T11:52:43.657012+0000","last_fullsized":"2024-09-18T11:52:43.657012+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648054,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10289,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10119","reported_seq":10235,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.696542+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:42.696542+0000","last_peered":"2024-09-18T11:52:42.696542+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:42.696542+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22026899,"num_objects":2690,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10118,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10234","reported_seq":10328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.657410+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:43.657410+0000","last_peered":"2024-09-18T11:52:43.657410+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:43.657410+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22050038,"num_objects":2692,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10232,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10152","reported_seq":10273,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.626220+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:42.626220+0000","last_peered":"2024-09-18T11:52:42.626220+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:42.626220+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22467830,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10150,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10146","reported_seq":10265,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.562346+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:42.562346+0000","last_peered":"2024-09-18T11:52:42.562346+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:42.562346+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21626880,"num_objects":2640,"num_object_clones":0,"num_object_copies":10560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10146,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":521}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10320","reported_seq":10426,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.746342+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:43.746342+0000","last_peered":"2024-09-18T11:52:43.746342+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:43.746342+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22082806,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10318,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":580}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10118","reported_seq":10382,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.719649+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:43.719649+0000","last_peered":"2024-09-18T11:52:43.719649+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:43.719649+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22582518,"num_objects":2757,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10116,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10411","reported_seq":14764,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:42.699615+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:42.699615+0000","last_peered":"2024-09-18T11:52:42.699615+0000","last_clean":"2024-09-18T11:52:42.699615+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:42.699615+0000","last_undegraded":"2024-09-18T11:52:42.699615+0000","last_fullsized":"2024-09-18T11:52:42.699615+0000","mapping_epoch":36,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22645228,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10408,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9949","reported_seq":10063,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.877929+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:42.877929+0000","last_peered":"2024-09-18T11:52:42.877929+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:42.877929+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21665014,"num_objects":2645,"num_object_clones":0,"num_object_copies":10580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9947,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":682}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10270","reported_seq":10391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.497248+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:42.497248+0000","last_peered":"2024-09-18T11:52:42.497248+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:42.497248+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22630257,"num_objects":2763,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10324","reported_seq":10417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.540823+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:42.540823+0000","last_peered":"2024-09-18T11:52:42.540823+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:42.540823+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22174331,"num_objects":2707,"num_object_clones":0,"num_object_copies":10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10323,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10237","reported_seq":10335,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.655543+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:42.655543+0000","last_peered":"2024-09-18T11:52:42.655543+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:42.655543+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22362747,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10236,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":684}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10340","reported_seq":10465,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:42.529712+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:42.529712+0000","last_peered":"2024-09-18T11:52:42.529712+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:42.529712+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22525174,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10338,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":627}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10176","reported_seq":14465,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:43.646753+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:43.646753+0000","last_peered":"2024-09-18T11:52:43.646753+0000","last_clean":"2024-09-18T11:52:43.646753+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:43.646753+0000","last_undegraded":"2024-09-18T11:52:43.646753+0000","last_fullsized":"2024-09-18T11:52:43.646753+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22336758,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10174,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10053","reported_seq":10142,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.718738+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:43.718738+0000","last_peered":"2024-09-18T11:52:43.718738+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:43.718738+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21738742,"num_objects":2654,"num_object_clones":0,"num_object_copies":10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10051,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":356312460,"num_objects":43501,"num_object_clones":0,"num_object_copies":174004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43501,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163586,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6716,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712429568,"data_stored":712429568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160613,"ondisk_log_size":160613,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769692,"kb_used_data":175112,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93602148,"statfs":{"total":96636764160,"available":95848599552,"internally_reserved":0,"allocated":179314688,"data_stored":179130867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,8,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.992000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":11.023999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":17.117999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.9880000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":1.99},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":6.0199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760332,"kb_used_data":174584,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93611508,"statfs":{"total":96636764160,"available":95858184192,"internally_reserved":0,"allocated":178774016,"data_stored":178590195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":3.4689999999999999},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":3.5009999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":15.564},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.536}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":3.556},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":13.574999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476368,"kb_used_data":174408,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93895472,"statfs":{"total":96636764160,"available":96148963328,"internally_reserved":0,"allocated":178593792,"data_stored":178409971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":12.708},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":12.577999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":9.9529999999999994},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":4.1379999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":9.9920000000000009}]}]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772672,"kb_used_data":174000,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93599168,"statfs":{"total":96636764160,"available":95845548032,"internally_reserved":0,"allocated":178176000,"data_stored":177992179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.5169999999999999},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.27800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.5960000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":11.715999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":1.55},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":11.673999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":177594368,"data_stored":177594368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":178032640,"data_stored":178032640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178335744,"data_stored":178335744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":178466816,"data_stored":178466816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:47.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:47.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10473", 'reported_seq': 10574, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:42.601542+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:42.601542+0000', 'last_peered': '2024-09-18T11:52:42.601542+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:42.601542+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22749184, 'num_objects': 2777, 'num_object_clones': 0, 'num_object_copies': 11108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10473, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 674}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:47.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:47.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10119", 'reported_seq': 10235, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:42.696542+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:42.696542+0000', 'last_peered': '2024-09-18T11:52:42.696542+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:42.696542+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22026899, 'num_objects': 2690, 'num_object_clones': 0, 'num_object_copies': 10760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2690, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10118, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 650}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:47.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:47.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10234", 'reported_seq': 10328, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:43.657410+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:43.657410+0000', 'last_peered': '2024-09-18T11:52:43.657410+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:43.657410+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22050038, 'num_objects': 2692, 'num_object_clones': 0, 'num_object_copies': 10768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10232, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:47.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:47.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10152", 'reported_seq': 10273, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:42.626220+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:42.626220+0000', 'last_peered': '2024-09-18T11:52:42.626220+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:42.626220+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22467830, 'num_objects': 2743, 'num_object_clones': 0, 'num_object_copies': 10972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10150, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 691}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:47.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:47.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10146", 'reported_seq': 10265, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:42.562346+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:42.562346+0000', 'last_peered': '2024-09-18T11:52:42.562346+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:42.562346+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21626880, 'num_objects': 2640, 'num_object_clones': 0, 'num_object_copies': 10560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10146, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 521}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10320", 'reported_seq': 10426, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:43.746342+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:43.746342+0000', 'last_peered': '2024-09-18T11:52:43.746342+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:43.746342+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22082806, 'num_objects': 2696, 'num_object_clones': 0, 'num_object_copies': 10784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10318, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 580}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10118", 'reported_seq': 10382, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:43.719649+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:52:43.719649+0000', 'last_peered': '2024-09-18T11:52:43.719649+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:43.719649+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22582518, 'num_objects': 2757, 'num_object_clones': 0, 'num_object_copies': 11028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10116, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 818}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9949", 'reported_seq': 10063, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:42.877929+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:42.877929+0000', 'last_peered': '2024-09-18T11:52:42.877929+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:42.877929+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9949, 'log_dups_size': 0, 'ondisk_log_size': 9949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21665014, 'num_objects': 2645, 'num_object_clones': 0, 'num_object_copies': 10580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9947, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 682}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10270", 'reported_seq': 10391, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:42.497248+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:42.497248+0000', 'last_peered': '2024-09-18T11:52:42.497248+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:42.497248+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22630257, 'num_objects': 2763, 'num_object_clones': 0, 'num_object_copies': 11052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10267, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 729}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10324", 'reported_seq': 10417, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:42.540823+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:42.540823+0000', 'last_peered': '2024-09-18T11:52:42.540823+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:42.540823+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22174331, 'num_objects': 2707, 'num_object_clones': 0, 'num_object_copies': 10828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10323, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 623}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:47.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:47.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10237", 'reported_seq': 10335, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:42.655543+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:42.655543+0000', 'last_peered': '2024-09-18T11:52:42.655543+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:42.655543+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22362747, 'num_objects': 2730, 'num_object_clones': 0, 'num_object_copies': 10920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10236, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 684}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:47.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:47.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10340", 'reported_seq': 10465, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:42.529712+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:42.529712+0000', 'last_peered': '2024-09-18T11:52:42.529712+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:42.529712+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22525174, 'num_objects': 2750, 'num_object_clones': 0, 'num_object_copies': 11000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10338, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 627}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:47.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:47.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10053", 'reported_seq': 10142, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:43.718738+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:43.718738+0000', 'last_peered': '2024-09-18T11:52:43.718738+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:43.718738+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21738742, 'num_objects': 2654, 'num_object_clones': 0, 'num_object_copies': 10616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2654, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10051, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 652}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:47.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:48.020 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:48.020 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26740/174004 objects degraded (15.367%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":566,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43501,"data_bytes":356312460,"bytes_used":2845761536,"bytes_avail":383701295104,"bytes_total":386547056640,"degraded_objects":26740,"degraded_total":174004,"degraded_ratio":0.15367462816946736,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:48.043 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:48.043 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:48.608 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:48.608 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:48.637 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-18T11:52:48.032431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":350963084,"num_objects":42848,"num_object_clones":0,"num_object_copies":171392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164239,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6716,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160666,"ondisk_log_size":160666,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779072,"kb_used_data":689816,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374708288,"statfs":{"total":386547056640,"available":383701286912,"internally_reserved":0,"allocated":706371584,"data_stored":705636300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,13,2,20,24,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21938176,"num_objects":-2678,"num_object_clones":0,"num_object_copies":-10712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2678,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.761254"},"pg_stats":[{"pgid":"1.f","version":"37'10535","reported_seq":10636,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.646680+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:47.646680+0000","last_peered":"2024-09-18T11:52:47.646680+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:47.646680+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22241280,"num_objects":2715,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10291","reported_seq":14583,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:43.657012+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:43.657012+0000","last_peered":"2024-09-18T11:52:43.657012+0000","last_clean":"2024-09-18T11:52:43.657012+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:43.657012+0000","last_undegraded":"2024-09-18T11:52:43.657012+0000","last_fullsized":"2024-09-18T11:52:43.657012+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648054,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10289,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10193","reported_seq":10310,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.654615+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:47.654615+0000","last_peered":"2024-09-18T11:52:47.654615+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:47.654615+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21420691,"num_objects":2616,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10192,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10234","reported_seq":10328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.657410+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:43.657410+0000","last_peered":"2024-09-18T11:52:43.657410+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:43.657410+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22050038,"num_objects":2692,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10232,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10223","reported_seq":10344,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.597154+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:47.597154+0000","last_peered":"2024-09-18T11:52:47.597154+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:47.597154+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21886198,"num_objects":2672,"num_object_clones":0,"num_object_copies":10688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10221,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10208","reported_seq":10326,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.694378+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:47.694378+0000","last_peered":"2024-09-18T11:52:47.694378+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:47.694378+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21118976,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":459}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10320","reported_seq":10426,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.746342+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:43.746342+0000","last_peered":"2024-09-18T11:52:43.746342+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:43.746342+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22082806,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10318,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":580}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10118","reported_seq":10382,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.719649+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:43.719649+0000","last_peered":"2024-09-18T11:52:43.719649+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:43.719649+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22582518,"num_objects":2757,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10116,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10485","reported_seq":14838,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:47.701825+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:47.701825+0000","last_peered":"2024-09-18T11:52:47.701825+0000","last_clean":"2024-09-18T11:52:47.701825+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:47.701825+0000","last_undegraded":"2024-09-18T11:52:47.701825+0000","last_fullsized":"2024-09-18T11:52:47.701825+0000","mapping_epoch":36,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22039020,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10482,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9997","reported_seq":10111,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.846876+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:47.846876+0000","last_peered":"2024-09-18T11:52:47.846876+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:47.846876+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9997,"log_dups_size":0,"ondisk_log_size":9997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21271798,"num_objects":2597,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9995,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10327","reported_seq":10446,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.607759+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:47.607759+0000","last_peered":"2024-09-18T11:52:47.607759+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:47.607759+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163313,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10403","reported_seq":10496,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.503682+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:47.503682+0000","last_peered":"2024-09-18T11:52:47.503682+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:47.503682+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21527163,"num_objects":2628,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10402,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10304","reported_seq":10402,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.900590+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:47.900590+0000","last_peered":"2024-09-18T11:52:47.900590+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:47.900590+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21813883,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10399","reported_seq":10524,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.613274+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:47.613274+0000","last_peered":"2024-09-18T11:52:47.613274+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:47.613274+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22041846,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10397,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":568}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10176","reported_seq":14465,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:43.646753+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:43.646753+0000","last_peered":"2024-09-18T11:52:43.646753+0000","last_clean":"2024-09-18T11:52:43.646753+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:43.646753+0000","last_undegraded":"2024-09-18T11:52:43.646753+0000","last_fullsized":"2024-09-18T11:52:43.646753+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22336758,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10174,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10053","reported_seq":10142,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:43.718738+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:43.718738+0000","last_peered":"2024-09-18T11:52:43.718738+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:43.718738+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21738742,"num_objects":2654,"num_object_clones":0,"num_object_copies":10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10051,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":350963084,"num_objects":42848,"num_object_clones":0,"num_object_copies":171392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164239,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6716,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":704004096,"data_stored":704004096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160666,"ondisk_log_size":160666,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769692,"kb_used_data":170964,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93602148,"statfs":{"total":96636764160,"available":95848599552,"internally_reserved":0,"allocated":175067136,"data_stored":174883315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,8,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.992000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":11.023999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":17.117999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.9880000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":1.99},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":6.0199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760340,"kb_used_data":170444,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93611500,"statfs":{"total":96636764160,"available":95858176000,"internally_reserved":0,"allocated":174534656,"data_stored":174350835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":0.69299999999999995},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":5.8639999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.2279999999999998},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":6.5259999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476368,"kb_used_data":174408,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93895472,"statfs":{"total":96636764160,"available":96148963328,"internally_reserved":0,"allocated":178593792,"data_stored":178409971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":12.708},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":12.577999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":9.9529999999999994},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":4.1379999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":9.9920000000000009}]}]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772672,"kb_used_data":174000,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93599168,"statfs":{"total":96636764160,"available":95845548032,"internally_reserved":0,"allocated":178176000,"data_stored":177992179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.5169999999999999},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.27800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":1.5960000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":11.715999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":1.55},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":11.673999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":177594368,"data_stored":177594368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":178032640,"data_stored":178032640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174096384,"data_stored":174096384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":174280704,"data_stored":174280704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:51.639 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:52.277 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:52.277 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:52.307 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-18T11:52:52.033682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":347891084,"num_objects":42473,"num_object_clones":0,"num_object_copies":169892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164614,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6765,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762620,"kb_used_data":681688,"kb_used_omap":14,"kb_used_meta":2080881,"kb_avail":374724740,"statfs":{"total":386547056640,"available":383718133760,"internally_reserved":0,"allocated":698048512,"data_stored":697313228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130822171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,17,3,19,14,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16867328,"num_objects":-2059,"num_object_clones":0,"num_object_copies":-8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002976"},"pg_stats":[{"pgid":"1.f","version":"37'10535","reported_seq":10636,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.646680+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:47.646680+0000","last_peered":"2024-09-18T11:52:47.646680+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:47.646680+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22241280,"num_objects":2715,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10346","reported_seq":14638,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:48.796234+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:48.796234+0000","last_peered":"2024-09-18T11:52:48.796234+0000","last_clean":"2024-09-18T11:52:48.796234+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:48.796234+0000","last_undegraded":"2024-09-18T11:52:48.796234+0000","last_fullsized":"2024-09-18T11:52:48.796234+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22197494,"num_objects":2710,"num_object_clones":0,"num_object_copies":10840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10344,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10193","reported_seq":10310,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.654615+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:47.654615+0000","last_peered":"2024-09-18T11:52:47.654615+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:47.654615+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21420691,"num_objects":2616,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10192,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10284","reported_seq":10378,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:48.663938+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:48.663938+0000","last_peered":"2024-09-18T11:52:48.663938+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:48.663938+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21640438,"num_objects":2642,"num_object_clones":0,"num_object_copies":10568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10282,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10223","reported_seq":10344,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.597154+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:47.597154+0000","last_peered":"2024-09-18T11:52:47.597154+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:47.597154+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21886198,"num_objects":2672,"num_object_clones":0,"num_object_copies":10688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10221,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10208","reported_seq":10326,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.694378+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:47.694378+0000","last_peered":"2024-09-18T11:52:47.694378+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:47.694378+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21118976,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":459}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10382","reported_seq":10489,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:48.594119+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:48.594119+0000","last_peered":"2024-09-18T11:52:48.594119+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:48.594119+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574902,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10380,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10186","reported_seq":10548,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:48.800141+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:48.800141+0000","last_peered":"2024-09-18T11:52:48.800141+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:48.800141+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22025462,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10184,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10485","reported_seq":14838,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:47.701825+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:47.701825+0000","last_peered":"2024-09-18T11:52:47.701825+0000","last_clean":"2024-09-18T11:52:47.701825+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:47.701825+0000","last_undegraded":"2024-09-18T11:52:47.701825+0000","last_fullsized":"2024-09-18T11:52:47.701825+0000","mapping_epoch":36,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22039020,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10482,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9997","reported_seq":10111,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.846876+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:47.846876+0000","last_peered":"2024-09-18T11:52:47.846876+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:47.846876+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9997,"log_dups_size":0,"ondisk_log_size":9997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21271798,"num_objects":2597,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9995,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10327","reported_seq":10446,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.607759+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:47.607759+0000","last_peered":"2024-09-18T11:52:47.607759+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:47.607759+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163313,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10403","reported_seq":10496,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.503682+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:47.503682+0000","last_peered":"2024-09-18T11:52:47.503682+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:47.503682+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21527163,"num_objects":2628,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10402,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10304","reported_seq":10402,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.900590+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:47.900590+0000","last_peered":"2024-09-18T11:52:47.900590+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:47.900590+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21813883,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10399","reported_seq":10524,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.613274+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:47.613274+0000","last_peered":"2024-09-18T11:52:47.613274+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:47.613274+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22041846,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10397,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":568}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10253","reported_seq":14542,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:48.573390+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:48.573390+0000","last_peered":"2024-09-18T11:52:48.573390+0000","last_clean":"2024-09-18T11:52:48.573390+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:48.573390+0000","last_undegraded":"2024-09-18T11:52:48.573390+0000","last_fullsized":"2024-09-18T11:52:48.573390+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21705974,"num_objects":2650,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10251,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10116","reported_seq":10206,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:48.764426+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:48.764426+0000","last_peered":"2024-09-18T11:52:48.764426+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:48.764426+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21222646,"num_objects":2591,"num_object_clones":0,"num_object_copies":10364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10114,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":589}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":347891084,"num_objects":42473,"num_object_clones":0,"num_object_copies":169892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164614,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6765,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":695554048,"data_stored":695554048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769692,"kb_used_data":170964,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93602148,"statfs":{"total":96636764160,"available":95848599552,"internally_reserved":0,"allocated":175067136,"data_stored":174883315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,8,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.992000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":11.023999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":17.117999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.9880000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":1.99},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":6.0199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760340,"kb_used_data":170444,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93611500,"statfs":{"total":96636764160,"available":95858176000,"internally_reserved":0,"allocated":174534656,"data_stored":174350835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":0.69299999999999995},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":5.8639999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.2279999999999998},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":6.5259999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476416,"kb_used_data":170296,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93895424,"statfs":{"total":96636764160,"available":96148914176,"internally_reserved":0,"allocated":174383104,"data_stored":174199283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.443},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.8399999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":0.48199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.216},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.42099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756172,"kb_used_data":169984,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93615668,"statfs":{"total":96636764160,"available":95862444032,"internally_reserved":0,"allocated":174063616,"data_stored":173879795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":9.6980000000000004},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":1.643}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.4939999999999998},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":1.8939999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":17.925999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":17.396999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":173420544,"data_stored":173420544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173756416,"data_stored":173756416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174096384,"data_stored":174096384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":174280704,"data_stored":174280704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:52.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:52.902 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:52.903 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:52.921 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-18T11:52:52.033682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":347891084,"num_objects":42473,"num_object_clones":0,"num_object_copies":169892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164614,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6765,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762620,"kb_used_data":681688,"kb_used_omap":14,"kb_used_meta":2080881,"kb_avail":374724740,"statfs":{"total":386547056640,"available":383718133760,"internally_reserved":0,"allocated":698048512,"data_stored":697313228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130822171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,17,3,19,14,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16867328,"num_objects":-2059,"num_object_clones":0,"num_object_copies":-8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002976"},"pg_stats":[{"pgid":"1.f","version":"37'10535","reported_seq":10636,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.646680+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:47.646680+0000","last_peered":"2024-09-18T11:52:47.646680+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:47.646680+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22241280,"num_objects":2715,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10346","reported_seq":14638,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:48.796234+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:48.796234+0000","last_peered":"2024-09-18T11:52:48.796234+0000","last_clean":"2024-09-18T11:52:48.796234+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:48.796234+0000","last_undegraded":"2024-09-18T11:52:48.796234+0000","last_fullsized":"2024-09-18T11:52:48.796234+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22197494,"num_objects":2710,"num_object_clones":0,"num_object_copies":10840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10344,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10193","reported_seq":10310,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.654615+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:47.654615+0000","last_peered":"2024-09-18T11:52:47.654615+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:47.654615+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21420691,"num_objects":2616,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10192,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10284","reported_seq":10378,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:48.663938+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:48.663938+0000","last_peered":"2024-09-18T11:52:48.663938+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:48.663938+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21640438,"num_objects":2642,"num_object_clones":0,"num_object_copies":10568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10282,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10223","reported_seq":10344,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.597154+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:47.597154+0000","last_peered":"2024-09-18T11:52:47.597154+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:47.597154+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21886198,"num_objects":2672,"num_object_clones":0,"num_object_copies":10688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10221,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10208","reported_seq":10326,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.694378+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:47.694378+0000","last_peered":"2024-09-18T11:52:47.694378+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:47.694378+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21118976,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":459}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10382","reported_seq":10489,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:48.594119+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:48.594119+0000","last_peered":"2024-09-18T11:52:48.594119+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:48.594119+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574902,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10380,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10186","reported_seq":10548,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:48.800141+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:48.800141+0000","last_peered":"2024-09-18T11:52:48.800141+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:48.800141+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22025462,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10184,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10485","reported_seq":14838,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:47.701825+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:47.701825+0000","last_peered":"2024-09-18T11:52:47.701825+0000","last_clean":"2024-09-18T11:52:47.701825+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:47.701825+0000","last_undegraded":"2024-09-18T11:52:47.701825+0000","last_fullsized":"2024-09-18T11:52:47.701825+0000","mapping_epoch":36,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22039020,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10482,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9997","reported_seq":10111,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.846876+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:47.846876+0000","last_peered":"2024-09-18T11:52:47.846876+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:47.846876+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":9997,"log_dups_size":0,"ondisk_log_size":9997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21271798,"num_objects":2597,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9995,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10327","reported_seq":10446,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.607759+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:47.607759+0000","last_peered":"2024-09-18T11:52:47.607759+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:47.607759+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163313,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10403","reported_seq":10496,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.503682+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:47.503682+0000","last_peered":"2024-09-18T11:52:47.503682+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:47.503682+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21527163,"num_objects":2628,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10402,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10304","reported_seq":10402,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.900590+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:47.900590+0000","last_peered":"2024-09-18T11:52:47.900590+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:47.900590+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21813883,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10399","reported_seq":10524,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:47.613274+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:47.613274+0000","last_peered":"2024-09-18T11:52:47.613274+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:47.613274+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22041846,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10397,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":568}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10253","reported_seq":14542,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:48.573390+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:48.573390+0000","last_peered":"2024-09-18T11:52:48.573390+0000","last_clean":"2024-09-18T11:52:48.573390+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:48.573390+0000","last_undegraded":"2024-09-18T11:52:48.573390+0000","last_fullsized":"2024-09-18T11:52:48.573390+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21705974,"num_objects":2650,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10251,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10116","reported_seq":10206,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:48.764426+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:48.764426+0000","last_peered":"2024-09-18T11:52:48.764426+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:48.764426+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21222646,"num_objects":2591,"num_object_clones":0,"num_object_copies":10364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10114,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":589}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":347891084,"num_objects":42473,"num_object_clones":0,"num_object_copies":169892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164614,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6765,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":695554048,"data_stored":695554048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769692,"kb_used_data":170964,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93602148,"statfs":{"total":96636764160,"available":95848599552,"internally_reserved":0,"allocated":175067136,"data_stored":174883315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,8,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":10.992000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":11.023999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":17.117999999999999},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":5.9880000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":1.99},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":6.0199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760340,"kb_used_data":170444,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93611500,"statfs":{"total":96636764160,"available":95858176000,"internally_reserved":0,"allocated":174534656,"data_stored":174350835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":0.69299999999999995},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":5.8639999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.2279999999999998},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":6.5259999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476416,"kb_used_data":170296,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93895424,"statfs":{"total":96636764160,"available":96148914176,"internally_reserved":0,"allocated":174383104,"data_stored":174199283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.443},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":6.8399999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":0.48199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.216},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":0.42099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756172,"kb_used_data":169984,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93615668,"statfs":{"total":96636764160,"available":95862444032,"internally_reserved":0,"allocated":174063616,"data_stored":173879795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":9.6980000000000004},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":1.643}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.4939999999999998},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":1.8939999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":17.925999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":17.396999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":173420544,"data_stored":173420544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173756416,"data_stored":173756416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174096384,"data_stored":174096384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":174280704,"data_stored":174280704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:52.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:52.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10535", 'reported_seq': 10636, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:47.646680+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:47.646680+0000', 'last_peered': '2024-09-18T11:52:47.646680+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:47.646680+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22241280, 'num_objects': 2715, 'num_object_clones': 0, 'num_object_copies': 10860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10535, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 612}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:52.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:52.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10193", 'reported_seq': 10310, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:47.654615+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:47.654615+0000', 'last_peered': '2024-09-18T11:52:47.654615+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:47.654615+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21420691, 'num_objects': 2616, 'num_object_clones': 0, 'num_object_copies': 10464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2616, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10192, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 576}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:52.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:52.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10284", 'reported_seq': 10378, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:48.663938+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:48.663938+0000', 'last_peered': '2024-09-18T11:52:48.663938+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:48.663938+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21640438, 'num_objects': 2642, 'num_object_clones': 0, 'num_object_copies': 10568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10282, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 541}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:52.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:52.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10223", 'reported_seq': 10344, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:47.597154+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:47.597154+0000', 'last_peered': '2024-09-18T11:52:47.597154+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:47.597154+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21886198, 'num_objects': 2672, 'num_object_clones': 0, 'num_object_copies': 10688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10221, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 620}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:52.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:52.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10208", 'reported_seq': 10326, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:47.694378+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:47.694378+0000', 'last_peered': '2024-09-18T11:52:47.694378+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:47.694378+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21118976, 'num_objects': 2578, 'num_object_clones': 0, 'num_object_copies': 10312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10208, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 459}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:52.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:52.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10382", 'reported_seq': 10489, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:48.594119+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:48.594119+0000', 'last_peered': '2024-09-18T11:52:48.594119+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:48.594119+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21574902, 'num_objects': 2634, 'num_object_clones': 0, 'num_object_copies': 10536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10380, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 518}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:52.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:52.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10186", 'reported_seq': 10548, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:48.800141+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:52:48.800141+0000', 'last_peered': '2024-09-18T11:52:48.800141+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:48.800141+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22025462, 'num_objects': 2689, 'num_object_clones': 0, 'num_object_copies': 10756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10184, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 148, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 799}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:52.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:52.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9997", 'reported_seq': 10111, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:47.846876+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:47.846876+0000', 'last_peered': '2024-09-18T11:52:47.846876+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:47.846876+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 9997, 'log_dups_size': 0, 'ondisk_log_size': 9997, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21271798, 'num_objects': 2597, 'num_object_clones': 0, 'num_object_copies': 10388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9995, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 634}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:52.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:52.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10327", 'reported_seq': 10446, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:47.607759+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:47.607759+0000', 'last_peered': '2024-09-18T11:52:47.607759+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:47.607759+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22163313, 'num_objects': 2706, 'num_object_clones': 0, 'num_object_copies': 10824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10324, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 672}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:52.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:52.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10403", 'reported_seq': 10496, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:47.503682+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:47.503682+0000', 'last_peered': '2024-09-18T11:52:47.503682+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:47.503682+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21527163, 'num_objects': 2628, 'num_object_clones': 0, 'num_object_copies': 10512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10402, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 544}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:52.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:52.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10304", 'reported_seq': 10402, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:47.900590+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:47.900590+0000', 'last_peered': '2024-09-18T11:52:47.900590+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:47.900590+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21813883, 'num_objects': 2663, 'num_object_clones': 0, 'num_object_copies': 10652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10303, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:52.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:52.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10399", 'reported_seq': 10524, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:47.613274+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:47.613274+0000', 'last_peered': '2024-09-18T11:52:47.613274+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:47.613274+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22041846, 'num_objects': 2691, 'num_object_clones': 0, 'num_object_copies': 10764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10397, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 568}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:52.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:52.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10116", 'reported_seq': 10206, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:48.764426+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:48.764426+0000', 'last_peered': '2024-09-18T11:52:48.764426+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:48.764426+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21222646, 'num_objects': 2591, 'num_object_clones': 0, 'num_object_copies': 10364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10114, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 589}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:52.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:53.617 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:53.617 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26691/169892 objects degraded (15.711%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":571,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42473,"data_bytes":347891084,"bytes_used":2828922880,"bytes_avail":383718133760,"bytes_total":386547056640,"degraded_objects":26691,"degraded_total":169892,"degraded_ratio":0.15710569067407529,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":171},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:53.652 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:53.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:54.255 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:54.255 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:54.281 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-18T11:52:54.034668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339486092,"num_objects":41447,"num_object_clones":0,"num_object_copies":165788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165640,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6820,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160967,"ondisk_log_size":160967,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758684,"kb_used_data":665268,"kb_used_omap":14,"kb_used_meta":2093361,"kb_avail":374728676,"statfs":{"total":386547056640,"available":383722164224,"internally_reserved":0,"allocated":681234432,"data_stored":680499148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143601691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,8,9,12,11,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25272320,"num_objects":-3085,"num_object_clones":0,"num_object_copies":-12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3085,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003539"},"pg_stats":[{"pgid":"1.f","version":"37'10587","reported_seq":10688,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.723219+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:52.723219+0000","last_peered":"2024-09-18T11:52:52.723219+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:52.723219+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21815296,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10419","reported_seq":14710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:53.637491+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:53.637491+0000","last_peered":"2024-09-18T11:52:53.637491+0000","last_clean":"2024-09-18T11:52:53.637491+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:53.637491+0000","last_undegraded":"2024-09-18T11:52:53.637491+0000","last_fullsized":"2024-09-18T11:52:53.637491+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21599478,"num_objects":2637,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10417,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10269","reported_seq":10386,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.889751+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:52.889751+0000","last_peered":"2024-09-18T11:52:52.889751+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:52.889751+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20798099,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10268,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":500}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10348","reported_seq":10442,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.746857+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:53.746857+0000","last_peered":"2024-09-18T11:52:53.746857+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:53.746857+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21116150,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10293","reported_seq":10414,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.645290+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:52.645290+0000","last_peered":"2024-09-18T11:52:52.645290+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:52.645290+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21312758,"num_objects":2602,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":550}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10273","reported_seq":10392,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.582684+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:52.582684+0000","last_peered":"2024-09-18T11:52:52.582684+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:52.582684+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20586496,"num_objects":2513,"num_object_clones":0,"num_object_copies":10052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10444","reported_seq":10551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.690434+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:53.690434+0000","last_peered":"2024-09-18T11:52:53.690434+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:53.690434+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21066998,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10442,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":456}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10241","reported_seq":10713,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.817476+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:53.817476+0000","last_peered":"2024-09-18T11:52:53.817476+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:53.817476+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574902,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10239,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10550","reported_seq":14903,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:52.720576+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:52.720576+0000","last_peered":"2024-09-18T11:52:52.720576+0000","last_clean":"2024-09-18T11:52:52.720576+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:52.720576+0000","last_undegraded":"2024-09-18T11:52:52.720576+0000","last_fullsized":"2024-09-18T11:52:52.720576+0000","mapping_epoch":36,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21506540,"num_objects":2626,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10547,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10078","reported_seq":10192,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.901857+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:52.901857+0000","last_peered":"2024-09-18T11:52:52.901857+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:52.901857+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20608246,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":553}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10390","reported_seq":10511,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.528026+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:52.528026+0000","last_peered":"2024-09-18T11:52:52.528026+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:52.528026+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21647217,"num_objects":2643,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10467","reported_seq":10560,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.724322+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:52.724322+0000","last_peered":"2024-09-18T11:52:52.724322+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:52.724322+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21002875,"num_objects":2564,"num_object_clones":0,"num_object_copies":10256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10466,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10369","reported_seq":10466,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.877436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:52.877436+0000","last_peered":"2024-09-18T11:52:52.877436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:52.877436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21281403,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10462","reported_seq":10587,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.645259+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:52.645259+0000","last_peered":"2024-09-18T11:52:52.645259+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:52.645259+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21525750,"num_objects":2628,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10460,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10316","reported_seq":14605,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:53.888873+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:53.888873+0000","last_peered":"2024-09-18T11:52:53.888873+0000","last_clean":"2024-09-18T11:52:53.888873+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:53.888873+0000","last_undegraded":"2024-09-18T11:52:53.888873+0000","last_fullsized":"2024-09-18T11:52:53.888873+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189878,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10314,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10161","reported_seq":10251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.759770+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:53.759770+0000","last_peered":"2024-09-18T11:52:53.759770+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:53.759770+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20854006,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339486092,"num_objects":41447,"num_object_clones":0,"num_object_copies":165788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165640,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6820,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":678600704,"data_stored":678600704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160967,"ondisk_log_size":160967,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765580,"kb_used_data":166852,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93606260,"statfs":{"total":96636764160,"available":95852810240,"internally_reserved":0,"allocated":170856448,"data_stored":170672627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,3,1,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.63},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.60599999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":3.7730000000000001},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":4.5449999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":3.7549999999999999},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.29699999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760392,"kb_used_data":166336,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93611448,"statfs":{"total":96636764160,"available":95858122752,"internally_reserved":0,"allocated":170328064,"data_stored":170144243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":0.69299999999999995},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":5.8639999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.2279999999999998},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":6.5259999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476472,"kb_used_data":166184,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93895368,"statfs":{"total":96636764160,"available":96148856832,"internally_reserved":0,"allocated":170172416,"data_stored":169988595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.468},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":3.4820000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":11.256}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.1479999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756240,"kb_used_data":165896,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93615600,"statfs":{"total":96636764160,"available":95862374400,"internally_reserved":0,"allocated":169877504,"data_stored":169693683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":9.6980000000000004},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":1.643}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.4939999999999998},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":1.8939999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":17.925999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":17.396999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":169201664,"data_stored":169201664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":169496576,"data_stored":169496576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169869312,"data_stored":169869312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":170033152,"data_stored":170033152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:57.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:57.911 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-18T11:52:56.036331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339486092,"num_objects":41447,"num_object_clones":0,"num_object_copies":165788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165640,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6820,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160967,"ondisk_log_size":160967,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758684,"kb_used_data":665268,"kb_used_omap":14,"kb_used_meta":2093361,"kb_avail":374728676,"statfs":{"total":386547056640,"available":383722164224,"internally_reserved":0,"allocated":681234432,"data_stored":680499148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143601691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,8,9,12,11,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16826368,"num_objects":-2054,"num_object_clones":0,"num_object_copies":-8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004792"},"pg_stats":[{"pgid":"1.f","version":"37'10587","reported_seq":10688,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.723219+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:52.723219+0000","last_peered":"2024-09-18T11:52:52.723219+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:52.723219+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21815296,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10419","reported_seq":14710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:53.637491+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:53.637491+0000","last_peered":"2024-09-18T11:52:53.637491+0000","last_clean":"2024-09-18T11:52:53.637491+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:53.637491+0000","last_undegraded":"2024-09-18T11:52:53.637491+0000","last_fullsized":"2024-09-18T11:52:53.637491+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21599478,"num_objects":2637,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10417,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10269","reported_seq":10386,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.889751+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:52.889751+0000","last_peered":"2024-09-18T11:52:52.889751+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:52.889751+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20798099,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10268,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":500}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10348","reported_seq":10442,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.746857+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:53.746857+0000","last_peered":"2024-09-18T11:52:53.746857+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:53.746857+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21116150,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10293","reported_seq":10414,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.645290+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:52.645290+0000","last_peered":"2024-09-18T11:52:52.645290+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:52.645290+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21312758,"num_objects":2602,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":550}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10273","reported_seq":10392,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.582684+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:52.582684+0000","last_peered":"2024-09-18T11:52:52.582684+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:52.582684+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20586496,"num_objects":2513,"num_object_clones":0,"num_object_copies":10052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10444","reported_seq":10551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.690434+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:53.690434+0000","last_peered":"2024-09-18T11:52:53.690434+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:53.690434+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21066998,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10442,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":456}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10241","reported_seq":10713,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.817476+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:53.817476+0000","last_peered":"2024-09-18T11:52:53.817476+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:53.817476+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574902,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10239,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10550","reported_seq":14903,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:52.720576+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:52.720576+0000","last_peered":"2024-09-18T11:52:52.720576+0000","last_clean":"2024-09-18T11:52:52.720576+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:52.720576+0000","last_undegraded":"2024-09-18T11:52:52.720576+0000","last_fullsized":"2024-09-18T11:52:52.720576+0000","mapping_epoch":36,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21506540,"num_objects":2626,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10547,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10078","reported_seq":10192,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.901857+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:52.901857+0000","last_peered":"2024-09-18T11:52:52.901857+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:52.901857+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20608246,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":553}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10390","reported_seq":10511,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.528026+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:52.528026+0000","last_peered":"2024-09-18T11:52:52.528026+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:52.528026+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21647217,"num_objects":2643,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10467","reported_seq":10560,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.724322+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:52.724322+0000","last_peered":"2024-09-18T11:52:52.724322+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:52.724322+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21002875,"num_objects":2564,"num_object_clones":0,"num_object_copies":10256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10466,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10369","reported_seq":10466,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.877436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:52.877436+0000","last_peered":"2024-09-18T11:52:52.877436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:52.877436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21281403,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10462","reported_seq":10587,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.645259+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:52.645259+0000","last_peered":"2024-09-18T11:52:52.645259+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:52.645259+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21525750,"num_objects":2628,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10460,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10316","reported_seq":14605,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:53.888873+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:53.888873+0000","last_peered":"2024-09-18T11:52:53.888873+0000","last_clean":"2024-09-18T11:52:53.888873+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:53.888873+0000","last_undegraded":"2024-09-18T11:52:53.888873+0000","last_fullsized":"2024-09-18T11:52:53.888873+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189878,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10314,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10161","reported_seq":10251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.759770+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:53.759770+0000","last_peered":"2024-09-18T11:52:53.759770+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:53.759770+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20854006,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339486092,"num_objects":41447,"num_object_clones":0,"num_object_copies":165788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165640,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6820,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":678600704,"data_stored":678600704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160967,"ondisk_log_size":160967,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765580,"kb_used_data":166852,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93606260,"statfs":{"total":96636764160,"available":95852810240,"internally_reserved":0,"allocated":170856448,"data_stored":170672627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,3,1,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":0.63},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.60599999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:51:54 2024","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":6.8449999999999998,"15min":5.8680000000000003},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":3.7730000000000001},{"interface":"front","average":{"1min":10.57,"5min":5.9550000000000001,"15min":4.7779999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":57.372,"5min":57.372,"15min":57.372},"last":4.5449999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":3.7549999999999999},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.29699999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760392,"kb_used_data":166336,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93611448,"statfs":{"total":96636764160,"available":95858122752,"internally_reserved":0,"allocated":170328064,"data_stored":170144243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":0.69299999999999995},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":5.8639999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.2279999999999998},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":6.5259999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476472,"kb_used_data":166184,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93895368,"statfs":{"total":96636764160,"available":96148856832,"internally_reserved":0,"allocated":170172416,"data_stored":169988595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.468},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":3.4820000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":11.256}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.1479999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756240,"kb_used_data":165896,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93615600,"statfs":{"total":96636764160,"available":95862374400,"internally_reserved":0,"allocated":169877504,"data_stored":169693683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":9.6980000000000004},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":1.643}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.4939999999999998},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":1.8939999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":17.925999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":17.396999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":169201664,"data_stored":169201664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":169496576,"data_stored":169496576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169869312,"data_stored":169869312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":170033152,"data_stored":170033152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:57.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:58.484 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:58.485 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:58.514 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-09-18T11:52:58.036743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":335840652,"num_objects":41002,"num_object_clones":0,"num_object_copies":164008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166085,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6820,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742144,"kb_used_data":661196,"kb_used_omap":14,"kb_used_meta":2080881,"kb_avail":374745216,"statfs":{"total":386547056640,"available":383739101184,"internally_reserved":0,"allocated":677064704,"data_stored":676329420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130822171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,9,9,19,13,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20471808,"num_objects":-2499,"num_object_clones":0,"num_object_copies":-9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004710"},"pg_stats":[{"pgid":"1.f","version":"37'10637","reported_seq":10738,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.694405+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:57.694405+0000","last_peered":"2024-09-18T11:52:57.694405+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:57.694405+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21405696,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10637,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10419","reported_seq":14710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:53.637491+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:53.637491+0000","last_peered":"2024-09-18T11:52:53.637491+0000","last_clean":"2024-09-18T11:52:53.637491+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:53.637491+0000","last_undegraded":"2024-09-18T11:52:53.637491+0000","last_fullsized":"2024-09-18T11:52:53.637491+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21599478,"num_objects":2637,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10417,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10269","reported_seq":10386,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.889751+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:52.889751+0000","last_peered":"2024-09-18T11:52:52.889751+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:52.889751+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20798099,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10268,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":500}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10348","reported_seq":10442,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.746857+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:53.746857+0000","last_peered":"2024-09-18T11:52:53.746857+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:53.746857+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21116150,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10366","reported_seq":10487,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.686945+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:57.686945+0000","last_peered":"2024-09-18T11:52:57.686945+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:57.686945+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20714742,"num_objects":2529,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10334","reported_seq":10453,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.677227+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:57.677227+0000","last_peered":"2024-09-18T11:52:57.677227+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:57.677227+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10334,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10444","reported_seq":10551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.690434+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:53.690434+0000","last_peered":"2024-09-18T11:52:53.690434+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:53.690434+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21066998,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10442,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":456}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10241","reported_seq":10713,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.817476+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:53.817476+0000","last_peered":"2024-09-18T11:52:53.817476+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:53.817476+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574902,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10239,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10616","reported_seq":14969,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:57.562393+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:57.562393+0000","last_peered":"2024-09-18T11:52:57.562393+0000","last_clean":"2024-09-18T11:52:57.562393+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:57.562393+0000","last_undegraded":"2024-09-18T11:52:57.562393+0000","last_fullsized":"2024-09-18T11:52:57.562393+0000","mapping_epoch":36,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20965868,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10078","reported_seq":10192,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.901857+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:52.901857+0000","last_peered":"2024-09-18T11:52:52.901857+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:52.901857+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20608246,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":553}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10462","reported_seq":10581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.829283+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:57.829283+0000","last_peered":"2024-09-18T11:52:57.829283+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:57.829283+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21057393,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10529","reported_seq":10619,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.651185+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:57.651185+0000","last_peered":"2024-09-18T11:52:57.651185+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:57.651185+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20494971,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":418}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10369","reported_seq":10466,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.877436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:52.877436+0000","last_peered":"2024-09-18T11:52:52.877436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:52.877436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21281403,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10523","reported_seq":10648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.697307+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:57.697307+0000","last_peered":"2024-09-18T11:52:57.697307+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:57.697307+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026038,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10521,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10316","reported_seq":14605,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:53.888873+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:53.888873+0000","last_peered":"2024-09-18T11:52:53.888873+0000","last_clean":"2024-09-18T11:52:53.888873+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:53.888873+0000","last_undegraded":"2024-09-18T11:52:53.888873+0000","last_fullsized":"2024-09-18T11:52:53.888873+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189878,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10314,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10161","reported_seq":10251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.759770+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:53.759770+0000","last_peered":"2024-09-18T11:52:53.759770+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:53.759770+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20854006,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":335840652,"num_objects":41002,"num_object_clones":0,"num_object_copies":164008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166085,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6820,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":674357248,"data_stored":674357248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749040,"kb_used_data":162780,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93622800,"statfs":{"total":96636764160,"available":95869747200,"internally_reserved":0,"allocated":166686720,"data_stored":166502899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":19.091999999999999},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":21.518999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":19.024000000000001},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":12.145}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.432},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":12.179}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760392,"kb_used_data":166336,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93611448,"statfs":{"total":96636764160,"available":95858122752,"internally_reserved":0,"allocated":170328064,"data_stored":170144243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":0.69299999999999995},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":5.8639999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.2279999999999998},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":6.5259999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476472,"kb_used_data":166184,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93895368,"statfs":{"total":96636764160,"available":96148856832,"internally_reserved":0,"allocated":170172416,"data_stored":169988595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.468},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":3.4820000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":11.256}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.1479999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756240,"kb_used_data":165896,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93615600,"statfs":{"total":96636764160,"available":95862374400,"internally_reserved":0,"allocated":169877504,"data_stored":169693683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":9.6980000000000004},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":1.643}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.4939999999999998},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":1.8939999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":17.925999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":17.396999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":169201664,"data_stored":169201664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":169496576,"data_stored":169496576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169869312,"data_stored":169869312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165789696,"data_stored":165789696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:52:58.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:52:58.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10637", 'reported_seq': 10738, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.694405+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:57.694405+0000', 'last_peered': '2024-09-18T11:52:57.694405+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:57.694405+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21405696, 'num_objects': 2613, 'num_object_clones': 0, 'num_object_copies': 10452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10637, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 510}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:58.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:52:58.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10269", 'reported_seq': 10386, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:52.889751+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:52.889751+0000', 'last_peered': '2024-09-18T11:52:52.889751+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:52.889751+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20798099, 'num_objects': 2540, 'num_object_clones': 0, 'num_object_copies': 10160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2540, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10268, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 500}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:58.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:52:58.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10348", 'reported_seq': 10442, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:53.746857+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:53.746857+0000', 'last_peered': '2024-09-18T11:52:53.746857+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:53.746857+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21116150, 'num_objects': 2578, 'num_object_clones': 0, 'num_object_copies': 10312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10346, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:58.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:52:58.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10366", 'reported_seq': 10487, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.686945+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:57.686945+0000', 'last_peered': '2024-09-18T11:52:57.686945+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:57.686945+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20714742, 'num_objects': 2529, 'num_object_clones': 0, 'num_object_copies': 10116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10364, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 477}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:58.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:52:58.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10334", 'reported_seq': 10453, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.677227+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:57.677227+0000', 'last_peered': '2024-09-18T11:52:57.677227+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:57.677227+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20086784, 'num_objects': 2452, 'num_object_clones': 0, 'num_object_copies': 9808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10334, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 333}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:58.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:52:58.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10444", 'reported_seq': 10551, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:53.690434+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:53.690434+0000', 'last_peered': '2024-09-18T11:52:53.690434+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:53.690434+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21066998, 'num_objects': 2572, 'num_object_clones': 0, 'num_object_copies': 10288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10442, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 456}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:58.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:52:58.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10241", 'reported_seq': 10713, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:53.817476+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:52:53.817476+0000', 'last_peered': '2024-09-18T11:52:53.817476+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:53.817476+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21574902, 'num_objects': 2634, 'num_object_clones': 0, 'num_object_copies': 10536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1835, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10239, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 799}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:58.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:52:58.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10078", 'reported_seq': 10192, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:52.901857+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:52.901857+0000', 'last_peered': '2024-09-18T11:52:52.901857+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:52.901857+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20608246, 'num_objects': 2516, 'num_object_clones': 0, 'num_object_copies': 10064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10076, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 553}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:58.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:52:58.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10462", 'reported_seq': 10581, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.829283+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:57.829283+0000', 'last_peered': '2024-09-18T11:52:57.829283+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:57.829283+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21057393, 'num_objects': 2571, 'num_object_clones': 0, 'num_object_copies': 10284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2571, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10459, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 537}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:58.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:52:58.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10529", 'reported_seq': 10619, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.651185+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:57.651185+0000', 'last_peered': '2024-09-18T11:52:57.651185+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:57.651185+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20494971, 'num_objects': 2502, 'num_object_clones': 0, 'num_object_copies': 10008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10528, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:58.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:52:58.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10369", 'reported_seq': 10466, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:52.877436+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:52.877436+0000', 'last_peered': '2024-09-18T11:52:52.877436+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:52.877436+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21281403, 'num_objects': 2598, 'num_object_clones': 0, 'num_object_copies': 10392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10368, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:52:58.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:52:58.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10523", 'reported_seq': 10648, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.697307+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:57.697307+0000', 'last_peered': '2024-09-18T11:52:57.697307+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:57.697307+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21026038, 'num_objects': 2567, 'num_object_clones': 0, 'num_object_copies': 10268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2567, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10521, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:52:58.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:52:58.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10161", 'reported_seq': 10251, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:53.759770+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:53.759770+0000', 'last_peered': '2024-09-18T11:52:53.759770+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:53.759770+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20854006, 'num_objects': 2546, 'num_object_clones': 0, 'num_object_copies': 10184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10159, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 544}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:52:58.518 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:52:59.229 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:59.229 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26636/164008 objects degraded (16.241%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":577,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41002,"data_bytes":335840652,"bytes_used":2807955456,"bytes_avail":383739101184,"bytes_total":386547056640,"degraded_objects":26636,"degraded_total":164008,"degraded_ratio":0.16240671186771377,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":208},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:50:53.985447+0000","services":{}},"progress_events":{}} 2024-09-18T11:52:59.255 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:52:59.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:52:59.837 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:52:59.837 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:52:59.856 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-09-18T11:52:58.036743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":335840652,"num_objects":41002,"num_object_clones":0,"num_object_copies":164008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166085,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6820,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742144,"kb_used_data":661196,"kb_used_omap":14,"kb_used_meta":2080881,"kb_avail":374745216,"statfs":{"total":386547056640,"available":383739101184,"internally_reserved":0,"allocated":677064704,"data_stored":676329420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130822171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,9,9,19,13,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20471808,"num_objects":-2499,"num_object_clones":0,"num_object_copies":-9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004710"},"pg_stats":[{"pgid":"1.f","version":"37'10637","reported_seq":10738,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.694405+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:57.694405+0000","last_peered":"2024-09-18T11:52:57.694405+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:57.694405+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21405696,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10637,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10419","reported_seq":14710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:53.637491+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:53.637491+0000","last_peered":"2024-09-18T11:52:53.637491+0000","last_clean":"2024-09-18T11:52:53.637491+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:53.637491+0000","last_undegraded":"2024-09-18T11:52:53.637491+0000","last_fullsized":"2024-09-18T11:52:53.637491+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21599478,"num_objects":2637,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10417,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10269","reported_seq":10386,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.889751+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:52.889751+0000","last_peered":"2024-09-18T11:52:52.889751+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:52.889751+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20798099,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10268,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":500}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10348","reported_seq":10442,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.746857+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:53.746857+0000","last_peered":"2024-09-18T11:52:53.746857+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:53.746857+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21116150,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10366","reported_seq":10487,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.686945+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:57.686945+0000","last_peered":"2024-09-18T11:52:57.686945+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:57.686945+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20714742,"num_objects":2529,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10334","reported_seq":10453,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.677227+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:57.677227+0000","last_peered":"2024-09-18T11:52:57.677227+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:57.677227+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10334,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10444","reported_seq":10551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.690434+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:53.690434+0000","last_peered":"2024-09-18T11:52:53.690434+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:53.690434+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21066998,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10442,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":456}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10241","reported_seq":10713,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.817476+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:53.817476+0000","last_peered":"2024-09-18T11:52:53.817476+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:53.817476+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574902,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10239,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10616","reported_seq":14969,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:57.562393+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:57.562393+0000","last_peered":"2024-09-18T11:52:57.562393+0000","last_clean":"2024-09-18T11:52:57.562393+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:57.562393+0000","last_undegraded":"2024-09-18T11:52:57.562393+0000","last_fullsized":"2024-09-18T11:52:57.562393+0000","mapping_epoch":36,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20965868,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10078","reported_seq":10192,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.901857+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:52.901857+0000","last_peered":"2024-09-18T11:52:52.901857+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:52.901857+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20608246,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":553}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10462","reported_seq":10581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.829283+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:57.829283+0000","last_peered":"2024-09-18T11:52:57.829283+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:57.829283+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21057393,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10529","reported_seq":10619,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.651185+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:57.651185+0000","last_peered":"2024-09-18T11:52:57.651185+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:57.651185+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20494971,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":418}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10369","reported_seq":10466,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:52.877436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:52.877436+0000","last_peered":"2024-09-18T11:52:52.877436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:52.877436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21281403,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10523","reported_seq":10648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.697307+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:57.697307+0000","last_peered":"2024-09-18T11:52:57.697307+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:57.697307+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026038,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10521,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10316","reported_seq":14605,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:53.888873+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:53.888873+0000","last_peered":"2024-09-18T11:52:53.888873+0000","last_clean":"2024-09-18T11:52:53.888873+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:53.888873+0000","last_undegraded":"2024-09-18T11:52:53.888873+0000","last_fullsized":"2024-09-18T11:52:53.888873+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189878,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10314,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10161","reported_seq":10251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:53.759770+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:53.759770+0000","last_peered":"2024-09-18T11:52:53.759770+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:53.759770+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20854006,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":335840652,"num_objects":41002,"num_object_clones":0,"num_object_copies":164008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166085,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6820,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":674357248,"data_stored":674357248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749040,"kb_used_data":162780,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93622800,"statfs":{"total":96636764160,"available":95869747200,"internally_reserved":0,"allocated":166686720,"data_stored":166502899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":19.091999999999999},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":21.518999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":19.024000000000001},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":12.145}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.432},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":12.179}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760392,"kb_used_data":166336,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93611448,"statfs":{"total":96636764160,"available":95858122752,"internally_reserved":0,"allocated":170328064,"data_stored":170144243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":0.69299999999999995},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":5.8639999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":7.2279999999999998},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":6.5259999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476472,"kb_used_data":166184,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93895368,"statfs":{"total":96636764160,"available":96148856832,"internally_reserved":0,"allocated":170172416,"data_stored":169988595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":8.468},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":3.4820000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":11.256}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.1479999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756240,"kb_used_data":165896,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93615600,"statfs":{"total":96636764160,"available":95862374400,"internally_reserved":0,"allocated":169877504,"data_stored":169693683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":9.6980000000000004},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":1.643}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":9.4939999999999998},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":1.8939999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":17.925999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":17.396999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":169201664,"data_stored":169201664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":169496576,"data_stored":169496576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169869312,"data_stored":169869312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165789696,"data_stored":165789696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:02.858 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:03.428 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:03.428 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:03.449 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-18T11:53:02.037713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":330900876,"num_objects":40399,"num_object_clones":0,"num_object_copies":161596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166688,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6851,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725552,"kb_used_data":648760,"kb_used_omap":14,"kb_used_meta":2076721,"kb_avail":374761808,"statfs":{"total":386547056640,"available":383756091392,"internally_reserved":0,"allocated":664330240,"data_stored":663594956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126562331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,9,11,23,10,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16990208,"num_objects":-2074,"num_object_clones":0,"num_object_copies":-8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004477"},"pg_stats":[{"pgid":"1.f","version":"37'10637","reported_seq":10738,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.694405+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:57.694405+0000","last_peered":"2024-09-18T11:52:57.694405+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:57.694405+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21405696,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10637,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10489","reported_seq":14781,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:58.818068+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:58.818068+0000","last_peered":"2024-09-18T11:52:58.818068+0000","last_clean":"2024-09-18T11:52:58.818068+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:58.818068+0000","last_undegraded":"2024-09-18T11:52:58.818068+0000","last_fullsized":"2024-09-18T11:52:58.818068+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026038,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10487,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10337","reported_seq":10454,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.883506+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:57.883506+0000","last_peered":"2024-09-18T11:52:57.883506+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:57.883506+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20241043,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10336,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":432}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10421","reported_seq":10515,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:58.784499+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:58.784499+0000","last_peered":"2024-09-18T11:52:58.784499+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:58.784499+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20518134,"num_objects":2505,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10366","reported_seq":10487,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.686945+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:57.686945+0000","last_peered":"2024-09-18T11:52:57.686945+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:57.686945+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20714742,"num_objects":2529,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10334","reported_seq":10453,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.677227+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:57.677227+0000","last_peered":"2024-09-18T11:52:57.677227+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:57.677227+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10334,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10500","reported_seq":10607,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:58.580736+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:58.580736+0000","last_peered":"2024-09-18T11:52:58.580736+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:58.580736+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20608246,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10498,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":400}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10311","reported_seq":10846,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:58.802440+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:58.802440+0000","last_peered":"2024-09-18T11:52:58.802440+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:58.802440+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21001462,"num_objects":2564,"num_object_clones":0,"num_object_copies":10256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1804,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10309,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10616","reported_seq":14969,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:57.562393+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:57.562393+0000","last_peered":"2024-09-18T11:52:57.562393+0000","last_clean":"2024-09-18T11:52:57.562393+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:57.562393+0000","last_undegraded":"2024-09-18T11:52:57.562393+0000","last_fullsized":"2024-09-18T11:52:57.562393+0000","mapping_epoch":36,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20965868,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10127","reported_seq":10241,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.956351+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:57.956351+0000","last_peered":"2024-09-18T11:52:57.956351+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:57.956351+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20206838,"num_objects":2467,"num_object_clones":0,"num_object_copies":9868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10125,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":504}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10462","reported_seq":10581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.829283+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:57.829283+0000","last_peered":"2024-09-18T11:52:57.829283+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:57.829283+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21057393,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10529","reported_seq":10619,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.651185+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:57.651185+0000","last_peered":"2024-09-18T11:52:57.651185+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:57.651185+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20494971,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":418}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10433","reported_seq":10531,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.825082+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:57.825082+0000","last_peered":"2024-09-18T11:52:57.825082+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:57.825082+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20757115,"num_objects":2534,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10432,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":488}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10523","reported_seq":10648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.697307+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:57.697307+0000","last_peered":"2024-09-18T11:52:57.697307+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:57.697307+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026038,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10521,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10389","reported_seq":14678,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:58.921701+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:58.921701+0000","last_peered":"2024-09-18T11:52:58.921701+0000","last_clean":"2024-09-18T11:52:58.921701+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:58.921701+0000","last_undegraded":"2024-09-18T11:52:58.921701+0000","last_fullsized":"2024-09-18T11:52:58.921701+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20591862,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10241","reported_seq":10331,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:58.771701+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:58.771701+0000","last_peered":"2024-09-18T11:52:58.771701+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:58.771701+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20198646,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10239,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":330900876,"num_objects":40399,"num_object_clones":0,"num_object_copies":161596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166688,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6851,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":661569536,"data_stored":661569536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749040,"kb_used_data":162780,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93622800,"statfs":{"total":96636764160,"available":95869747200,"internally_reserved":0,"allocated":166686720,"data_stored":166502899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":19.091999999999999},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":21.518999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":19.024000000000001},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":12.145}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.432},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":12.179}]}]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760424,"kb_used_data":162208,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93611416,"statfs":{"total":96636764160,"available":95858089984,"internally_reserved":0,"allocated":166100992,"data_stored":165917171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":3.5},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":3.5209999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.0179999999999998},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.4700000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":12.382999999999999},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":10.19}]}]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459872,"kb_used_data":162072,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93911968,"statfs":{"total":96636764160,"available":96165855232,"internally_reserved":0,"allocated":165961728,"data_stored":165777907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,5,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.6980000000000004},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":2.6349999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":3.2469999999999999},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":10.734}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":2.7429999999999999},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":16.140999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756216,"kb_used_data":161700,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93615624,"statfs":{"total":96636764160,"available":95862398976,"internally_reserved":0,"allocated":165580800,"data_stored":165396979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,1,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.8300000000000001},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":9.0069999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":11.742000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":8.9540000000000006}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.859},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":8.9800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164970496,"data_stored":164970496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165253120,"data_stored":165253120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165556224,"data_stored":165556224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165789696,"data_stored":165789696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:03.450 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:03.968 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:03.970 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:03.996 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-18T11:53:02.037713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":330900876,"num_objects":40399,"num_object_clones":0,"num_object_copies":161596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166688,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6851,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725552,"kb_used_data":648760,"kb_used_omap":14,"kb_used_meta":2076721,"kb_avail":374761808,"statfs":{"total":386547056640,"available":383756091392,"internally_reserved":0,"allocated":664330240,"data_stored":663594956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126562331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,9,11,23,10,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16990208,"num_objects":-2074,"num_object_clones":0,"num_object_copies":-8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004477"},"pg_stats":[{"pgid":"1.f","version":"37'10637","reported_seq":10738,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.694405+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:52:57.694405+0000","last_peered":"2024-09-18T11:52:57.694405+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:52:57.694405+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21405696,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10637,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10489","reported_seq":14781,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:58.818068+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:58.818068+0000","last_peered":"2024-09-18T11:52:58.818068+0000","last_clean":"2024-09-18T11:52:58.818068+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:58.818068+0000","last_undegraded":"2024-09-18T11:52:58.818068+0000","last_fullsized":"2024-09-18T11:52:58.818068+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026038,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10487,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10337","reported_seq":10454,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.883506+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:52:57.883506+0000","last_peered":"2024-09-18T11:52:57.883506+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:52:57.883506+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20241043,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10336,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":432}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10421","reported_seq":10515,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:58.784499+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:52:58.784499+0000","last_peered":"2024-09-18T11:52:58.784499+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:52:58.784499+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20518134,"num_objects":2505,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10366","reported_seq":10487,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.686945+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:52:57.686945+0000","last_peered":"2024-09-18T11:52:57.686945+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:52:57.686945+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20714742,"num_objects":2529,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10334","reported_seq":10453,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.677227+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:52:57.677227+0000","last_peered":"2024-09-18T11:52:57.677227+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:52:57.677227+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10334,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10500","reported_seq":10607,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:58.580736+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:52:58.580736+0000","last_peered":"2024-09-18T11:52:58.580736+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:52:58.580736+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20608246,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10498,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":400}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10311","reported_seq":10846,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:58.802440+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:52:58.802440+0000","last_peered":"2024-09-18T11:52:58.802440+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:52:58.802440+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21001462,"num_objects":2564,"num_object_clones":0,"num_object_copies":10256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1804,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10309,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10616","reported_seq":14969,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:57.562393+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:52:57.562393+0000","last_peered":"2024-09-18T11:52:57.562393+0000","last_clean":"2024-09-18T11:52:57.562393+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:52:57.562393+0000","last_undegraded":"2024-09-18T11:52:57.562393+0000","last_fullsized":"2024-09-18T11:52:57.562393+0000","mapping_epoch":36,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20965868,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10127","reported_seq":10241,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.956351+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:52:57.956351+0000","last_peered":"2024-09-18T11:52:57.956351+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:52:57.956351+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20206838,"num_objects":2467,"num_object_clones":0,"num_object_copies":9868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10125,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":504}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10462","reported_seq":10581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.829283+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:52:57.829283+0000","last_peered":"2024-09-18T11:52:57.829283+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:52:57.829283+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21057393,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10529","reported_seq":10619,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.651185+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:52:57.651185+0000","last_peered":"2024-09-18T11:52:57.651185+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:52:57.651185+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20494971,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":418}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10433","reported_seq":10531,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.825082+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:52:57.825082+0000","last_peered":"2024-09-18T11:52:57.825082+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:52:57.825082+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20757115,"num_objects":2534,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10432,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":488}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10523","reported_seq":10648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:57.697307+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:52:57.697307+0000","last_peered":"2024-09-18T11:52:57.697307+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:52:57.697307+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026038,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10521,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10389","reported_seq":14678,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:58.921701+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:58.921701+0000","last_peered":"2024-09-18T11:52:58.921701+0000","last_clean":"2024-09-18T11:52:58.921701+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:58.921701+0000","last_undegraded":"2024-09-18T11:52:58.921701+0000","last_fullsized":"2024-09-18T11:52:58.921701+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20591862,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10241","reported_seq":10331,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:52:58.771701+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:52:58.771701+0000","last_peered":"2024-09-18T11:52:58.771701+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:52:58.771701+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20198646,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10239,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":330900876,"num_objects":40399,"num_object_clones":0,"num_object_copies":161596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166688,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6851,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":661569536,"data_stored":661569536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749040,"kb_used_data":162780,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93622800,"statfs":{"total":96636764160,"available":95869747200,"internally_reserved":0,"allocated":166686720,"data_stored":166502899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":19.091999999999999},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":21.518999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":19.024000000000001},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":12.145}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.432},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":12.179}]}]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760424,"kb_used_data":162208,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93611416,"statfs":{"total":96636764160,"available":95858089984,"internally_reserved":0,"allocated":166100992,"data_stored":165917171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":3.5},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":3.5209999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":8.734,"5min":6.8109999999999999,"15min":6.2930000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":26.457000000000001,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.0179999999999998},{"interface":"front","average":{"1min":4.4189999999999996,"5min":4.7320000000000002,"15min":4.0039999999999996},"min":{"1min":0.25600000000000001,"5min":0.182,"15min":0.182},"max":{"1min":16.885999999999999,"5min":19.789999999999999,"15min":19.789999999999999},"last":3.4700000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":12.382999999999999},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":10.19}]}]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459872,"kb_used_data":162072,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93911968,"statfs":{"total":96636764160,"available":96165855232,"internally_reserved":0,"allocated":165961728,"data_stored":165777907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,5,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.6980000000000004},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":2.6349999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":3.2469999999999999},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":10.734}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":2.7429999999999999},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":16.140999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756216,"kb_used_data":161700,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93615624,"statfs":{"total":96636764160,"available":95862398976,"internally_reserved":0,"allocated":165580800,"data_stored":165396979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,1,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:51:57 2024","interfaces":[{"interface":"back","average":{"1min":9.0920000000000005,"5min":5.7380000000000004,"15min":4.3780000000000001},"min":{"1min":1.3759999999999999,"5min":0.19,"15min":0.14399999999999999},"max":{"1min":28.059000000000001,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.8300000000000001},{"interface":"front","average":{"1min":7.282,"5min":5.609,"15min":4.3310000000000004},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":34.786999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":9.0069999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":11.742000000000001},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":8.9540000000000006}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.859},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":8.9800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164970496,"data_stored":164970496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165253120,"data_stored":165253120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165556224,"data_stored":165556224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165789696,"data_stored":165789696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:03.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:03.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10637", 'reported_seq': 10738, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.694405+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:52:57.694405+0000', 'last_peered': '2024-09-18T11:52:57.694405+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:52:57.694405+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21405696, 'num_objects': 2613, 'num_object_clones': 0, 'num_object_copies': 10452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10637, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 510}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:03.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:03.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10337", 'reported_seq': 10454, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.883506+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:52:57.883506+0000', 'last_peered': '2024-09-18T11:52:57.883506+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:52:57.883506+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20241043, 'num_objects': 2472, 'num_object_clones': 0, 'num_object_copies': 9888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2472, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10336, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 432}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:03.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:03.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10421", 'reported_seq': 10515, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:58.784499+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:52:58.784499+0000', 'last_peered': '2024-09-18T11:52:58.784499+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:52:58.784499+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20518134, 'num_objects': 2505, 'num_object_clones': 0, 'num_object_copies': 10020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2505, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10419, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:03.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:03.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10366", 'reported_seq': 10487, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.686945+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:52:57.686945+0000', 'last_peered': '2024-09-18T11:52:57.686945+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:52:57.686945+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20714742, 'num_objects': 2529, 'num_object_clones': 0, 'num_object_copies': 10116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10364, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 477}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:03.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:03.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10334", 'reported_seq': 10453, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.677227+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:52:57.677227+0000', 'last_peered': '2024-09-18T11:52:57.677227+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:52:57.677227+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20086784, 'num_objects': 2452, 'num_object_clones': 0, 'num_object_copies': 9808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10334, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 333}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:03.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:03.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10500", 'reported_seq': 10607, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:58.580736+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:52:58.580736+0000', 'last_peered': '2024-09-18T11:52:58.580736+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:52:58.580736+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20608246, 'num_objects': 2516, 'num_object_clones': 0, 'num_object_copies': 10064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10498, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 400}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:03.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:03.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10311", 'reported_seq': 10846, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:58.802440+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:52:58.802440+0000', 'last_peered': '2024-09-18T11:52:58.802440+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:52:58.802440+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21001462, 'num_objects': 2564, 'num_object_clones': 0, 'num_object_copies': 10256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1804, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10309, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 234, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 760}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:03.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:03.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10127", 'reported_seq': 10241, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.956351+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:52:57.956351+0000', 'last_peered': '2024-09-18T11:52:57.956351+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:52:57.956351+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20206838, 'num_objects': 2467, 'num_object_clones': 0, 'num_object_copies': 9868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10125, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 504}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:03.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:03.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10462", 'reported_seq': 10581, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.829283+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:52:57.829283+0000', 'last_peered': '2024-09-18T11:52:57.829283+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:52:57.829283+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21057393, 'num_objects': 2571, 'num_object_clones': 0, 'num_object_copies': 10284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2571, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10459, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 537}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:04.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:04.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10529", 'reported_seq': 10619, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.651185+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:52:57.651185+0000', 'last_peered': '2024-09-18T11:52:57.651185+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:52:57.651185+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20494971, 'num_objects': 2502, 'num_object_clones': 0, 'num_object_copies': 10008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10528, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:04.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:04.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10433", 'reported_seq': 10531, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.825082+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:52:57.825082+0000', 'last_peered': '2024-09-18T11:52:57.825082+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:52:57.825082+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20757115, 'num_objects': 2534, 'num_object_clones': 0, 'num_object_copies': 10136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10432, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 488}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:04.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:04.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10523", 'reported_seq': 10648, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:57.697307+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:52:57.697307+0000', 'last_peered': '2024-09-18T11:52:57.697307+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:52:57.697307+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21026038, 'num_objects': 2567, 'num_object_clones': 0, 'num_object_copies': 10268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2567, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10521, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:04.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:04.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10241", 'reported_seq': 10331, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:52:58.771701+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:52:58.771701+0000', 'last_peered': '2024-09-18T11:52:58.771701+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:52:58.771701+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20198646, 'num_objects': 2466, 'num_object_clones': 0, 'num_object_copies': 9864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10239, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:04.000 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:53:04.652 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:04.652 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26540/158064 objects degraded (16.791%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":582,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39516,"data_bytes":323667340,"bytes_used":2774454272,"bytes_avail":383772602368,"bytes_total":386547056640,"degraded_objects":26540,"degraded_total":158064,"degraded_ratio":0.16790667071565948,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":246},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:53:04.673 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:53:04.673 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:05.265 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:05.265 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:05.285 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-18T11:53:04.038149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":323667340,"num_objects":39516,"num_object_clones":0,"num_object_copies":158064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167571,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6916,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161098,"ondisk_log_size":161098,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2709428,"kb_used_data":636792,"kb_used_omap":14,"kb_used_meta":2072561,"kb_avail":374777932,"statfs":{"total":386547056640,"available":383772602368,"internally_reserved":0,"allocated":652075008,"data_stored":651339724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2122302491},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,3,8,17,13,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24223744,"num_objects":-2957,"num_object_clones":0,"num_object_copies":-11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004467"},"pg_stats":[{"pgid":"1.f","version":"37'10706","reported_seq":10807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:02.784949+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:02.784949+0000","last_peered":"2024-09-18T11:53:02.784949+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:02.784949+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":2544,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10706,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":441}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10489","reported_seq":14781,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:58.818068+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:52:58.818068+0000","last_peered":"2024-09-18T11:52:58.818068+0000","last_clean":"2024-09-18T11:52:58.818068+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:52:58.818068+0000","last_undegraded":"2024-09-18T11:52:58.818068+0000","last_fullsized":"2024-09-18T11:52:58.818068+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026038,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10487,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10398","reported_seq":10514,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.194065+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:03.194065+0000","last_peered":"2024-09-18T11:53:03.194065+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:03.194065+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19741331,"num_objects":2411,"num_object_clones":0,"num_object_copies":9644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10397,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10488","reported_seq":10582,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.876865+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:03.876865+0000","last_peered":"2024-09-18T11:53:03.876865+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:03.876865+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19969270,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10486,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10426","reported_seq":10547,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:02.942458+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:02.942458+0000","last_peered":"2024-09-18T11:53:02.942458+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:02.942458+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20223222,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10390","reported_seq":10509,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:02.905749+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:02.905749+0000","last_peered":"2024-09-18T11:53:02.905749+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:02.905749+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19628032,"num_objects":2396,"num_object_clones":0,"num_object_copies":9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10566","reported_seq":10673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.749379+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:03.749379+0000","last_peered":"2024-09-18T11:53:03.749379+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:03.749379+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067574,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10564,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10367","reported_seq":11031,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.844454+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:03.844454+0000","last_peered":"2024-09-18T11:53:03.844454+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:03.844454+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20542710,"num_objects":2508,"num_object_clones":0,"num_object_copies":10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10365,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10682","reported_seq":15035,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:02.647552+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:02.647552+0000","last_peered":"2024-09-18T11:53:02.647552+0000","last_clean":"2024-09-18T11:53:02.647552+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:02.647552+0000","last_undegraded":"2024-09-18T11:53:02.647552+0000","last_fullsized":"2024-09-18T11:53:02.647552+0000","mapping_epoch":36,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20425196,"num_objects":2494,"num_object_clones":0,"num_object_copies":9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10679,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10187","reported_seq":10301,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.072552+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:03.072552+0000","last_peered":"2024-09-18T11:53:03.072552+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:03.072552+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19715318,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":444}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10515","reported_seq":10636,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:02.803709+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:02.803709+0000","last_peered":"2024-09-18T11:53:02.803709+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:02.803709+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20623217,"num_objects":2518,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10512,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":484}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10609","reported_seq":10700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:02.841839+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:02.841839+0000","last_peered":"2024-09-18T11:53:02.841839+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:02.841839+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19839611,"num_objects":2422,"num_object_clones":0,"num_object_copies":9688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10608,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":338}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10495","reported_seq":10593,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.128633+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:03.128633+0000","last_peered":"2024-09-18T11:53:03.128633+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:03.128633+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20249211,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":426}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10593","reported_seq":10718,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:02.890408+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:02.890408+0000","last_peered":"2024-09-18T11:53:02.890408+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:02.890408+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20452598,"num_objects":2497,"num_object_clones":0,"num_object_copies":9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":374}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10389","reported_seq":14678,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:52:58.921701+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:52:58.921701+0000","last_peered":"2024-09-18T11:52:58.921701+0000","last_clean":"2024-09-18T11:52:58.921701+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:52:58.921701+0000","last_undegraded":"2024-09-18T11:52:58.921701+0000","last_fullsized":"2024-09-18T11:52:58.921701+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20591862,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10298","reported_seq":10388,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.919361+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:03.919361+0000","last_peered":"2024-09-18T11:53:03.919361+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:03.919361+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19731702,"num_objects":2409,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10296,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":407}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":323667340,"num_objects":39516,"num_object_clones":0,"num_object_copies":158064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167571,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6916,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":649089024,"data_stored":649089024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161098,"ondisk_log_size":161098,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749244,"kb_used_data":158836,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93622596,"statfs":{"total":96636764160,"available":95869538304,"internally_reserved":0,"allocated":162648064,"data_stored":162464243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":7.2850000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":7.2560000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":2.3010000000000002},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":8.266}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":14.742000000000001},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":14.603}]}]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743952,"kb_used_data":158216,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93627888,"statfs":{"total":96636764160,"available":95874957312,"internally_reserved":0,"allocated":162013184,"data_stored":161829363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":1.282},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.3089999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":7.7590000000000003},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":12.749000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.1160000000000001},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":1.3}]}]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459872,"kb_used_data":162072,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93911968,"statfs":{"total":96636764160,"available":96165855232,"internally_reserved":0,"allocated":165961728,"data_stored":165777907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,5,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":5.6980000000000004},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":2.6349999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":3.2469999999999999},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":10.734}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":2.7429999999999999},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":16.140999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756360,"kb_used_data":157668,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93615480,"statfs":{"total":96636764160,"available":95862251520,"internally_reserved":0,"allocated":161452032,"data_stored":161268211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,4,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":10.888},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.6800000000000002},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":0.40899999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":23.18}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160837632,"data_stored":160837632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165253120,"data_stored":165253120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161402880,"data_stored":161402880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":161595392,"data_stored":161595392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:08.287 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:08.891 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:08.891 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:08.907 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-09-18T11:53:08.039177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":318997900,"num_objects":38946,"num_object_clones":0,"num_object_copies":155784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38946,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168141,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6916,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161168,"ondisk_log_size":161168,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2709776,"kb_used_data":628800,"kb_used_omap":14,"kb_used_meta":2080881,"kb_avail":374777584,"statfs":{"total":386547056640,"available":383772246016,"internally_reserved":0,"allocated":643891200,"data_stored":643155916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130822171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,12,9,9,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20488192,"num_objects":-2501,"num_object_clones":0,"num_object_copies":-10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2501,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002845"},"pg_stats":[{"pgid":"1.f","version":"37'10767","reported_seq":10868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.796980+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:07.796980+0000","last_peered":"2024-09-18T11:53:07.796980+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:07.796980+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20340736,"num_objects":2483,"num_object_clones":0,"num_object_copies":9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10767,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10558","reported_seq":14850,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:03.934636+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:03.934636+0000","last_peered":"2024-09-18T11:53:03.934636+0000","last_clean":"2024-09-18T11:53:03.934636+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:03.934636+0000","last_undegraded":"2024-09-18T11:53:03.934636+0000","last_fullsized":"2024-09-18T11:53:03.934636+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20460790,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10556,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10398","reported_seq":10514,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.194065+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:03.194065+0000","last_peered":"2024-09-18T11:53:03.194065+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:03.194065+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19741331,"num_objects":2411,"num_object_clones":0,"num_object_copies":9644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10397,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10488","reported_seq":10582,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.876865+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:03.876865+0000","last_peered":"2024-09-18T11:53:03.876865+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:03.876865+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19969270,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10486,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10487","reported_seq":10608,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.745689+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:07.745689+0000","last_peered":"2024-09-18T11:53:07.745689+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:07.745689+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19723510,"num_objects":2408,"num_object_clones":0,"num_object_copies":9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10485,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10453","reported_seq":10572,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.829479+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:07.829479+0000","last_peered":"2024-09-18T11:53:07.829479+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:07.829479+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19111936,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10453,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10566","reported_seq":10673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.749379+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:03.749379+0000","last_peered":"2024-09-18T11:53:03.749379+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:03.749379+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067574,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10564,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10367","reported_seq":11031,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.844454+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:03.844454+0000","last_peered":"2024-09-18T11:53:03.844454+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:03.844454+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20542710,"num_objects":2508,"num_object_clones":0,"num_object_copies":10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10365,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10751","reported_seq":15104,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:07.899969+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:07.899969+0000","last_peered":"2024-09-18T11:53:07.899969+0000","last_clean":"2024-09-18T11:53:07.899969+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:07.899969+0000","last_undegraded":"2024-09-18T11:53:07.899969+0000","last_fullsized":"2024-09-18T11:53:07.899969+0000","mapping_epoch":36,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19859948,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10748,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10187","reported_seq":10301,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.072552+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:03.072552+0000","last_peered":"2024-09-18T11:53:03.072552+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:03.072552+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19715318,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":444}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10582","reported_seq":10702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.021582+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:08.021582+0000","last_peered":"2024-09-18T11:53:08.021582+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:08.021582+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20074353,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10667","reported_seq":10760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.949046+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:07.949046+0000","last_peered":"2024-09-18T11:53:07.949046+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:07.949046+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19364475,"num_objects":2364,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10666,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":280}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10495","reported_seq":10593,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.128633+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:03.128633+0000","last_peered":"2024-09-18T11:53:03.128633+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:03.128633+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20249211,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":426}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10655","reported_seq":10780,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.974186+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:07.974186+0000","last_peered":"2024-09-18T11:53:07.974186+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:07.974186+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19944694,"num_objects":2435,"num_object_clones":0,"num_object_copies":9740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":312}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10449","reported_seq":14738,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:03.884500+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:03.884500+0000","last_peered":"2024-09-18T11:53:03.884500+0000","last_clean":"2024-09-18T11:53:03.884500+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:03.884500+0000","last_undegraded":"2024-09-18T11:53:03.884500+0000","last_fullsized":"2024-09-18T11:53:03.884500+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20100342,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10447,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10298","reported_seq":10388,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.919361+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:03.919361+0000","last_peered":"2024-09-18T11:53:03.919361+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:03.919361+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19731702,"num_objects":2409,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10296,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":407}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":318997900,"num_objects":38946,"num_object_clones":0,"num_object_copies":155784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38946,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168141,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6916,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":640806912,"data_stored":640806912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161168,"ondisk_log_size":161168,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749380,"kb_used_data":154796,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93622460,"statfs":{"total":96636764160,"available":95869399040,"internally_reserved":0,"allocated":158511104,"data_stored":158327283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,1,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":17.914000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":17.943999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.432},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.052}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":12.06},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":7.9100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743952,"kb_used_data":158216,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93627888,"statfs":{"total":96636764160,"available":95874957312,"internally_reserved":0,"allocated":162013184,"data_stored":161829363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":1.282},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.3089999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":7.7590000000000003},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":12.749000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.1160000000000001},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":1.3}]}]},{"osd":1,"up_from":29,"seq":124554051684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460084,"kb_used_data":158120,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93911756,"statfs":{"total":96636764160,"available":96165638144,"internally_reserved":0,"allocated":161914880,"data_stored":161731059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.8179999999999996},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.7089999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":17.116},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":12.135999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":1.256},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.1279999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756360,"kb_used_data":157668,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93615480,"statfs":{"total":96636764160,"available":95862251520,"internally_reserved":0,"allocated":161452032,"data_stored":161268211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,4,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":10.888},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.6800000000000002},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":0.40899999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":23.18}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160837632,"data_stored":160837632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161103872,"data_stored":161103872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161402880,"data_stored":161402880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157462528,"data_stored":157462528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:08.908 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:09.500 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:09.501 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:09.522 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-09-18T11:53:08.039177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":318997900,"num_objects":38946,"num_object_clones":0,"num_object_copies":155784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38946,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168141,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6916,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161168,"ondisk_log_size":161168,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2709776,"kb_used_data":628800,"kb_used_omap":14,"kb_used_meta":2080881,"kb_avail":374777584,"statfs":{"total":386547056640,"available":383772246016,"internally_reserved":0,"allocated":643891200,"data_stored":643155916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130822171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,12,9,9,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20488192,"num_objects":-2501,"num_object_clones":0,"num_object_copies":-10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2501,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002845"},"pg_stats":[{"pgid":"1.f","version":"37'10767","reported_seq":10868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.796980+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:07.796980+0000","last_peered":"2024-09-18T11:53:07.796980+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:07.796980+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20340736,"num_objects":2483,"num_object_clones":0,"num_object_copies":9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10767,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10558","reported_seq":14850,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:03.934636+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:03.934636+0000","last_peered":"2024-09-18T11:53:03.934636+0000","last_clean":"2024-09-18T11:53:03.934636+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:03.934636+0000","last_undegraded":"2024-09-18T11:53:03.934636+0000","last_fullsized":"2024-09-18T11:53:03.934636+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20460790,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10556,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10398","reported_seq":10514,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.194065+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:03.194065+0000","last_peered":"2024-09-18T11:53:03.194065+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:03.194065+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19741331,"num_objects":2411,"num_object_clones":0,"num_object_copies":9644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10397,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10488","reported_seq":10582,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.876865+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:03.876865+0000","last_peered":"2024-09-18T11:53:03.876865+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:03.876865+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19969270,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10486,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10487","reported_seq":10608,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.745689+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:07.745689+0000","last_peered":"2024-09-18T11:53:07.745689+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:07.745689+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19723510,"num_objects":2408,"num_object_clones":0,"num_object_copies":9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10485,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10453","reported_seq":10572,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.829479+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:07.829479+0000","last_peered":"2024-09-18T11:53:07.829479+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:07.829479+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19111936,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10453,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10566","reported_seq":10673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.749379+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:03.749379+0000","last_peered":"2024-09-18T11:53:03.749379+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:03.749379+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067574,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10564,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10367","reported_seq":11031,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.844454+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:03.844454+0000","last_peered":"2024-09-18T11:53:03.844454+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:03.844454+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20542710,"num_objects":2508,"num_object_clones":0,"num_object_copies":10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10365,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10751","reported_seq":15104,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:07.899969+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:07.899969+0000","last_peered":"2024-09-18T11:53:07.899969+0000","last_clean":"2024-09-18T11:53:07.899969+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:07.899969+0000","last_undegraded":"2024-09-18T11:53:07.899969+0000","last_fullsized":"2024-09-18T11:53:07.899969+0000","mapping_epoch":36,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19859948,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10748,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10187","reported_seq":10301,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.072552+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:03.072552+0000","last_peered":"2024-09-18T11:53:03.072552+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:03.072552+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19715318,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":444}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10582","reported_seq":10702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.021582+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:08.021582+0000","last_peered":"2024-09-18T11:53:08.021582+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:08.021582+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20074353,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10667","reported_seq":10760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.949046+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:07.949046+0000","last_peered":"2024-09-18T11:53:07.949046+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:07.949046+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19364475,"num_objects":2364,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10666,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":280}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10495","reported_seq":10593,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.128633+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:03.128633+0000","last_peered":"2024-09-18T11:53:03.128633+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:03.128633+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20249211,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":426}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10655","reported_seq":10780,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.974186+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:07.974186+0000","last_peered":"2024-09-18T11:53:07.974186+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:07.974186+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19944694,"num_objects":2435,"num_object_clones":0,"num_object_copies":9740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":312}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10449","reported_seq":14738,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:03.884500+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:03.884500+0000","last_peered":"2024-09-18T11:53:03.884500+0000","last_clean":"2024-09-18T11:53:03.884500+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:03.884500+0000","last_undegraded":"2024-09-18T11:53:03.884500+0000","last_fullsized":"2024-09-18T11:53:03.884500+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20100342,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10447,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10298","reported_seq":10388,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:03.919361+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:03.919361+0000","last_peered":"2024-09-18T11:53:03.919361+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:03.919361+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19731702,"num_objects":2409,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10296,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":407}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":318997900,"num_objects":38946,"num_object_clones":0,"num_object_copies":155784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38946,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168141,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6916,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":640806912,"data_stored":640806912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161168,"ondisk_log_size":161168,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749380,"kb_used_data":154796,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93622460,"statfs":{"total":96636764160,"available":95869399040,"internally_reserved":0,"allocated":158511104,"data_stored":158327283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,1,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":17.914000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":17.943999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.432},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.052}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":12.06},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":7.9100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743952,"kb_used_data":158216,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93627888,"statfs":{"total":96636764160,"available":95874957312,"internally_reserved":0,"allocated":162013184,"data_stored":161829363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":1.282},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.3089999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":7.7590000000000003},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":12.749000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.1160000000000001},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":1.3}]}]},{"osd":1,"up_from":29,"seq":124554051684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460084,"kb_used_data":158120,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93911756,"statfs":{"total":96636764160,"available":96165638144,"internally_reserved":0,"allocated":161914880,"data_stored":161731059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":5.7350000000000003,"15min":4.1159999999999997},"min":{"1min":0.29799999999999999,"5min":0.224,"15min":0.159},"max":{"1min":32.371000000000002,"5min":45.462000000000003,"15min":45.462000000000003},"last":4.8179999999999996},{"interface":"front","average":{"1min":6.5739999999999998,"5min":5.5069999999999997,"15min":5.1130000000000004},"min":{"1min":0.33200000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":20.417999999999999,"5min":25.187000000000001,"15min":25.187000000000001},"last":7.7089999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":8.452,"5min":6.3120000000000003,"15min":4.782},"min":{"1min":0.29299999999999998,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":23.850000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":17.116},{"interface":"front","average":{"1min":4.2770000000000001,"5min":4.3120000000000003,"15min":4.0999999999999996},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":25.722999999999999,"5min":25.722999999999999,"15min":26.719000000000001},"last":12.135999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:03 2024","interfaces":[{"interface":"back","average":{"1min":6.5469999999999997,"5min":5.5369999999999999,"15min":4.5650000000000004},"min":{"1min":0.25900000000000001,"5min":0.19,"15min":0.19},"max":{"1min":20.373999999999999,"5min":20.373999999999999,"15min":20.373999999999999},"last":1.256},{"interface":"front","average":{"1min":6.8570000000000002,"5min":6.657,"15min":5.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":16.262,"5min":25.440000000000001,"15min":25.440000000000001},"last":1.1279999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756360,"kb_used_data":157668,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93615480,"statfs":{"total":96636764160,"available":95862251520,"internally_reserved":0,"allocated":161452032,"data_stored":161268211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,4,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":10.888},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":2.6800000000000002},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":0.40899999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":23.18}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160837632,"data_stored":160837632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161103872,"data_stored":161103872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161402880,"data_stored":161402880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157462528,"data_stored":157462528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:09.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:09.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10767", 'reported_seq': 10868, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:07.796980+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:53:07.796980+0000', 'last_peered': '2024-09-18T11:53:07.796980+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:53:07.796980+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20340736, 'num_objects': 2483, 'num_object_clones': 0, 'num_object_copies': 9932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10767, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 380}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:09.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:09.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10398", 'reported_seq': 10514, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:03.194065+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:53:03.194065+0000', 'last_peered': '2024-09-18T11:53:03.194065+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:53:03.194065+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19741331, 'num_objects': 2411, 'num_object_clones': 0, 'num_object_copies': 9644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2411, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10397, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 371}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:09.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:09.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10488", 'reported_seq': 10582, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:03.876865+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:03.876865+0000', 'last_peered': '2024-09-18T11:53:03.876865+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:03.876865+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19969270, 'num_objects': 2438, 'num_object_clones': 0, 'num_object_copies': 9752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10486, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:09.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:09.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10487", 'reported_seq': 10608, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:07.745689+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:53:07.745689+0000', 'last_peered': '2024-09-18T11:53:07.745689+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:53:07.745689+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19723510, 'num_objects': 2408, 'num_object_clones': 0, 'num_object_copies': 9632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10485, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 356}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:09.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:09.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10453", 'reported_seq': 10572, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:07.829479+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:53:07.829479+0000', 'last_peered': '2024-09-18T11:53:07.829479+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:53:07.829479+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19111936, 'num_objects': 2333, 'num_object_clones': 0, 'num_object_copies': 9332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2333, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10453, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 214}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:09.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10566", 'reported_seq': 10673, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:03.749379+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:03.749379+0000', 'last_peered': '2024-09-18T11:53:03.749379+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:03.749379+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20067574, 'num_objects': 2450, 'num_object_clones': 0, 'num_object_copies': 9800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10564, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 334}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10367", 'reported_seq': 11031, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:03.844454+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:03.844454+0000', 'last_peered': '2024-09-18T11:53:03.844454+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:03.844454+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20542710, 'num_objects': 2508, 'num_object_clones': 0, 'num_object_copies': 10032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1739, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10365, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 299, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 769}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10187", 'reported_seq': 10301, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:03.072552+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:53:03.072552+0000', 'last_peered': '2024-09-18T11:53:03.072552+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:53:03.072552+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19715318, 'num_objects': 2407, 'num_object_clones': 0, 'num_object_copies': 9628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10185, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 444}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10582", 'reported_seq': 10702, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:08.021582+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:53:08.021582+0000', 'last_peered': '2024-09-18T11:53:08.021582+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:53:08.021582+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20074353, 'num_objects': 2451, 'num_object_clones': 0, 'num_object_copies': 9804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10579, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 417}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10667", 'reported_seq': 10760, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:07.949046+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:53:07.949046+0000', 'last_peered': '2024-09-18T11:53:07.949046+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:53:07.949046+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19364475, 'num_objects': 2364, 'num_object_clones': 0, 'num_object_copies': 9456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2364, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10666, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 280}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10495", 'reported_seq': 10593, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:03.128633+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:53:03.128633+0000', 'last_peered': '2024-09-18T11:53:03.128633+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:53:03.128633+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20249211, 'num_objects': 2472, 'num_object_clones': 0, 'num_object_copies': 9888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10494, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 426}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10655", 'reported_seq': 10780, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:07.974186+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:53:07.974186+0000', 'last_peered': '2024-09-18T11:53:07.974186+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:53:07.974186+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19944694, 'num_objects': 2435, 'num_object_clones': 0, 'num_object_copies': 9740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10653, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 312}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10298", 'reported_seq': 10388, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:03.919361+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:03.919361+0000', 'last_peered': '2024-09-18T11:53:03.919361+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:03.919361+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19731702, 'num_objects': 2409, 'num_object_clones': 0, 'num_object_copies': 9636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10296, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 407}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:09.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:53:10.217 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:10.217 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26497/153476 objects degraded (17.265%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":588,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":38369,"data_bytes":314271116,"bytes_used":2758385664,"bytes_avail":383788670976,"bytes_total":386547056640,"degraded_objects":26497,"degraded_total":153476,"degraded_ratio":0.17264588600172015,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":139,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":219},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:53:10.237 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:53:10.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:10.777 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:10.778 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:10.799 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-18T11:53:10.039636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":314271116,"num_objects":38369,"num_object_clones":0,"num_object_copies":153476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38369,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168718,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6959,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2693736,"kb_used_data":616896,"kb_used_omap":14,"kb_used_meta":2076721,"kb_avail":374793624,"statfs":{"total":386547056640,"available":383788670976,"internally_reserved":0,"allocated":631701504,"data_stored":630966220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126562331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,12,7,8,14,13,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21569536,"num_objects":-2633,"num_object_clones":0,"num_object_copies":-10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002892"},"pg_stats":[{"pgid":"1.f","version":"37'10767","reported_seq":10868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.796980+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:07.796980+0000","last_peered":"2024-09-18T11:53:07.796980+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:07.796980+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20340736,"num_objects":2483,"num_object_clones":0,"num_object_copies":9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10767,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10626","reported_seq":14918,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:09.028126+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:09.028126+0000","last_peered":"2024-09-18T11:53:09.028126+0000","last_clean":"2024-09-18T11:53:09.028126+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:09.028126+0000","last_undegraded":"2024-09-18T11:53:09.028126+0000","last_fullsized":"2024-09-18T11:53:09.028126+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19903734,"num_objects":2430,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10624,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10453","reported_seq":10569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.095223+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:08.095223+0000","last_peered":"2024-09-18T11:53:08.095223+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:08.095223+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19290771,"num_objects":2356,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2356,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10452,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10563","reported_seq":10656,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:09.019458+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:09.019458+0000","last_peered":"2024-09-18T11:53:09.019458+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:09.019458+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19354870,"num_objects":2363,"num_object_clones":0,"num_object_copies":9452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10487","reported_seq":10608,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.745689+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:07.745689+0000","last_peered":"2024-09-18T11:53:07.745689+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:07.745689+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19723510,"num_objects":2408,"num_object_clones":0,"num_object_copies":9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10485,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10453","reported_seq":10572,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.829479+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:07.829479+0000","last_peered":"2024-09-18T11:53:07.829479+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:07.829479+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19111936,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10453,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10627","reported_seq":10734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.961275+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:08.961275+0000","last_peered":"2024-09-18T11:53:08.961275+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:08.961275+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19567862,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":273}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10431","reported_seq":11181,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.964401+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:08.964401+0000","last_peered":"2024-09-18T11:53:08.964401+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:08.964401+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20018422,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10429,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10751","reported_seq":15104,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:07.899969+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:07.899969+0000","last_peered":"2024-09-18T11:53:07.899969+0000","last_clean":"2024-09-18T11:53:07.899969+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:07.899969+0000","last_undegraded":"2024-09-18T11:53:07.899969+0000","last_fullsized":"2024-09-18T11:53:07.899969+0000","mapping_epoch":36,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19859948,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10748,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10249","reported_seq":10363,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.213217+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:08.213217+0000","last_peered":"2024-09-18T11:53:08.213217+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:08.213217+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19207414,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10247,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":382}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10582","reported_seq":10702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.021582+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:08.021582+0000","last_peered":"2024-09-18T11:53:08.021582+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:08.021582+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20074353,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10667","reported_seq":10760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.949046+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:07.949046+0000","last_peered":"2024-09-18T11:53:07.949046+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:07.949046+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19364475,"num_objects":2364,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10666,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":280}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10569","reported_seq":10667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.230176+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:08.230176+0000","last_peered":"2024-09-18T11:53:08.230176+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:08.230176+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19643003,"num_objects":2398,"num_object_clones":0,"num_object_copies":9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10568,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":352}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10655","reported_seq":10780,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:07.974186+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:07.974186+0000","last_peered":"2024-09-18T11:53:07.974186+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:07.974186+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19944694,"num_objects":2435,"num_object_clones":0,"num_object_copies":9740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":312}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10506","reported_seq":14795,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:08.986362+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:08.986362+0000","last_peered":"2024-09-18T11:53:08.986362+0000","last_clean":"2024-09-18T11:53:08.986362+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:08.986362+0000","last_undegraded":"2024-09-18T11:53:08.986362+0000","last_fullsized":"2024-09-18T11:53:08.986362+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633398,"num_objects":2397,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10359","reported_seq":10449,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.876617+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:08.876617+0000","last_peered":"2024-09-18T11:53:08.876617+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:08.876617+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19231990,"num_objects":2348,"num_object_clones":0,"num_object_copies":9392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10357,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":314271116,"num_objects":38369,"num_object_clones":0,"num_object_copies":153476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38369,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168718,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6959,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":628371456,"data_stored":628371456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749380,"kb_used_data":154796,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93622460,"statfs":{"total":96636764160,"available":95869399040,"internally_reserved":0,"allocated":158511104,"data_stored":158327283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,1,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":17.914000000000001},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":17.943999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.432},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.052}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":12.06},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":7.9100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744216,"kb_used_data":154308,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93627624,"statfs":{"total":96636764160,"available":95874686976,"internally_reserved":0,"allocated":158011392,"data_stored":157827571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:10 2024","interfaces":[{"interface":"back","average":{"1min":5.2750000000000004,"5min":5.2290000000000001,"15min":3.1680000000000001},"min":{"1min":0.253,"5min":0.153,"15min":0.153},"max":{"1min":24.965,"5min":24.965,"15min":24.965},"last":2.9420000000000002},{"interface":"front","average":{"1min":5.6449999999999996,"5min":4.5860000000000003,"15min":2.7639999999999998},"min":{"1min":0.46000000000000002,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":24.995000000000001,"5min":24.995000000000001,"15min":24.995000000000001},"last":2.964}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":14.064},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":5.4630000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.9100000000000001},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":9.6709999999999994}]}]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460192,"kb_used_data":154068,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93911648,"statfs":{"total":96636764160,"available":96165527552,"internally_reserved":0,"allocated":157765632,"data_stored":157581811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":5.5410000000000004},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":26.870000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":16.405000000000001},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.5709999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.968},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":17.045000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739948,"kb_used_data":153724,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93631892,"statfs":{"total":96636764160,"available":95879057408,"internally_reserved":0,"allocated":157413376,"data_stored":157229555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,12,0,2,7,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.583},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.981999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":6.3970000000000002},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":0.26100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.249},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":11.023}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":156672000,"data_stored":156672000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156950528,"data_stored":156950528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157286400,"data_stored":157286400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157462528,"data_stored":157462528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:13.801 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:14.395 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:14.418 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-18T11:53:14.040542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":309159308,"num_objects":37745,"num_object_clones":0,"num_object_copies":150980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169342,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6959,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160469,"ondisk_log_size":160469,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694152,"kb_used_data":609012,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374793208,"statfs":{"total":386547056640,"available":383788244992,"internally_reserved":0,"allocated":623628288,"data_stored":622893004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,13,5,8,17,17,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21741568,"num_objects":-2654,"num_object_clones":0,"num_object_copies":-10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002829"},"pg_stats":[{"pgid":"1.f","version":"37'10840","reported_seq":10938,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.944308+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:12.944308+0000","last_peered":"2024-09-18T11:53:12.944308+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:12.944308+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19742720,"num_objects":2410,"num_object_clones":0,"num_object_copies":9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10626","reported_seq":14918,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:09.028126+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:09.028126+0000","last_peered":"2024-09-18T11:53:09.028126+0000","last_clean":"2024-09-18T11:53:09.028126+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:09.028126+0000","last_undegraded":"2024-09-18T11:53:09.028126+0000","last_fullsized":"2024-09-18T11:53:09.028126+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19903734,"num_objects":2430,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10624,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10505","reported_seq":10622,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.181610+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:13.181610+0000","last_peered":"2024-09-18T11:53:13.181610+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:13.181610+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18864787,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10504,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10563","reported_seq":10656,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:09.019458+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:09.019458+0000","last_peered":"2024-09-18T11:53:09.019458+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:09.019458+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19354870,"num_objects":2363,"num_object_clones":0,"num_object_copies":9452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10549","reported_seq":10670,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.847378+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:12.847378+0000","last_peered":"2024-09-18T11:53:12.847378+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:12.847378+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19215606,"num_objects":2346,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10547,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10502","reported_seq":10621,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.972340+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:12.972340+0000","last_peered":"2024-09-18T11:53:12.972340+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:12.972340+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18710528,"num_objects":2284,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10502,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10627","reported_seq":10734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.961275+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:08.961275+0000","last_peered":"2024-09-18T11:53:08.961275+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:08.961275+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19567862,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":273}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10431","reported_seq":11181,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.964401+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:08.964401+0000","last_peered":"2024-09-18T11:53:08.964401+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:08.964401+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20018422,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10429,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10814","reported_seq":15167,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:13.010181+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:13.010181+0000","last_peered":"2024-09-18T11:53:13.010181+0000","last_clean":"2024-09-18T11:53:13.010181+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:13.010181+0000","last_undegraded":"2024-09-18T11:53:13.010181+0000","last_fullsized":"2024-09-18T11:53:13.010181+0000","mapping_epoch":36,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19343852,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10314","reported_seq":10428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.233465+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:13.233465+0000","last_peered":"2024-09-18T11:53:13.233465+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:13.233465+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18674934,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10312,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10644","reported_seq":10765,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.997605+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:12.997605+0000","last_peered":"2024-09-18T11:53:12.997605+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:12.997605+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19566449,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10641,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10725","reported_seq":10817,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.827195+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:12.827195+0000","last_peered":"2024-09-18T11:53:12.827195+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:12.827195+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18889339,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10724,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":222}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10634","reported_seq":10732,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.141010+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:13.141010+0000","last_peered":"2024-09-18T11:53:13.141010+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:13.141010+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19110523,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10633,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10730","reported_seq":10854,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.923123+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:12.923123+0000","last_peered":"2024-09-18T11:53:12.923123+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:12.923123+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330294,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10728,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10506","reported_seq":14795,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:08.986362+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:08.986362+0000","last_peered":"2024-09-18T11:53:08.986362+0000","last_clean":"2024-09-18T11:53:08.986362+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:08.986362+0000","last_undegraded":"2024-09-18T11:53:08.986362+0000","last_fullsized":"2024-09-18T11:53:08.986362+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633398,"num_objects":2397,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10359","reported_seq":10449,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.876617+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:08.876617+0000","last_peered":"2024-09-18T11:53:08.876617+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:08.876617+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19231990,"num_objects":2348,"num_object_clones":0,"num_object_copies":9392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10357,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":309159308,"num_objects":37745,"num_object_clones":0,"num_object_copies":150980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169342,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6959,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":620158976,"data_stored":620158976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160469,"ondisk_log_size":160469,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749584,"kb_used_data":150852,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93622256,"statfs":{"total":96636764160,"available":95869190144,"internally_reserved":0,"allocated":154472448,"data_stored":154288627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.4429999999999996},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.0980000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":8.5399999999999991}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":8.5920000000000005},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":8.5060000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744428,"kb_used_data":150368,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93627412,"statfs":{"total":96636764160,"available":95874469888,"internally_reserved":0,"allocated":153976832,"data_stored":153793011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":4.1109999999999998},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":4.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.1070000000000002},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.109}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":4.242},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":19.797000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460192,"kb_used_data":154068,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93911648,"statfs":{"total":96636764160,"available":96165527552,"internally_reserved":0,"allocated":157765632,"data_stored":157581811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":5.5410000000000004},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":26.870000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":16.405000000000001},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.5709999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.968},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":17.045000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739948,"kb_used_data":153724,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93631892,"statfs":{"total":96636764160,"available":95879057408,"internally_reserved":0,"allocated":157413376,"data_stored":157229555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,12,0,2,7,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.583},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.981999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":6.3970000000000002},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":0.26100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.249},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":11.023}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":156672000,"data_stored":156672000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156950528,"data_stored":156950528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153186304,"data_stored":153186304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153350144,"data_stored":153350144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:14.420 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:14.961 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:14.961 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:14.995 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-18T11:53:14.040542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":309159308,"num_objects":37745,"num_object_clones":0,"num_object_copies":150980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169342,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6959,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160469,"ondisk_log_size":160469,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694152,"kb_used_data":609012,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374793208,"statfs":{"total":386547056640,"available":383788244992,"internally_reserved":0,"allocated":623628288,"data_stored":622893004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,13,5,8,17,17,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21741568,"num_objects":-2654,"num_object_clones":0,"num_object_copies":-10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002829"},"pg_stats":[{"pgid":"1.f","version":"37'10840","reported_seq":10938,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.944308+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:12.944308+0000","last_peered":"2024-09-18T11:53:12.944308+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:12.944308+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19742720,"num_objects":2410,"num_object_clones":0,"num_object_copies":9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10626","reported_seq":14918,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:09.028126+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:09.028126+0000","last_peered":"2024-09-18T11:53:09.028126+0000","last_clean":"2024-09-18T11:53:09.028126+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:09.028126+0000","last_undegraded":"2024-09-18T11:53:09.028126+0000","last_fullsized":"2024-09-18T11:53:09.028126+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19903734,"num_objects":2430,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10624,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10505","reported_seq":10622,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.181610+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:13.181610+0000","last_peered":"2024-09-18T11:53:13.181610+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:13.181610+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18864787,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10504,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10563","reported_seq":10656,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:09.019458+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:09.019458+0000","last_peered":"2024-09-18T11:53:09.019458+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:09.019458+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19354870,"num_objects":2363,"num_object_clones":0,"num_object_copies":9452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10549","reported_seq":10670,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.847378+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:12.847378+0000","last_peered":"2024-09-18T11:53:12.847378+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:12.847378+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19215606,"num_objects":2346,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10547,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10502","reported_seq":10621,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.972340+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:12.972340+0000","last_peered":"2024-09-18T11:53:12.972340+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:12.972340+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18710528,"num_objects":2284,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10502,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10627","reported_seq":10734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.961275+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:08.961275+0000","last_peered":"2024-09-18T11:53:08.961275+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:08.961275+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19567862,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":273}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10431","reported_seq":11181,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.964401+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:08.964401+0000","last_peered":"2024-09-18T11:53:08.964401+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:08.964401+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20018422,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10429,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10814","reported_seq":15167,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:13.010181+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:13.010181+0000","last_peered":"2024-09-18T11:53:13.010181+0000","last_clean":"2024-09-18T11:53:13.010181+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:13.010181+0000","last_undegraded":"2024-09-18T11:53:13.010181+0000","last_fullsized":"2024-09-18T11:53:13.010181+0000","mapping_epoch":36,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19343852,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10314","reported_seq":10428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.233465+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:13.233465+0000","last_peered":"2024-09-18T11:53:13.233465+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:13.233465+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18674934,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10312,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10644","reported_seq":10765,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.997605+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:12.997605+0000","last_peered":"2024-09-18T11:53:12.997605+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:12.997605+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19566449,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10641,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10725","reported_seq":10817,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.827195+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:12.827195+0000","last_peered":"2024-09-18T11:53:12.827195+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:12.827195+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18889339,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10724,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":222}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10634","reported_seq":10732,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.141010+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:13.141010+0000","last_peered":"2024-09-18T11:53:13.141010+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:13.141010+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19110523,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10633,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10730","reported_seq":10854,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.923123+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:12.923123+0000","last_peered":"2024-09-18T11:53:12.923123+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:12.923123+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330294,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10728,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10506","reported_seq":14795,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:08.986362+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:08.986362+0000","last_peered":"2024-09-18T11:53:08.986362+0000","last_clean":"2024-09-18T11:53:08.986362+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:08.986362+0000","last_undegraded":"2024-09-18T11:53:08.986362+0000","last_fullsized":"2024-09-18T11:53:08.986362+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633398,"num_objects":2397,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10359","reported_seq":10449,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:08.876617+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:08.876617+0000","last_peered":"2024-09-18T11:53:08.876617+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:08.876617+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19231990,"num_objects":2348,"num_object_clones":0,"num_object_copies":9392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10357,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":309159308,"num_objects":37745,"num_object_clones":0,"num_object_copies":150980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169342,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6959,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":620158976,"data_stored":620158976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160469,"ondisk_log_size":160469,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749584,"kb_used_data":150852,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93622256,"statfs":{"total":96636764160,"available":95869190144,"internally_reserved":0,"allocated":154472448,"data_stored":154288627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.4429999999999996},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.0980000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":8.5399999999999991}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":8.5920000000000005},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":8.5060000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744428,"kb_used_data":150368,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93627412,"statfs":{"total":96636764160,"available":95874469888,"internally_reserved":0,"allocated":153976832,"data_stored":153793011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":4.1109999999999998},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":4.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.1070000000000002},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.109}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":4.242},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":19.797000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460192,"kb_used_data":154068,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93911648,"statfs":{"total":96636764160,"available":96165527552,"internally_reserved":0,"allocated":157765632,"data_stored":157581811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":5.5410000000000004},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":26.870000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":16.405000000000001},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.5709999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.968},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":17.045000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739948,"kb_used_data":153724,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93631892,"statfs":{"total":96636764160,"available":95879057408,"internally_reserved":0,"allocated":157413376,"data_stored":157229555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,12,0,2,7,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.583},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.981999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":6.3970000000000002},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":0.26100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.249},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":11.023}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":156672000,"data_stored":156672000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156950528,"data_stored":156950528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153186304,"data_stored":153186304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153350144,"data_stored":153350144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:14.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:14.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10840", 'reported_seq': 10938, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:12.944308+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:53:12.944308+0000', 'last_peered': '2024-09-18T11:53:12.944308+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:53:12.944308+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19742720, 'num_objects': 2410, 'num_object_clones': 0, 'num_object_copies': 9640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10840, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 307}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:14.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:14.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10505", 'reported_seq': 10622, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:13.181610+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:53:13.181610+0000', 'last_peered': '2024-09-18T11:53:13.181610+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:53:13.181610+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18864787, 'num_objects': 2304, 'num_object_clones': 0, 'num_object_copies': 9216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2304, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10504, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 264}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:14.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:14.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10563", 'reported_seq': 10656, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:09.019458+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:09.019458+0000', 'last_peered': '2024-09-18T11:53:09.019458+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:09.019458+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19354870, 'num_objects': 2363, 'num_object_clones': 0, 'num_object_copies': 9452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10561, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10549", 'reported_seq': 10670, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:12.847378+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:53:12.847378+0000', 'last_peered': '2024-09-18T11:53:12.847378+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:53:12.847378+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19215606, 'num_objects': 2346, 'num_object_clones': 0, 'num_object_copies': 9384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10547, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10502", 'reported_seq': 10621, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:12.972340+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:53:12.972340+0000', 'last_peered': '2024-09-18T11:53:12.972340+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:53:12.972340+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18710528, 'num_objects': 2284, 'num_object_clones': 0, 'num_object_copies': 9136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10502, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 165}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10627", 'reported_seq': 10734, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:08.961275+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:08.961275+0000', 'last_peered': '2024-09-18T11:53:08.961275+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:08.961275+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19567862, 'num_objects': 2389, 'num_object_clones': 0, 'num_object_copies': 9556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10625, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 273}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10431", 'reported_seq': 11181, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:08.964401+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:08.964401+0000', 'last_peered': '2024-09-18T11:53:08.964401+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:08.964401+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20018422, 'num_objects': 2444, 'num_object_clones': 0, 'num_object_copies': 9776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1696, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10429, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 342, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 748}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10314", 'reported_seq': 10428, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:13.233465+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:53:13.233465+0000', 'last_peered': '2024-09-18T11:53:13.233465+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:53:13.233465+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18674934, 'num_objects': 2280, 'num_object_clones': 0, 'num_object_copies': 9120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10312, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 317}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:14.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:14.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10644", 'reported_seq': 10765, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:12.997605+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:53:12.997605+0000', 'last_peered': '2024-09-18T11:53:12.997605+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:53:12.997605+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19566449, 'num_objects': 2389, 'num_object_clones': 0, 'num_object_copies': 9556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10641, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 355}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:14.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:14.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10725", 'reported_seq': 10817, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:12.827195+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:53:12.827195+0000', 'last_peered': '2024-09-18T11:53:12.827195+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:53:12.827195+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18889339, 'num_objects': 2306, 'num_object_clones': 0, 'num_object_copies': 9224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10724, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 222}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:14.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:14.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10634", 'reported_seq': 10732, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:13.141010+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:53:13.141010+0000', 'last_peered': '2024-09-18T11:53:13.141010+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:53:13.141010+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19110523, 'num_objects': 2333, 'num_object_clones': 0, 'num_object_copies': 9332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2333, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10633, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:14.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:14.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10730", 'reported_seq': 10854, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:12.923123+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:53:12.923123+0000', 'last_peered': '2024-09-18T11:53:12.923123+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:53:12.923123+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19330294, 'num_objects': 2360, 'num_object_clones': 0, 'num_object_copies': 9440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10728, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:14.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:14.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10359", 'reported_seq': 10449, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:08.876617+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:08.876617+0000', 'last_peered': '2024-09-18T11:53:08.876617+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:08.876617+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19231990, 'num_objects': 2348, 'num_object_clones': 0, 'num_object_copies': 9392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10357, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:14.999 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:53:15.685 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:15.685 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26497/150980 objects degraded (17.550%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":593,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37745,"data_bytes":309159308,"bytes_used":2758811648,"bytes_avail":383788244992,"bytes_total":386547056640,"degraded_objects":26497,"degraded_total":150980,"degraded_ratio":0.17550006623393827,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:53:15.705 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:53:15.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:16.258 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:16.258 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:16.274 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-18T11:53:16.041015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306013580,"num_objects":37361,"num_object_clones":0,"num_object_copies":149444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169726,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7005,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160653,"ondisk_log_size":160653,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694380,"kb_used_data":600940,"kb_used_omap":14,"kb_used_meta":2093361,"kb_avail":374792980,"statfs":{"total":386547056640,"available":383788011520,"internally_reserved":0,"allocated":615362560,"data_stored":614627276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143601691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,8,22,20,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17653760,"num_objects":-2155,"num_object_clones":0,"num_object_copies":-8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2155,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002866"},"pg_stats":[{"pgid":"1.f","version":"37'10840","reported_seq":10938,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.944308+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:12.944308+0000","last_peered":"2024-09-18T11:53:12.944308+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:12.944308+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19742720,"num_objects":2410,"num_object_clones":0,"num_object_copies":9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10700","reported_seq":14992,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:14.305346+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:14.305346+0000","last_peered":"2024-09-18T11:53:14.305346+0000","last_clean":"2024-09-18T11:53:14.305346+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:14.305346+0000","last_undegraded":"2024-09-18T11:53:14.305346+0000","last_fullsized":"2024-09-18T11:53:14.305346+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19297526,"num_objects":2356,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10698,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10505","reported_seq":10622,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.181610+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:13.181610+0000","last_peered":"2024-09-18T11:53:13.181610+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:13.181610+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18864787,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10504,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10633","reported_seq":10727,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.987427+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:13.987427+0000","last_peered":"2024-09-18T11:53:13.987427+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:13.987427+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18781430,"num_objects":2293,"num_object_clones":0,"num_object_copies":9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10631,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10549","reported_seq":10670,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.847378+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:12.847378+0000","last_peered":"2024-09-18T11:53:12.847378+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:12.847378+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19215606,"num_objects":2346,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10547,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10502","reported_seq":10621,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.972340+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:12.972340+0000","last_peered":"2024-09-18T11:53:12.972340+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:12.972340+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18710528,"num_objects":2284,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10502,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10682","reported_seq":10789,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.946355+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:13.946355+0000","last_peered":"2024-09-18T11:53:13.946355+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:13.946355+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19117302,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10680,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":218}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10498","reported_seq":11340,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.921327+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:13.921327+0000","last_peered":"2024-09-18T11:53:13.921327+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:13.921327+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19469558,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10496,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10814","reported_seq":15167,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:13.010181+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:13.010181+0000","last_peered":"2024-09-18T11:53:13.010181+0000","last_clean":"2024-09-18T11:53:13.010181+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:13.010181+0000","last_undegraded":"2024-09-18T11:53:13.010181+0000","last_fullsized":"2024-09-18T11:53:13.010181+0000","mapping_epoch":36,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19343852,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10314","reported_seq":10428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.233465+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:13.233465+0000","last_peered":"2024-09-18T11:53:13.233465+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:13.233465+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18674934,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10312,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10644","reported_seq":10765,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.997605+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:12.997605+0000","last_peered":"2024-09-18T11:53:12.997605+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:12.997605+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19566449,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10641,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10725","reported_seq":10817,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.827195+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:12.827195+0000","last_peered":"2024-09-18T11:53:12.827195+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:12.827195+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18889339,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10724,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":222}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10634","reported_seq":10732,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.141010+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:13.141010+0000","last_peered":"2024-09-18T11:53:13.141010+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:13.141010+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19110523,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10633,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10730","reported_seq":10854,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.923123+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:12.923123+0000","last_peered":"2024-09-18T11:53:12.923123+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:12.923123+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330294,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10728,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10568","reported_seq":14856,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:14.224484+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:14.224484+0000","last_peered":"2024-09-18T11:53:14.224484+0000","last_clean":"2024-09-18T11:53:14.224484+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:14.224484+0000","last_undegraded":"2024-09-18T11:53:14.224484+0000","last_fullsized":"2024-09-18T11:53:14.224484+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19125494,"num_objects":2335,"num_object_clones":0,"num_object_copies":9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10566,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10415","reported_seq":10505,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.905196+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:13.905196+0000","last_peered":"2024-09-18T11:53:13.905196+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:13.905196+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18773238,"num_objects":2292,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":290}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":306013580,"num_objects":37361,"num_object_clones":0,"num_object_copies":149444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169726,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7005,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":611741696,"data_stored":611741696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160653,"ondisk_log_size":160653,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749584,"kb_used_data":150852,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93622256,"statfs":{"total":96636764160,"available":95869190144,"internally_reserved":0,"allocated":154472448,"data_stored":154288627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.4429999999999996},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.0980000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":8.5399999999999991}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":8.5920000000000005},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":8.5060000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744428,"kb_used_data":150368,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93627412,"statfs":{"total":96636764160,"available":95874469888,"internally_reserved":0,"allocated":153976832,"data_stored":153793011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":4.1109999999999998},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":4.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.1070000000000002},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.109}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":4.242},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":19.797000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460348,"kb_used_data":150064,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93911492,"statfs":{"total":96636764160,"available":96165367808,"internally_reserved":0,"allocated":153665536,"data_stored":153481715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,8,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":6.2649999999999997},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.80200000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":6.3470000000000004},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":6.1740000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":6.2889999999999997},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":6.3120000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740020,"kb_used_data":149656,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93631820,"statfs":{"total":96636764160,"available":95878983680,"internally_reserved":0,"allocated":153247744,"data_stored":153063923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,8,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":25.960999999999999},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.3609999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.3170000000000002},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":6.3890000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":6.6890000000000001},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":6.3259999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":152526848,"data_stored":152526848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152678400,"data_stored":152678400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153186304,"data_stored":153186304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153350144,"data_stored":153350144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:19.276 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:19.881 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:19.882 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:19.904 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-18T11:53:18.041489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306013580,"num_objects":37361,"num_object_clones":0,"num_object_copies":149444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169726,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7005,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160653,"ondisk_log_size":160653,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694380,"kb_used_data":600940,"kb_used_omap":14,"kb_used_meta":2093361,"kb_avail":374792980,"statfs":{"total":386547056640,"available":383788011520,"internally_reserved":0,"allocated":615362560,"data_stored":614627276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143601691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,8,22,20,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16596992,"num_objects":-2026,"num_object_clones":0,"num_object_copies":-8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002806"},"pg_stats":[{"pgid":"1.f","version":"37'10840","reported_seq":10938,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.944308+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:12.944308+0000","last_peered":"2024-09-18T11:53:12.944308+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:12.944308+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19742720,"num_objects":2410,"num_object_clones":0,"num_object_copies":9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10700","reported_seq":14992,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:14.305346+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:14.305346+0000","last_peered":"2024-09-18T11:53:14.305346+0000","last_clean":"2024-09-18T11:53:14.305346+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:14.305346+0000","last_undegraded":"2024-09-18T11:53:14.305346+0000","last_fullsized":"2024-09-18T11:53:14.305346+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19297526,"num_objects":2356,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10698,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10505","reported_seq":10622,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.181610+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:13.181610+0000","last_peered":"2024-09-18T11:53:13.181610+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:13.181610+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18864787,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10504,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10633","reported_seq":10727,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.987427+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:13.987427+0000","last_peered":"2024-09-18T11:53:13.987427+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:13.987427+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18781430,"num_objects":2293,"num_object_clones":0,"num_object_copies":9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10631,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10549","reported_seq":10670,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.847378+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:12.847378+0000","last_peered":"2024-09-18T11:53:12.847378+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:12.847378+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19215606,"num_objects":2346,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10547,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10502","reported_seq":10621,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.972340+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:12.972340+0000","last_peered":"2024-09-18T11:53:12.972340+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:12.972340+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18710528,"num_objects":2284,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10502,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10682","reported_seq":10789,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.946355+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:13.946355+0000","last_peered":"2024-09-18T11:53:13.946355+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:13.946355+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19117302,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10680,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":218}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10498","reported_seq":11340,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.921327+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:13.921327+0000","last_peered":"2024-09-18T11:53:13.921327+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:13.921327+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19469558,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10496,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10814","reported_seq":15167,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:13.010181+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:13.010181+0000","last_peered":"2024-09-18T11:53:13.010181+0000","last_clean":"2024-09-18T11:53:13.010181+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:13.010181+0000","last_undegraded":"2024-09-18T11:53:13.010181+0000","last_fullsized":"2024-09-18T11:53:13.010181+0000","mapping_epoch":36,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19343852,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10314","reported_seq":10428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.233465+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:13.233465+0000","last_peered":"2024-09-18T11:53:13.233465+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:13.233465+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18674934,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10312,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10644","reported_seq":10765,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.997605+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:12.997605+0000","last_peered":"2024-09-18T11:53:12.997605+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:12.997605+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19566449,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10641,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10725","reported_seq":10817,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.827195+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:12.827195+0000","last_peered":"2024-09-18T11:53:12.827195+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:12.827195+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18889339,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10724,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":222}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10634","reported_seq":10732,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.141010+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:13.141010+0000","last_peered":"2024-09-18T11:53:13.141010+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:13.141010+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19110523,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10633,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10730","reported_seq":10854,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:12.923123+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:12.923123+0000","last_peered":"2024-09-18T11:53:12.923123+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:12.923123+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330294,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10728,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10568","reported_seq":14856,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:14.224484+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:14.224484+0000","last_peered":"2024-09-18T11:53:14.224484+0000","last_clean":"2024-09-18T11:53:14.224484+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:14.224484+0000","last_undegraded":"2024-09-18T11:53:14.224484+0000","last_fullsized":"2024-09-18T11:53:14.224484+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19125494,"num_objects":2335,"num_object_clones":0,"num_object_copies":9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10566,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10415","reported_seq":10505,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:13.905196+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:13.905196+0000","last_peered":"2024-09-18T11:53:13.905196+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:13.905196+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18773238,"num_objects":2292,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":290}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":306013580,"num_objects":37361,"num_object_clones":0,"num_object_copies":149444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169726,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7005,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":611741696,"data_stored":611741696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160653,"ondisk_log_size":160653,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749584,"kb_used_data":150852,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93622256,"statfs":{"total":96636764160,"available":95869190144,"internally_reserved":0,"allocated":154472448,"data_stored":154288627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":5.4429999999999996},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.0980000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":8.5399999999999991}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":8.5920000000000005},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":8.5060000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744428,"kb_used_data":150368,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93627412,"statfs":{"total":96636764160,"available":95874469888,"internally_reserved":0,"allocated":153976832,"data_stored":153793011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":4.1109999999999998},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":4.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":6.1070000000000002},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.109}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":4.242},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":19.797000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460348,"kb_used_data":150064,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93911492,"statfs":{"total":96636764160,"available":96165367808,"internally_reserved":0,"allocated":153665536,"data_stored":153481715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,8,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":6.2649999999999997},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.80200000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":6.3470000000000004},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":6.1740000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":6.2889999999999997},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":6.3120000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740020,"kb_used_data":149656,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93631820,"statfs":{"total":96636764160,"available":95878983680,"internally_reserved":0,"allocated":153247744,"data_stored":153063923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,8,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":25.960999999999999},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.3609999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:52:12 2024","interfaces":[{"interface":"back","average":{"1min":4.7430000000000003,"5min":6.2089999999999996,"15min":3.1440000000000001},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.14299999999999999},"max":{"1min":14.653,"5min":32.795000000000002,"15min":32.795000000000002},"last":7.3170000000000002},{"interface":"front","average":{"1min":6.8390000000000004,"5min":5.0650000000000004,"15min":2.915},"min":{"1min":0.42799999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":25.309999999999999,"5min":25.309999999999999,"15min":25.309999999999999},"last":6.3890000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":6.6890000000000001},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":6.3259999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":152526848,"data_stored":152526848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152678400,"data_stored":152678400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153186304,"data_stored":153186304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153350144,"data_stored":153350144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:19.906 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:20.488 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:20.488 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:20.505 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-18T11:53:20.041920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":297805196,"num_objects":36359,"num_object_clones":0,"num_object_copies":145436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170728,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7054,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661428,"kb_used_data":584644,"kb_used_omap":14,"kb_used_meta":2076721,"kb_avail":374825932,"statfs":{"total":386547056640,"available":383821754368,"internally_reserved":0,"allocated":598675456,"data_stored":597940172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126562331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,2,10,13,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21192704,"num_objects":-2587,"num_object_clones":0,"num_object_copies":-10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002743"},"pg_stats":[{"pgid":"1.f","version":"37'10916","reported_seq":11016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.157978+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:18.157978+0000","last_peered":"2024-09-18T11:53:18.157978+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:18.157978+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19120128,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":231}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10768","reported_seq":15060,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:19.209524+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:19.209524+0000","last_peered":"2024-09-18T11:53:19.209524+0000","last_clean":"2024-09-18T11:53:19.209524+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:19.209524+0000","last_undegraded":"2024-09-18T11:53:19.209524+0000","last_fullsized":"2024-09-18T11:53:19.209524+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18740470,"num_objects":2288,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10766,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10570","reported_seq":10687,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.318338+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:18.318338+0000","last_peered":"2024-09-18T11:53:18.318338+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:18.318338+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18332307,"num_objects":2239,"num_object_clones":0,"num_object_copies":8956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10569,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10685","reported_seq":10779,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.945535+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:18.945535+0000","last_peered":"2024-09-18T11:53:18.945535+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:18.945535+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355446,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10683,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10619","reported_seq":10740,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.079746+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:18.079746+0000","last_peered":"2024-09-18T11:53:18.079746+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:18.079746+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18642166,"num_objects":2276,"num_object_clones":0,"num_object_copies":9104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10617,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":224}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10566","reported_seq":10685,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.090704+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:18.090704+0000","last_peered":"2024-09-18T11:53:18.090704+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:18.090704+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18186240,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10566,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":101}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10743","reported_seq":10850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:19.058558+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:19.058558+0000","last_peered":"2024-09-18T11:53:19.058558+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:19.058558+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18617590,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":157}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10574","reported_seq":11513,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:19.057684+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:19.057684+0000","last_peered":"2024-09-18T11:53:19.057684+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:19.057684+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18846966,"num_objects":2301,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10572,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":700}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10874","reported_seq":15227,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:18.066552+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:18.066552+0000","last_peered":"2024-09-18T11:53:18.066552+0000","last_clean":"2024-09-18T11:53:18.066552+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:18.066552+0000","last_undegraded":"2024-09-18T11:53:18.066552+0000","last_fullsized":"2024-09-18T11:53:18.066552+0000","mapping_epoch":36,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18852332,"num_objects":2302,"num_object_clones":0,"num_object_copies":9208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10871,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10374","reported_seq":10488,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.063970+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:18.063970+0000","last_peered":"2024-09-18T11:53:18.063970+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:18.063970+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183414,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10372,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10713","reported_seq":10832,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.044897+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:18.044897+0000","last_peered":"2024-09-18T11:53:18.044897+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:18.044897+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19001201,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10780","reported_seq":10873,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:17.996810+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:17.996810+0000","last_peered":"2024-09-18T11:53:17.996810+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:17.996810+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18438779,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":167}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10707","reported_seq":10805,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.266274+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:18.266274+0000","last_peered":"2024-09-18T11:53:18.266274+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:18.266274+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18512507,"num_objects":2260,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10706,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10785","reported_seq":10910,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:17.867458+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:17.867458+0000","last_peered":"2024-09-18T11:53:17.867458+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:17.867458+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18879734,"num_objects":2305,"num_object_clones":0,"num_object_copies":9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10783,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":182}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10622","reported_seq":14911,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:19.173198+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:19.173198+0000","last_peered":"2024-09-18T11:53:19.173198+0000","last_clean":"2024-09-18T11:53:19.173198+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:19.173198+0000","last_undegraded":"2024-09-18T11:53:19.173198+0000","last_fullsized":"2024-09-18T11:53:19.173198+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18683126,"num_objects":2281,"num_object_clones":0,"num_object_copies":9124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10620,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10459","reported_seq":10548,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.877376+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:18.877376+0000","last_peered":"2024-09-18T11:53:18.877376+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:18.877376+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18412790,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":297805196,"num_objects":36359,"num_object_clones":0,"num_object_copies":145436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170728,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7054,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595124224,"data_stored":595124224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733004,"kb_used_data":146756,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93638836,"statfs":{"total":96636764160,"available":95886168064,"internally_reserved":0,"allocated":150278144,"data_stored":150094323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,6,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":14.856},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.28999999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":14.818},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.9019999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.8759999999999999},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":2.278}]}]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744540,"kb_used_data":146308,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93627300,"statfs":{"total":96636764160,"available":95874355200,"internally_reserved":0,"allocated":149819392,"data_stored":149635571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":5.2549999999999999},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.2930000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":12.145},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":5.3559999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.3300000000000001},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.69}]}]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443796,"kb_used_data":145996,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93928044,"statfs":{"total":96636764160,"available":96182317056,"internally_reserved":0,"allocated":149499904,"data_stored":149316083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":3.6400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":5.1059999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.6019999999999999},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":0.60199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740088,"kb_used_data":145584,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93631752,"statfs":{"total":96636764160,"available":95878914048,"internally_reserved":0,"allocated":149078016,"data_stored":148894195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":6.4139999999999997},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":1.4199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":0.311},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.4710000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":25.798999999999999},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":1.45}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148430848,"data_stored":148430848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148443136,"data_stored":148443136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149073920,"data_stored":149073920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149176320,"data_stored":149176320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:20.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:20.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10916", 'reported_seq': 11016, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.157978+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:53:18.157978+0000', 'last_peered': '2024-09-18T11:53:18.157978+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:53:18.157978+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19120128, 'num_objects': 2334, 'num_object_clones': 0, 'num_object_copies': 9336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10916, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 231}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:20.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:20.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10570", 'reported_seq': 10687, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.318338+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:53:18.318338+0000', 'last_peered': '2024-09-18T11:53:18.318338+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:53:18.318338+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18332307, 'num_objects': 2239, 'num_object_clones': 0, 'num_object_copies': 8956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2239, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10569, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 199}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:20.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:20.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10685", 'reported_seq': 10779, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.945535+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:18.945535+0000', 'last_peered': '2024-09-18T11:53:18.945535+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:18.945535+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18355446, 'num_objects': 2241, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10683, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:20.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:20.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10619", 'reported_seq': 10740, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.079746+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:53:18.079746+0000', 'last_peered': '2024-09-18T11:53:18.079746+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:53:18.079746+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18642166, 'num_objects': 2276, 'num_object_clones': 0, 'num_object_copies': 9104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10617, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 224}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:20.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:20.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10566", 'reported_seq': 10685, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.090704+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:53:18.090704+0000', 'last_peered': '2024-09-18T11:53:18.090704+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:53:18.090704+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18186240, 'num_objects': 2220, 'num_object_clones': 0, 'num_object_copies': 8880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10566, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 101}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:20.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10743", 'reported_seq': 10850, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:19.058558+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:19.058558+0000', 'last_peered': '2024-09-18T11:53:19.058558+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:19.058558+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18617590, 'num_objects': 2273, 'num_object_clones': 0, 'num_object_copies': 9092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10741, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 157}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10574", 'reported_seq': 11513, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:19.057684+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:19.057684+0000', 'last_peered': '2024-09-18T11:53:19.057684+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:19.057684+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18846966, 'num_objects': 2301, 'num_object_clones': 0, 'num_object_copies': 9204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10572, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 437, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 700}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10374", 'reported_seq': 10488, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.063970+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:53:18.063970+0000', 'last_peered': '2024-09-18T11:53:18.063970+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:53:18.063970+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18183414, 'num_objects': 2220, 'num_object_clones': 0, 'num_object_copies': 8880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10372, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10713", 'reported_seq': 10832, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.044897+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:53:18.044897+0000', 'last_peered': '2024-09-18T11:53:18.044897+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:53:18.044897+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19001201, 'num_objects': 2320, 'num_object_clones': 0, 'num_object_copies': 9280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10710, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 286}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10780", 'reported_seq': 10873, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:17.996810+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:53:17.996810+0000', 'last_peered': '2024-09-18T11:53:17.996810+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:53:17.996810+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18438779, 'num_objects': 2251, 'num_object_clones': 0, 'num_object_copies': 9004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10779, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:20.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10707", 'reported_seq': 10805, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.266274+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:53:18.266274+0000', 'last_peered': '2024-09-18T11:53:18.266274+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:53:18.266274+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18512507, 'num_objects': 2260, 'num_object_clones': 0, 'num_object_copies': 9040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10706, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 214}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:20.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:20.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10785", 'reported_seq': 10910, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:17.867458+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:53:17.867458+0000', 'last_peered': '2024-09-18T11:53:17.867458+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:53:17.867458+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18879734, 'num_objects': 2305, 'num_object_clones': 0, 'num_object_copies': 9220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10783, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 182}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:20.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:20.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10459", 'reported_seq': 10548, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.877376+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:18.877376+0000', 'last_peered': '2024-09-18T11:53:18.877376+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:18.877376+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18412790, 'num_objects': 2248, 'num_object_clones': 0, 'num_object_copies': 8992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10457, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 246}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:20.509 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:53:21.222 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:21.222 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26402/145436 objects degraded (18.154%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":599,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":36359,"data_bytes":297805196,"bytes_used":2725302272,"bytes_avail":383821754368,"bytes_total":386547056640,"degraded_objects":26402,"degraded_total":145436,"degraded_ratio":0.18153689595423417,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":215},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:53:21.246 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:53:21.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:21.819 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:21.819 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:21.837 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-18T11:53:20.041920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":297805196,"num_objects":36359,"num_object_clones":0,"num_object_copies":145436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170728,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7054,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661428,"kb_used_data":584644,"kb_used_omap":14,"kb_used_meta":2076721,"kb_avail":374825932,"statfs":{"total":386547056640,"available":383821754368,"internally_reserved":0,"allocated":598675456,"data_stored":597940172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126562331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,2,10,13,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21192704,"num_objects":-2587,"num_object_clones":0,"num_object_copies":-10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002743"},"pg_stats":[{"pgid":"1.f","version":"37'10916","reported_seq":11016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.157978+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:18.157978+0000","last_peered":"2024-09-18T11:53:18.157978+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:18.157978+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19120128,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":231}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10768","reported_seq":15060,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:19.209524+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:19.209524+0000","last_peered":"2024-09-18T11:53:19.209524+0000","last_clean":"2024-09-18T11:53:19.209524+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:19.209524+0000","last_undegraded":"2024-09-18T11:53:19.209524+0000","last_fullsized":"2024-09-18T11:53:19.209524+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18740470,"num_objects":2288,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10766,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10570","reported_seq":10687,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.318338+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:18.318338+0000","last_peered":"2024-09-18T11:53:18.318338+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:18.318338+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18332307,"num_objects":2239,"num_object_clones":0,"num_object_copies":8956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10569,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10685","reported_seq":10779,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.945535+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:18.945535+0000","last_peered":"2024-09-18T11:53:18.945535+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:18.945535+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355446,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10683,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10619","reported_seq":10740,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.079746+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:18.079746+0000","last_peered":"2024-09-18T11:53:18.079746+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:18.079746+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18642166,"num_objects":2276,"num_object_clones":0,"num_object_copies":9104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10617,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":224}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10566","reported_seq":10685,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.090704+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:18.090704+0000","last_peered":"2024-09-18T11:53:18.090704+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:18.090704+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18186240,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10566,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":101}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10743","reported_seq":10850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:19.058558+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:19.058558+0000","last_peered":"2024-09-18T11:53:19.058558+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:19.058558+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18617590,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":157}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10574","reported_seq":11513,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:19.057684+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:19.057684+0000","last_peered":"2024-09-18T11:53:19.057684+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:19.057684+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18846966,"num_objects":2301,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10572,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":700}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10874","reported_seq":15227,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:18.066552+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:18.066552+0000","last_peered":"2024-09-18T11:53:18.066552+0000","last_clean":"2024-09-18T11:53:18.066552+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:18.066552+0000","last_undegraded":"2024-09-18T11:53:18.066552+0000","last_fullsized":"2024-09-18T11:53:18.066552+0000","mapping_epoch":36,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18852332,"num_objects":2302,"num_object_clones":0,"num_object_copies":9208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10871,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10374","reported_seq":10488,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.063970+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:18.063970+0000","last_peered":"2024-09-18T11:53:18.063970+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:18.063970+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183414,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10372,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10713","reported_seq":10832,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.044897+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:18.044897+0000","last_peered":"2024-09-18T11:53:18.044897+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:18.044897+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19001201,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10780","reported_seq":10873,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:17.996810+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:17.996810+0000","last_peered":"2024-09-18T11:53:17.996810+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:17.996810+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18438779,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":167}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10707","reported_seq":10805,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.266274+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:18.266274+0000","last_peered":"2024-09-18T11:53:18.266274+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:18.266274+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18512507,"num_objects":2260,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10706,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10785","reported_seq":10910,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:17.867458+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:17.867458+0000","last_peered":"2024-09-18T11:53:17.867458+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:17.867458+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18879734,"num_objects":2305,"num_object_clones":0,"num_object_copies":9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10783,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":182}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10622","reported_seq":14911,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:19.173198+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:19.173198+0000","last_peered":"2024-09-18T11:53:19.173198+0000","last_clean":"2024-09-18T11:53:19.173198+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:19.173198+0000","last_undegraded":"2024-09-18T11:53:19.173198+0000","last_fullsized":"2024-09-18T11:53:19.173198+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18683126,"num_objects":2281,"num_object_clones":0,"num_object_copies":9124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10620,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10459","reported_seq":10548,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.877376+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:18.877376+0000","last_peered":"2024-09-18T11:53:18.877376+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:18.877376+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18412790,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":297805196,"num_objects":36359,"num_object_clones":0,"num_object_copies":145436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170728,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7054,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595124224,"data_stored":595124224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733004,"kb_used_data":146756,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93638836,"statfs":{"total":96636764160,"available":95886168064,"internally_reserved":0,"allocated":150278144,"data_stored":150094323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,6,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":6.665,"5min":5.8899999999999997,"15min":4.0640000000000001},"min":{"1min":0.217,"5min":0.186,"15min":0.155},"max":{"1min":21.420000000000002,"5min":21.539999999999999,"15min":21.539999999999999},"last":14.856},{"interface":"front","average":{"1min":7.7460000000000004,"5min":5.6509999999999998,"15min":4.0490000000000004},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":22.969000000000001,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.28999999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":14.818},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.9019999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:52:20 2024","interfaces":[{"interface":"back","average":{"1min":8.9269999999999996,"5min":7.1639999999999997,"15min":3.4129999999999998},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":28.827000000000002,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.8759999999999999},{"interface":"front","average":{"1min":3.9350000000000001,"5min":4.4329999999999998,"15min":3.532},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.20200000000000001},"max":{"1min":14.959,"5min":21.431999999999999,"15min":21.431999999999999},"last":2.278}]}]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744540,"kb_used_data":146308,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93627300,"statfs":{"total":96636764160,"available":95874355200,"internally_reserved":0,"allocated":149819392,"data_stored":149635571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":5.2549999999999999},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.2930000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":12.145},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":5.3559999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:52:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0709999999999997,"5min":4.2060000000000004,"15min":3.2440000000000002},"min":{"1min":0.38600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":15.058,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.3300000000000001},{"interface":"front","average":{"1min":4.6399999999999997,"5min":3.6150000000000002,"15min":2.6150000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":17.888999999999999,"5min":18.884,"15min":18.884},"last":11.69}]}]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443796,"kb_used_data":145996,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93928044,"statfs":{"total":96636764160,"available":96182317056,"internally_reserved":0,"allocated":149499904,"data_stored":149316083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":3.6400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":5.1059999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.6019999999999999},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":0.60199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740088,"kb_used_data":145584,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93631752,"statfs":{"total":96636764160,"available":95878914048,"internally_reserved":0,"allocated":149078016,"data_stored":148894195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":6.4139999999999997},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":1.4199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":0.311},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.4710000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":25.798999999999999},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":1.45}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148430848,"data_stored":148430848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148443136,"data_stored":148443136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149073920,"data_stored":149073920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149176320,"data_stored":149176320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:24.839 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:25.417 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:25.417 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:25.442 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-18T11:53:24.042892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":292660620,"num_objects":35731,"num_object_clones":0,"num_object_copies":142924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35731,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171356,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7054,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661672,"kb_used_data":576576,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374825688,"statfs":{"total":386547056640,"available":383821504512,"internally_reserved":0,"allocated":590413824,"data_stored":589678540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,5,2,8,14,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21610496,"num_objects":-2638,"num_object_clones":0,"num_object_copies":-10552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002774"},"pg_stats":[{"pgid":"1.f","version":"37'10976","reported_seq":11077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.080364+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:23.080364+0000","last_peered":"2024-09-18T11:53:23.080364+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:23.080364+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18628608,"num_objects":2274,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10976,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10768","reported_seq":15060,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:19.209524+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:19.209524+0000","last_peered":"2024-09-18T11:53:19.209524+0000","last_clean":"2024-09-18T11:53:19.209524+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:19.209524+0000","last_undegraded":"2024-09-18T11:53:19.209524+0000","last_fullsized":"2024-09-18T11:53:19.209524+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18740470,"num_objects":2288,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10766,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10630","reported_seq":10747,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.188067+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:23.188067+0000","last_peered":"2024-09-18T11:53:23.188067+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:23.188067+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17840787,"num_objects":2179,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10629,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":139}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10685","reported_seq":10779,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.945535+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:18.945535+0000","last_peered":"2024-09-18T11:53:18.945535+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:18.945535+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355446,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10683,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10675","reported_seq":10796,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:22.977875+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:22.977875+0000","last_peered":"2024-09-18T11:53:22.977875+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:22.977875+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183414,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10673,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10615","reported_seq":10734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.139715+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:23.139715+0000","last_peered":"2024-09-18T11:53:23.139715+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:23.139715+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17784832,"num_objects":2171,"num_object_clones":0,"num_object_copies":8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10615,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":52}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10743","reported_seq":10850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:19.058558+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:19.058558+0000","last_peered":"2024-09-18T11:53:19.058558+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:19.058558+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18617590,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":157}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10574","reported_seq":11513,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:19.057684+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:19.057684+0000","last_peered":"2024-09-18T11:53:19.057684+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:19.057684+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18846966,"num_objects":2301,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10572,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":700}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10951","reported_seq":15304,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:23.138729+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:23.138729+0000","last_peered":"2024-09-18T11:53:23.138729+0000","last_clean":"2024-09-18T11:53:23.138729+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:23.138729+0000","last_undegraded":"2024-09-18T11:53:23.138729+0000","last_fullsized":"2024-09-18T11:53:23.138729+0000","mapping_epoch":36,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18221548,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10449","reported_seq":10562,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.186829+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:23.186829+0000","last_peered":"2024-09-18T11:53:23.186829+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:23.186829+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17569014,"num_objects":2145,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10447,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":182}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10779","reported_seq":10900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.113061+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:23.113061+0000","last_peered":"2024-09-18T11:53:23.113061+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:23.113061+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18460529,"num_objects":2254,"num_object_clones":0,"num_object_copies":9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10843","reported_seq":10936,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.197495+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:23.197495+0000","last_peered":"2024-09-18T11:53:23.197495+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:23.197495+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17922683,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10773","reported_seq":10871,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.329168+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:23.329168+0000","last_peered":"2024-09-18T11:53:23.329168+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:23.329168+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17971835,"num_objects":2194,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10772,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":148}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10841","reported_seq":10966,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.021064+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:23.021064+0000","last_peered":"2024-09-18T11:53:23.021064+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:23.021064+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18420982,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10839,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10622","reported_seq":14911,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:19.173198+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:19.173198+0000","last_peered":"2024-09-18T11:53:19.173198+0000","last_clean":"2024-09-18T11:53:19.173198+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:19.173198+0000","last_undegraded":"2024-09-18T11:53:19.173198+0000","last_fullsized":"2024-09-18T11:53:19.173198+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18683126,"num_objects":2281,"num_object_clones":0,"num_object_copies":9124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10620,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10459","reported_seq":10548,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.877376+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:18.877376+0000","last_peered":"2024-09-18T11:53:18.877376+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:18.877376+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18412790,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":292660620,"num_objects":35731,"num_object_clones":0,"num_object_copies":142924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35731,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171356,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7054,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":586887168,"data_stored":586887168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733104,"kb_used_data":142692,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93638736,"statfs":{"total":96636764160,"available":95886065664,"internally_reserved":0,"allocated":146116608,"data_stored":145932787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,4,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.095},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.891}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.0640000000000001},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":12.029}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.157},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.6779999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744684,"kb_used_data":142304,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":93627156,"statfs":{"total":96636764160,"available":95874207744,"internally_reserved":0,"allocated":145719296,"data_stored":145535475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":15.929},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":15.949}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":8.9469999999999992},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":15.898999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":15.864000000000001},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":17.661999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443796,"kb_used_data":145996,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93928044,"statfs":{"total":96636764160,"available":96182317056,"internally_reserved":0,"allocated":149499904,"data_stored":149316083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":3.6400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":5.1059999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.6019999999999999},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":0.60199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740088,"kb_used_data":145584,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93631752,"statfs":{"total":96636764160,"available":95878914048,"internally_reserved":0,"allocated":149078016,"data_stored":148894195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":6.4139999999999997},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":1.4199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":0.311},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.4710000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":25.798999999999999},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":1.45}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148430848,"data_stored":148430848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148443136,"data_stored":148443136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144961536,"data_stored":144961536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145051648,"data_stored":145051648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:25.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:26.044 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:26.071 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-18T11:53:24.042892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":292660620,"num_objects":35731,"num_object_clones":0,"num_object_copies":142924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35731,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171356,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7054,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661672,"kb_used_data":576576,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374825688,"statfs":{"total":386547056640,"available":383821504512,"internally_reserved":0,"allocated":590413824,"data_stored":589678540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,5,2,8,14,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21610496,"num_objects":-2638,"num_object_clones":0,"num_object_copies":-10552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002774"},"pg_stats":[{"pgid":"1.f","version":"37'10976","reported_seq":11077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.080364+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:23.080364+0000","last_peered":"2024-09-18T11:53:23.080364+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:23.080364+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18628608,"num_objects":2274,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10976,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10768","reported_seq":15060,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:19.209524+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:19.209524+0000","last_peered":"2024-09-18T11:53:19.209524+0000","last_clean":"2024-09-18T11:53:19.209524+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:19.209524+0000","last_undegraded":"2024-09-18T11:53:19.209524+0000","last_fullsized":"2024-09-18T11:53:19.209524+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18740470,"num_objects":2288,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10766,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10630","reported_seq":10747,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.188067+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:23.188067+0000","last_peered":"2024-09-18T11:53:23.188067+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:23.188067+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17840787,"num_objects":2179,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10629,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":139}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10685","reported_seq":10779,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.945535+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:18.945535+0000","last_peered":"2024-09-18T11:53:18.945535+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:18.945535+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355446,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10683,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10675","reported_seq":10796,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:22.977875+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:22.977875+0000","last_peered":"2024-09-18T11:53:22.977875+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:22.977875+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183414,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10673,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10615","reported_seq":10734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.139715+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:23.139715+0000","last_peered":"2024-09-18T11:53:23.139715+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:23.139715+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17784832,"num_objects":2171,"num_object_clones":0,"num_object_copies":8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10615,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":52}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10743","reported_seq":10850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:19.058558+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:19.058558+0000","last_peered":"2024-09-18T11:53:19.058558+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:19.058558+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18617590,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":157}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10574","reported_seq":11513,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:19.057684+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:19.057684+0000","last_peered":"2024-09-18T11:53:19.057684+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:19.057684+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18846966,"num_objects":2301,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10572,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":700}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10951","reported_seq":15304,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:23.138729+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:23.138729+0000","last_peered":"2024-09-18T11:53:23.138729+0000","last_clean":"2024-09-18T11:53:23.138729+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:23.138729+0000","last_undegraded":"2024-09-18T11:53:23.138729+0000","last_fullsized":"2024-09-18T11:53:23.138729+0000","mapping_epoch":36,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18221548,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10449","reported_seq":10562,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.186829+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:23.186829+0000","last_peered":"2024-09-18T11:53:23.186829+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:23.186829+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17569014,"num_objects":2145,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10447,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":182}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10779","reported_seq":10900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.113061+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:23.113061+0000","last_peered":"2024-09-18T11:53:23.113061+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:23.113061+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18460529,"num_objects":2254,"num_object_clones":0,"num_object_copies":9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10843","reported_seq":10936,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.197495+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:23.197495+0000","last_peered":"2024-09-18T11:53:23.197495+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:23.197495+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17922683,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10773","reported_seq":10871,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.329168+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:23.329168+0000","last_peered":"2024-09-18T11:53:23.329168+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:23.329168+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17971835,"num_objects":2194,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10772,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":148}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10841","reported_seq":10966,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.021064+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:23.021064+0000","last_peered":"2024-09-18T11:53:23.021064+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:23.021064+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18420982,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10839,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10622","reported_seq":14911,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:19.173198+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:19.173198+0000","last_peered":"2024-09-18T11:53:19.173198+0000","last_clean":"2024-09-18T11:53:19.173198+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:19.173198+0000","last_undegraded":"2024-09-18T11:53:19.173198+0000","last_fullsized":"2024-09-18T11:53:19.173198+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18683126,"num_objects":2281,"num_object_clones":0,"num_object_copies":9124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10620,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10459","reported_seq":10548,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:18.877376+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:18.877376+0000","last_peered":"2024-09-18T11:53:18.877376+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:18.877376+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18412790,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":292660620,"num_objects":35731,"num_object_clones":0,"num_object_copies":142924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35731,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171356,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7054,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":586887168,"data_stored":586887168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733104,"kb_used_data":142692,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93638736,"statfs":{"total":96636764160,"available":95886065664,"internally_reserved":0,"allocated":146116608,"data_stored":145932787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,4,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.095},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.891}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.0640000000000001},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":12.029}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.157},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.6779999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744684,"kb_used_data":142304,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":93627156,"statfs":{"total":96636764160,"available":95874207744,"internally_reserved":0,"allocated":145719296,"data_stored":145535475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":15.929},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":15.949}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":8.9469999999999992},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":15.898999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":15.864000000000001},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":17.661999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443796,"kb_used_data":145996,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93928044,"statfs":{"total":96636764160,"available":96182317056,"internally_reserved":0,"allocated":149499904,"data_stored":149316083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":3.6400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":5.1059999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.6019999999999999},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":0.60199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740088,"kb_used_data":145584,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93631752,"statfs":{"total":96636764160,"available":95878914048,"internally_reserved":0,"allocated":149078016,"data_stored":148894195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":6.4139999999999997},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":1.4199999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":0.311},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.4710000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:52:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0589999999999993,"5min":5.0229999999999997,"15min":3.3980000000000001},"min":{"1min":0.35199999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.015000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":25.798999999999999},{"interface":"front","average":{"1min":10.02,"5min":7.1120000000000001,"15min":3.653},"min":{"1min":0.26100000000000001,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":32.947000000000003,"5min":32.947000000000003,"15min":32.947000000000003},"last":1.45}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148430848,"data_stored":148430848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148443136,"data_stored":148443136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144961536,"data_stored":144961536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145051648,"data_stored":145051648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:26.072 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:26.072 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10976", 'reported_seq': 11077, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:23.080364+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:53:23.080364+0000', 'last_peered': '2024-09-18T11:53:23.080364+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:53:23.080364+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18628608, 'num_objects': 2274, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10976, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 171}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:26.072 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:26.072 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10630", 'reported_seq': 10747, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:23.188067+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:53:23.188067+0000', 'last_peered': '2024-09-18T11:53:23.188067+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:53:23.188067+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17840787, 'num_objects': 2179, 'num_object_clones': 0, 'num_object_copies': 8716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2179, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10629, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 139}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:26.072 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:26.072 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10685", 'reported_seq': 10779, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.945535+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:18.945535+0000', 'last_peered': '2024-09-18T11:53:18.945535+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:18.945535+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18355446, 'num_objects': 2241, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10683, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:26.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:26.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10675", 'reported_seq': 10796, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:22.977875+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:53:22.977875+0000', 'last_peered': '2024-09-18T11:53:22.977875+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:53:22.977875+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18183414, 'num_objects': 2220, 'num_object_clones': 0, 'num_object_copies': 8880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10673, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 168}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:26.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:26.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10615", 'reported_seq': 10734, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:23.139715+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:53:23.139715+0000', 'last_peered': '2024-09-18T11:53:23.139715+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:53:23.139715+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17784832, 'num_objects': 2171, 'num_object_clones': 0, 'num_object_copies': 8684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10615, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 52}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:26.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:26.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10743", 'reported_seq': 10850, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:19.058558+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:19.058558+0000', 'last_peered': '2024-09-18T11:53:19.058558+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:19.058558+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18617590, 'num_objects': 2273, 'num_object_clones': 0, 'num_object_copies': 9092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10741, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 157}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:26.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:26.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10574", 'reported_seq': 11513, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:19.057684+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:19.057684+0000', 'last_peered': '2024-09-18T11:53:19.057684+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:19.057684+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18846966, 'num_objects': 2301, 'num_object_clones': 0, 'num_object_copies': 9204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10572, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 437, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 700}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:26.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:26.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10449", 'reported_seq': 10562, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:23.186829+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:53:23.186829+0000', 'last_peered': '2024-09-18T11:53:23.186829+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:53:23.186829+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17569014, 'num_objects': 2145, 'num_object_clones': 0, 'num_object_copies': 8580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10447, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 182}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:26.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:26.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10779", 'reported_seq': 10900, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:23.113061+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:53:23.113061+0000', 'last_peered': '2024-09-18T11:53:23.113061+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:53:23.113061+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18460529, 'num_objects': 2254, 'num_object_clones': 0, 'num_object_copies': 9016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10776, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 220}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:26.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:26.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10843", 'reported_seq': 10936, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:23.197495+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:53:23.197495+0000', 'last_peered': '2024-09-18T11:53:23.197495+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:53:23.197495+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17922683, 'num_objects': 2188, 'num_object_clones': 0, 'num_object_copies': 8752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10842, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:26.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:26.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10773", 'reported_seq': 10871, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:23.329168+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:53:23.329168+0000', 'last_peered': '2024-09-18T11:53:23.329168+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:53:23.329168+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17971835, 'num_objects': 2194, 'num_object_clones': 0, 'num_object_copies': 8776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10772, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 148}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:26.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:26.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10841", 'reported_seq': 10966, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:23.021064+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:53:23.021064+0000', 'last_peered': '2024-09-18T11:53:23.021064+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:53:23.021064+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18420982, 'num_objects': 2249, 'num_object_clones': 0, 'num_object_copies': 8996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10839, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:26.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:26.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10459", 'reported_seq': 10548, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:18.877376+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:18.877376+0000', 'last_peered': '2024-09-18T11:53:18.877376+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:18.877376+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18412790, 'num_objects': 2248, 'num_object_clones': 0, 'num_object_copies': 8992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10457, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 246}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:26.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:53:26.728 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:26.728 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26357/141396 objects degraded (18.641%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":604,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35349,"data_bytes":289531276,"bytes_used":2725130240,"bytes_avail":383821926400,"bytes_total":386547056640,"degraded_objects":26357,"degraded_total":141396,"degraded_ratio":0.18640555602704464,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":199},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:53:26.752 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:53:26.753 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:27.273 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:27.274 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:27.302 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-18T11:53:26.043375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":289531276,"num_objects":35349,"num_object_clones":0,"num_object_copies":141396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35349,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171738,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7099,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160765,"ondisk_log_size":160765,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661260,"kb_used_data":567824,"kb_used_omap":14,"kb_used_meta":2093361,"kb_avail":374826100,"statfs":{"total":386547056640,"available":383821926400,"internally_reserved":0,"allocated":581451776,"data_stored":580716492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143601691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,11,18,18,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19628032,"num_objects":-2396,"num_object_clones":0,"num_object_copies":-9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2396,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002832"},"pg_stats":[{"pgid":"1.f","version":"37'10976","reported_seq":11077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.080364+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:23.080364+0000","last_peered":"2024-09-18T11:53:23.080364+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:23.080364+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18628608,"num_objects":2274,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10976,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10829","reported_seq":15121,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:24.394242+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:24.394242+0000","last_peered":"2024-09-18T11:53:24.394242+0000","last_clean":"2024-09-18T11:53:24.394242+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:24.394242+0000","last_undegraded":"2024-09-18T11:53:24.394242+0000","last_fullsized":"2024-09-18T11:53:24.394242+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18240758,"num_objects":2227,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10630","reported_seq":10747,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.188067+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:23.188067+0000","last_peered":"2024-09-18T11:53:23.188067+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:23.188067+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17840787,"num_objects":2179,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10629,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":139}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10748","reported_seq":10842,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:24.022032+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:24.022032+0000","last_peered":"2024-09-18T11:53:24.022032+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:24.022032+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17839350,"num_objects":2178,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10746,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":77}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10675","reported_seq":10796,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:22.977875+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:22.977875+0000","last_peered":"2024-09-18T11:53:22.977875+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:22.977875+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183414,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10673,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10615","reported_seq":10734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.139715+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:23.139715+0000","last_peered":"2024-09-18T11:53:23.139715+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:23.139715+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17784832,"num_objects":2171,"num_object_clones":0,"num_object_copies":8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10615,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":52}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10806","reported_seq":10913,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:24.020857+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:24.020857+0000","last_peered":"2024-09-18T11:53:24.020857+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:24.020857+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18101494,"num_objects":2210,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":94}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10641","reported_seq":11670,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:24.080194+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:24.080194+0000","last_peered":"2024-09-18T11:53:24.080194+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:24.080194+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18298102,"num_objects":2234,"num_object_clones":0,"num_object_copies":8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10639,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":482,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":678}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10951","reported_seq":15304,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:23.138729+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:23.138729+0000","last_peered":"2024-09-18T11:53:23.138729+0000","last_clean":"2024-09-18T11:53:23.138729+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:23.138729+0000","last_undegraded":"2024-09-18T11:53:23.138729+0000","last_fullsized":"2024-09-18T11:53:23.138729+0000","mapping_epoch":36,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18221548,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10449","reported_seq":10562,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.186829+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:23.186829+0000","last_peered":"2024-09-18T11:53:23.186829+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:23.186829+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17569014,"num_objects":2145,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10447,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":182}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10779","reported_seq":10900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.113061+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:23.113061+0000","last_peered":"2024-09-18T11:53:23.113061+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:23.113061+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18460529,"num_objects":2254,"num_object_clones":0,"num_object_copies":9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10843","reported_seq":10936,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.197495+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:23.197495+0000","last_peered":"2024-09-18T11:53:23.197495+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:23.197495+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17922683,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10773","reported_seq":10871,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.329168+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:23.329168+0000","last_peered":"2024-09-18T11:53:23.329168+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:23.329168+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17971835,"num_objects":2194,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10772,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":148}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10841","reported_seq":10966,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:23.021064+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:23.021064+0000","last_peered":"2024-09-18T11:53:23.021064+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:23.021064+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18420982,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10839,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10685","reported_seq":14974,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:24.516659+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:24.516659+0000","last_peered":"2024-09-18T11:53:24.516659+0000","last_clean":"2024-09-18T11:53:24.516659+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:24.516659+0000","last_undegraded":"2024-09-18T11:53:24.516659+0000","last_fullsized":"2024-09-18T11:53:24.516659+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18167030,"num_objects":2218,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10683,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10524","reported_seq":10614,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:24.007884+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:24.007884+0000","last_peered":"2024-09-18T11:53:24.007884+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:24.007884+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17880310,"num_objects":2183,"num_object_clones":0,"num_object_copies":8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10522,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":181}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":289531276,"num_objects":35349,"num_object_clones":0,"num_object_copies":141396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35349,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171738,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7099,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":578523136,"data_stored":578523136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160765,"ondisk_log_size":160765,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733104,"kb_used_data":142692,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93638736,"statfs":{"total":96636764160,"available":95886065664,"internally_reserved":0,"allocated":146116608,"data_stored":145932787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,4,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.095},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.891}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.0640000000000001},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":12.029}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.157},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.6779999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744684,"kb_used_data":142304,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":93627156,"statfs":{"total":96636764160,"available":95874207744,"internally_reserved":0,"allocated":145719296,"data_stored":145535475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":15.929},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":15.949}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":8.9469999999999992},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":15.898999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":15.864000000000001},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":17.661999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443128,"kb_used_data":141152,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93928712,"statfs":{"total":96636764160,"available":96183001088,"internally_reserved":0,"allocated":144539648,"data_stored":144355827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":3.077},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":4.9489999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":45.701999999999998},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.40600000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.052},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":45.786999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740344,"kb_used_data":141676,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93631496,"statfs":{"total":96636764160,"available":95878651904,"internally_reserved":0,"allocated":145076224,"data_stored":144892403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.4019999999999999},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":8.7029999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":2.9889999999999999},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":5.0590000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":144334848,"data_stored":144334848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144175104,"data_stored":144175104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144961536,"data_stored":144961536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145051648,"data_stored":145051648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:30.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:30.806 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:30.806 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:30.825 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-18T11:53:30.044276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":281109900,"num_objects":34321,"num_object_clones":0,"num_object_copies":137284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172766,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7146,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160593,"ondisk_log_size":160593,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2628412,"kb_used_data":551604,"kb_used_omap":14,"kb_used_meta":2076721,"kb_avail":374858948,"statfs":{"total":386547056640,"available":383855562752,"internally_reserved":0,"allocated":564842496,"data_stored":564107212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126562331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,7,14,10,16,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24903680,"num_objects":-3040,"num_object_clones":0,"num_object_copies":-12160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002787"},"pg_stats":[{"pgid":"1.f","version":"37'11045","reported_seq":11145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.306069+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:28.306069+0000","last_peered":"2024-09-18T11:53:28.306069+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:28.306069+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18063360,"num_objects":2205,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11045,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10904","reported_seq":15196,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:29.680217+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:29.680217+0000","last_peered":"2024-09-18T11:53:29.680217+0000","last_clean":"2024-09-18T11:53:29.680217+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:29.680217+0000","last_undegraded":"2024-09-18T11:53:29.680217+0000","last_fullsized":"2024-09-18T11:53:29.680217+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17626358,"num_objects":2152,"num_object_clones":0,"num_object_copies":8608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10687","reported_seq":10804,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.161795+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:28.161795+0000","last_peered":"2024-09-18T11:53:28.161795+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:28.161795+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17373843,"num_objects":2122,"num_object_clones":0,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10686,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":82}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10820","reported_seq":10912,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:29.077525+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:29.077525+0000","last_peered":"2024-09-18T11:53:29.077525+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:29.077525+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17249526,"num_objects":2106,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10818,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10744","reported_seq":10864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.149766+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:28.149766+0000","last_peered":"2024-09-18T11:53:28.149766+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:28.149766+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17618166,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10742,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":99}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10671","reported_seq":10790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.141183+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:28.141183+0000","last_peered":"2024-09-18T11:53:28.141183+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:28.141183+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17326080,"num_objects":2115,"num_object_clones":0,"num_object_copies":8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10671,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10876","reported_seq":10983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:29.039060+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:29.039060+0000","last_peered":"2024-09-18T11:53:29.039060+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:29.039060+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17528054,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10874,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":24}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10702","reported_seq":11825,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:29.001570+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:29.001570+0000","last_peered":"2024-09-18T11:53:29.001570+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:29.001570+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17798390,"num_objects":2173,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":529,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11019","reported_seq":15372,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:28.190747+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:28.190747+0000","last_peered":"2024-09-18T11:53:28.190747+0000","last_clean":"2024-09-18T11:53:28.190747+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:28.190747+0000","last_undegraded":"2024-09-18T11:53:28.190747+0000","last_fullsized":"2024-09-18T11:53:28.190747+0000","mapping_epoch":36,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17664492,"num_objects":2157,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11016,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10513","reported_seq":10627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.154462+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:28.154462+0000","last_peered":"2024-09-18T11:53:28.154462+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:28.154462+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044726,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10511,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":118}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10843","reported_seq":10964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.112832+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:28.112832+0000","last_peered":"2024-09-18T11:53:28.112832+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:28.112832+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17936241,"num_objects":2190,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10910","reported_seq":11002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.161074+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:28.161074+0000","last_peered":"2024-09-18T11:53:28.161074+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:28.161074+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17373819,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10909,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":37}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10833","reported_seq":10931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.226362+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:28.226362+0000","last_peered":"2024-09-18T11:53:28.226362+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:28.226362+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17480315,"num_objects":2134,"num_object_clones":0,"num_object_copies":8536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10832,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":88}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10902","reported_seq":11026,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.280139+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:28.280139+0000","last_peered":"2024-09-18T11:53:28.280139+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:28.280139+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17921270,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10900,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10743","reported_seq":15032,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:29.551210+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:29.551210+0000","last_peered":"2024-09-18T11:53:29.551210+0000","last_clean":"2024-09-18T11:53:29.551210+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:29.551210+0000","last_undegraded":"2024-09-18T11:53:29.551210+0000","last_fullsized":"2024-09-18T11:53:29.551210+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17691894,"num_objects":2160,"num_object_clones":0,"num_object_copies":8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10581","reported_seq":10671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.856147+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:28.856147+0000","last_peered":"2024-09-18T11:53:28.856147+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:28.856147+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17413366,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":281109900,"num_objects":34321,"num_object_clones":0,"num_object_copies":137284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172766,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7146,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561807360,"data_stored":561807360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160593,"ondisk_log_size":160593,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733256,"kb_used_data":138688,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93638584,"statfs":{"total":96636764160,"available":95885910016,"internally_reserved":0,"allocated":142016512,"data_stored":141832691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,0,7,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.3499999999999996},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":12.964}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.4220000000000006},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":25.013999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.952},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":24.974}]}]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728096,"kb_used_data":138180,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93643744,"statfs":{"total":96636764160,"available":95891193856,"internally_reserved":0,"allocated":141496320,"data_stored":141312499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":12.869999999999999},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":12.903}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.4480000000000004},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":6.2999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443152,"kb_used_data":137016,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93928688,"statfs":{"total":96636764160,"available":96182976512,"internally_reserved":0,"allocated":140304384,"data_stored":140120563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,0,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":3.077},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":4.9489999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":10.497},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":2.1930000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.052},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":45.786999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723908,"kb_used_data":137720,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93647932,"statfs":{"total":96636764160,"available":95895482368,"internally_reserved":0,"allocated":141025280,"data_stored":140841459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,9,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":2.3380000000000001},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":12.289},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":12.381}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":12.311999999999999},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":12.339}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140161024,"data_stored":140161024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139964416,"data_stored":139964416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140816384,"data_stored":140816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140865536,"data_stored":140865536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:30.826 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:31.436 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:31.463 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-18T11:53:30.044276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":281109900,"num_objects":34321,"num_object_clones":0,"num_object_copies":137284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172766,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7146,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160593,"ondisk_log_size":160593,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2628412,"kb_used_data":551604,"kb_used_omap":14,"kb_used_meta":2076721,"kb_avail":374858948,"statfs":{"total":386547056640,"available":383855562752,"internally_reserved":0,"allocated":564842496,"data_stored":564107212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126562331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,7,14,10,16,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24903680,"num_objects":-3040,"num_object_clones":0,"num_object_copies":-12160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002787"},"pg_stats":[{"pgid":"1.f","version":"37'11045","reported_seq":11145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.306069+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:28.306069+0000","last_peered":"2024-09-18T11:53:28.306069+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:28.306069+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18063360,"num_objects":2205,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11045,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10904","reported_seq":15196,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:29.680217+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:29.680217+0000","last_peered":"2024-09-18T11:53:29.680217+0000","last_clean":"2024-09-18T11:53:29.680217+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:29.680217+0000","last_undegraded":"2024-09-18T11:53:29.680217+0000","last_fullsized":"2024-09-18T11:53:29.680217+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17626358,"num_objects":2152,"num_object_clones":0,"num_object_copies":8608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10687","reported_seq":10804,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.161795+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:28.161795+0000","last_peered":"2024-09-18T11:53:28.161795+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:28.161795+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17373843,"num_objects":2122,"num_object_clones":0,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10686,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":82}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10820","reported_seq":10912,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:29.077525+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:29.077525+0000","last_peered":"2024-09-18T11:53:29.077525+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:29.077525+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17249526,"num_objects":2106,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10818,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10744","reported_seq":10864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.149766+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:28.149766+0000","last_peered":"2024-09-18T11:53:28.149766+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:28.149766+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17618166,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10742,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":99}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10671","reported_seq":10790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.141183+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:28.141183+0000","last_peered":"2024-09-18T11:53:28.141183+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:28.141183+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17326080,"num_objects":2115,"num_object_clones":0,"num_object_copies":8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10671,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10876","reported_seq":10983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:29.039060+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:29.039060+0000","last_peered":"2024-09-18T11:53:29.039060+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:29.039060+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17528054,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10874,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":24}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10702","reported_seq":11825,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:29.001570+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:29.001570+0000","last_peered":"2024-09-18T11:53:29.001570+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:29.001570+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17798390,"num_objects":2173,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":529,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11019","reported_seq":15372,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:28.190747+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:28.190747+0000","last_peered":"2024-09-18T11:53:28.190747+0000","last_clean":"2024-09-18T11:53:28.190747+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:28.190747+0000","last_undegraded":"2024-09-18T11:53:28.190747+0000","last_fullsized":"2024-09-18T11:53:28.190747+0000","mapping_epoch":36,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17664492,"num_objects":2157,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11016,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10513","reported_seq":10627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.154462+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:28.154462+0000","last_peered":"2024-09-18T11:53:28.154462+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:28.154462+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044726,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10511,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":118}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10843","reported_seq":10964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.112832+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:28.112832+0000","last_peered":"2024-09-18T11:53:28.112832+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:28.112832+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17936241,"num_objects":2190,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10910","reported_seq":11002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.161074+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:28.161074+0000","last_peered":"2024-09-18T11:53:28.161074+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:28.161074+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17373819,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10909,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":37}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10833","reported_seq":10931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.226362+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:28.226362+0000","last_peered":"2024-09-18T11:53:28.226362+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:28.226362+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17480315,"num_objects":2134,"num_object_clones":0,"num_object_copies":8536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10832,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":88}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10902","reported_seq":11026,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.280139+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:28.280139+0000","last_peered":"2024-09-18T11:53:28.280139+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:28.280139+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17921270,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10900,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10743","reported_seq":15032,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:29.551210+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:29.551210+0000","last_peered":"2024-09-18T11:53:29.551210+0000","last_clean":"2024-09-18T11:53:29.551210+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:29.551210+0000","last_undegraded":"2024-09-18T11:53:29.551210+0000","last_fullsized":"2024-09-18T11:53:29.551210+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17691894,"num_objects":2160,"num_object_clones":0,"num_object_copies":8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10581","reported_seq":10671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.856147+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:28.856147+0000","last_peered":"2024-09-18T11:53:28.856147+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:28.856147+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17413366,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":281109900,"num_objects":34321,"num_object_clones":0,"num_object_copies":137284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172766,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7146,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561807360,"data_stored":561807360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160593,"ondisk_log_size":160593,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733256,"kb_used_data":138688,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93638584,"statfs":{"total":96636764160,"available":95885910016,"internally_reserved":0,"allocated":142016512,"data_stored":141832691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,0,7,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.3499999999999996},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":12.964}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.4220000000000006},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":25.013999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.952},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":24.974}]}]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728096,"kb_used_data":138180,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93643744,"statfs":{"total":96636764160,"available":95891193856,"internally_reserved":0,"allocated":141496320,"data_stored":141312499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":12.869999999999999},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":12.903}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.4480000000000004},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":6.2999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443152,"kb_used_data":137016,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93928688,"statfs":{"total":96636764160,"available":96182976512,"internally_reserved":0,"allocated":140304384,"data_stored":140120563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,0,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":3.077},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":4.9489999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":10.497},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":2.1930000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.052},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":45.786999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723908,"kb_used_data":137720,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93647932,"statfs":{"total":96636764160,"available":95895482368,"internally_reserved":0,"allocated":141025280,"data_stored":140841459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,9,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":2.3380000000000001},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":12.289},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":12.381}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":12.311999999999999},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":12.339}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140161024,"data_stored":140161024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139964416,"data_stored":139964416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140816384,"data_stored":140816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140865536,"data_stored":140865536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:31.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:31.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11045", 'reported_seq': 11145, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:28.306069+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:53:28.306069+0000', 'last_peered': '2024-09-18T11:53:28.306069+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:53:28.306069+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18063360, 'num_objects': 2205, 'num_object_clones': 0, 'num_object_copies': 8820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11045, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 102}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:31.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:31.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10687", 'reported_seq': 10804, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:28.161795+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:53:28.161795+0000', 'last_peered': '2024-09-18T11:53:28.161795+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:53:28.161795+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17373843, 'num_objects': 2122, 'num_object_clones': 0, 'num_object_copies': 8488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2122, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10686, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 82}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:31.465 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:31.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10820", 'reported_seq': 10912, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:29.077525+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:29.077525+0000', 'last_peered': '2024-09-18T11:53:29.077525+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:29.077525+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17249526, 'num_objects': 2106, 'num_object_clones': 0, 'num_object_copies': 8424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10818, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:31.465 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:31.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10744", 'reported_seq': 10864, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:28.149766+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:53:28.149766+0000', 'last_peered': '2024-09-18T11:53:28.149766+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:53:28.149766+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17618166, 'num_objects': 2151, 'num_object_clones': 0, 'num_object_copies': 8604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10742, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 99}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:31.465 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:31.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10671", 'reported_seq': 10790, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:28.141183+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:53:28.141183+0000', 'last_peered': '2024-09-18T11:53:28.141183+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:53:28.141183+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17326080, 'num_objects': 2115, 'num_object_clones': 0, 'num_object_copies': 8460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2115, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10671, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -4}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:31.465 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:31.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10876", 'reported_seq': 10983, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:29.039060+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:29.039060+0000', 'last_peered': '2024-09-18T11:53:29.039060+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:29.039060+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17528054, 'num_objects': 2140, 'num_object_clones': 0, 'num_object_copies': 8560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10874, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 24}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:31.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:31.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10702", 'reported_seq': 11825, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:29.001570+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:29.001570+0000', 'last_peered': '2024-09-18T11:53:29.001570+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:29.001570+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17798390, 'num_objects': 2173, 'num_object_clones': 0, 'num_object_copies': 8692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1509, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10700, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 529, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 664}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:31.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:31.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10513", 'reported_seq': 10627, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:28.154462+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:53:28.154462+0000', 'last_peered': '2024-09-18T11:53:28.154462+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:53:28.154462+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17044726, 'num_objects': 2081, 'num_object_clones': 0, 'num_object_copies': 8324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10511, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 118}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:31.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:31.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10843", 'reported_seq': 10964, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:28.112832+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:53:28.112832+0000', 'last_peered': '2024-09-18T11:53:28.112832+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:53:28.112832+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17936241, 'num_objects': 2190, 'num_object_clones': 0, 'num_object_copies': 8760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10840, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 156}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:31.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:31.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10910", 'reported_seq': 11002, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:28.161074+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:53:28.161074+0000', 'last_peered': '2024-09-18T11:53:28.161074+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:53:28.161074+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17373819, 'num_objects': 2121, 'num_object_clones': 0, 'num_object_copies': 8484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10909, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 37}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:31.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:31.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10833", 'reported_seq': 10931, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:28.226362+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:53:28.226362+0000', 'last_peered': '2024-09-18T11:53:28.226362+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:53:28.226362+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17480315, 'num_objects': 2134, 'num_object_clones': 0, 'num_object_copies': 8536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10832, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 88}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:31.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:31.467 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10902", 'reported_seq': 11026, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:28.280139+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:53:28.280139+0000', 'last_peered': '2024-09-18T11:53:28.280139+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:53:28.280139+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17921270, 'num_objects': 2188, 'num_object_clones': 0, 'num_object_copies': 8752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10900, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 65}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:31.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:31.467 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10581", 'reported_seq': 10671, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:28.856147+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:28.856147+0000', 'last_peered': '2024-09-18T11:53:28.856147+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:28.856147+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17413366, 'num_objects': 2126, 'num_object_clones': 0, 'num_object_copies': 8504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10579, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:31.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:53:32.147 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:32.147 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26310/137284 objects degraded (19.165%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":610,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":34321,"data_bytes":281109900,"bytes_used":2691493888,"bytes_avail":383855562752,"bytes_total":386547056640,"degraded_objects":26310,"degraded_total":137284,"degraded_ratio":0.19164651379621805,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":253},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:53:32.174 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:53:32.175 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:32.735 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:32.735 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:32.757 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-18T11:53:32.044806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":281109900,"num_objects":34321,"num_object_clones":0,"num_object_copies":137284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172766,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7146,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160593,"ondisk_log_size":160593,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2628412,"kb_used_data":551604,"kb_used_omap":14,"kb_used_meta":2076721,"kb_avail":374858948,"statfs":{"total":386547056640,"available":383855562752,"internally_reserved":0,"allocated":564842496,"data_stored":564107212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126562331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,7,14,10,16,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16695296,"num_objects":-2038,"num_object_clones":0,"num_object_copies":-8152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002886"},"pg_stats":[{"pgid":"1.f","version":"37'11045","reported_seq":11145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.306069+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:28.306069+0000","last_peered":"2024-09-18T11:53:28.306069+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:28.306069+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18063360,"num_objects":2205,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11045,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10904","reported_seq":15196,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:29.680217+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:29.680217+0000","last_peered":"2024-09-18T11:53:29.680217+0000","last_clean":"2024-09-18T11:53:29.680217+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:29.680217+0000","last_undegraded":"2024-09-18T11:53:29.680217+0000","last_fullsized":"2024-09-18T11:53:29.680217+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17626358,"num_objects":2152,"num_object_clones":0,"num_object_copies":8608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10687","reported_seq":10804,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.161795+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:28.161795+0000","last_peered":"2024-09-18T11:53:28.161795+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:28.161795+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17373843,"num_objects":2122,"num_object_clones":0,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10686,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":82}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10820","reported_seq":10912,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:29.077525+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:29.077525+0000","last_peered":"2024-09-18T11:53:29.077525+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:29.077525+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17249526,"num_objects":2106,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10818,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10744","reported_seq":10864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.149766+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:28.149766+0000","last_peered":"2024-09-18T11:53:28.149766+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:28.149766+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17618166,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10742,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":99}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10671","reported_seq":10790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.141183+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:28.141183+0000","last_peered":"2024-09-18T11:53:28.141183+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:28.141183+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17326080,"num_objects":2115,"num_object_clones":0,"num_object_copies":8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10671,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10876","reported_seq":10983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:29.039060+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:29.039060+0000","last_peered":"2024-09-18T11:53:29.039060+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:29.039060+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17528054,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10874,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":24}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10702","reported_seq":11825,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:29.001570+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:29.001570+0000","last_peered":"2024-09-18T11:53:29.001570+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:29.001570+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17798390,"num_objects":2173,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":529,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11019","reported_seq":15372,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:28.190747+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:28.190747+0000","last_peered":"2024-09-18T11:53:28.190747+0000","last_clean":"2024-09-18T11:53:28.190747+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:28.190747+0000","last_undegraded":"2024-09-18T11:53:28.190747+0000","last_fullsized":"2024-09-18T11:53:28.190747+0000","mapping_epoch":36,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17664492,"num_objects":2157,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11016,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10513","reported_seq":10627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.154462+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:28.154462+0000","last_peered":"2024-09-18T11:53:28.154462+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:28.154462+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044726,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10511,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":118}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10843","reported_seq":10964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.112832+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:28.112832+0000","last_peered":"2024-09-18T11:53:28.112832+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:28.112832+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17936241,"num_objects":2190,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10910","reported_seq":11002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.161074+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:28.161074+0000","last_peered":"2024-09-18T11:53:28.161074+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:28.161074+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17373819,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10909,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":37}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10833","reported_seq":10931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.226362+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:28.226362+0000","last_peered":"2024-09-18T11:53:28.226362+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:28.226362+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17480315,"num_objects":2134,"num_object_clones":0,"num_object_copies":8536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10832,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":88}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10902","reported_seq":11026,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.280139+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:28.280139+0000","last_peered":"2024-09-18T11:53:28.280139+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:28.280139+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17921270,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10900,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10743","reported_seq":15032,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:29.551210+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:29.551210+0000","last_peered":"2024-09-18T11:53:29.551210+0000","last_clean":"2024-09-18T11:53:29.551210+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:29.551210+0000","last_undegraded":"2024-09-18T11:53:29.551210+0000","last_fullsized":"2024-09-18T11:53:29.551210+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17691894,"num_objects":2160,"num_object_clones":0,"num_object_copies":8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10581","reported_seq":10671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:28.856147+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:28.856147+0000","last_peered":"2024-09-18T11:53:28.856147+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:28.856147+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17413366,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":281109900,"num_objects":34321,"num_object_clones":0,"num_object_copies":137284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172766,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7146,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561807360,"data_stored":561807360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160593,"ondisk_log_size":160593,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733256,"kb_used_data":138688,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93638584,"statfs":{"total":96636764160,"available":95885910016,"internally_reserved":0,"allocated":142016512,"data_stored":141832691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,0,7,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.3499999999999996},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":12.964}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.4220000000000006},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":25.013999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.952},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":24.974}]}]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728096,"kb_used_data":138180,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93643744,"statfs":{"total":96636764160,"available":95891193856,"internally_reserved":0,"allocated":141496320,"data_stored":141312499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":12.869999999999999},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":12.903}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.4480000000000004},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":6.2999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443152,"kb_used_data":137016,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93928688,"statfs":{"total":96636764160,"available":96182976512,"internally_reserved":0,"allocated":140304384,"data_stored":140120563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,0,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":3.077},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":4.9489999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":10.497},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":2.1930000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.052},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":45.786999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723908,"kb_used_data":137720,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93647932,"statfs":{"total":96636764160,"available":95895482368,"internally_reserved":0,"allocated":141025280,"data_stored":140841459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,9,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":2.3380000000000001},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":12.289},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":12.381}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":12.311999999999999},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":12.339}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140161024,"data_stored":140161024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139964416,"data_stored":139964416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140816384,"data_stored":140816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140865536,"data_stored":140865536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:35.759 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:36.331 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:36.362 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-18T11:53:36.045749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":272868748,"num_objects":33315,"num_object_clones":0,"num_object_copies":133260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173772,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2624616,"kb_used_data":535360,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374862744,"statfs":{"total":386547056640,"available":383859449856,"internally_reserved":0,"allocated":548208640,"data_stored":547473356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,11,12,11,14,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19791872,"num_objects":-2416,"num_object_clones":0,"num_object_copies":-9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002856"},"pg_stats":[{"pgid":"1.f","version":"37'11111","reported_seq":11212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.260475+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:33.260475+0000","last_peered":"2024-09-18T11:53:33.260475+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:33.260475+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17522688,"num_objects":2139,"num_object_clones":0,"num_object_copies":8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":36}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10968","reported_seq":15260,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:34.859893+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:34.859893+0000","last_peered":"2024-09-18T11:53:34.859893+0000","last_clean":"2024-09-18T11:53:34.859893+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:34.859893+0000","last_undegraded":"2024-09-18T11:53:34.859893+0000","last_fullsized":"2024-09-18T11:53:34.859893+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17102070,"num_objects":2088,"num_object_clones":0,"num_object_copies":8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10966,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10733","reported_seq":10850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:32.864780+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:32.864780+0000","last_peered":"2024-09-18T11:53:32.864780+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:32.864780+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16997011,"num_objects":2076,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10732,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":36}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10880","reported_seq":10974,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:34.041046+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:34.041046+0000","last_peered":"2024-09-18T11:53:34.041046+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:34.041046+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16758006,"num_objects":2046,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-55}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10799","reported_seq":10920,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.012858+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:33.012858+0000","last_peered":"2024-09-18T11:53:33.012858+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:33.012858+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167606,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10797,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":44}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10738","reported_seq":10857,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.221018+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:33.221018+0000","last_peered":"2024-09-18T11:53:33.221018+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:33.221018+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10738,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-71}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10943","reported_seq":11050,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:34.014253+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:34.014253+0000","last_peered":"2024-09-18T11:53:34.014253+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:34.014253+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16979190,"num_objects":2073,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-43}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10766","reported_seq":11992,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:34.140105+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:34.140105+0000","last_peered":"2024-09-18T11:53:34.140105+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:34.140105+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17274102,"num_objects":2109,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10764,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11079","reported_seq":15431,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:33.246221+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:33.246221+0000","last_peered":"2024-09-18T11:53:33.246221+0000","last_clean":"2024-09-18T11:53:33.246221+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:33.246221+0000","last_undegraded":"2024-09-18T11:53:33.246221+0000","last_fullsized":"2024-09-18T11:53:33.246221+0000","mapping_epoch":36,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17172972,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11076,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10571","reported_seq":10685,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.163892+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:33.163892+0000","last_peered":"2024-09-18T11:53:33.163892+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:33.163892+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16569590,"num_objects":2023,"num_object_clones":0,"num_object_copies":8092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10569,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10908","reported_seq":11029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.198996+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:33.198996+0000","last_peered":"2024-09-18T11:53:33.198996+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:33.198996+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17403761,"num_objects":2125,"num_object_clones":0,"num_object_copies":8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10990","reported_seq":11083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.253083+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:33.253083+0000","last_peered":"2024-09-18T11:53:33.253083+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:33.253083+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16718459,"num_objects":2041,"num_object_clones":0,"num_object_copies":8164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10989,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-43}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10904","reported_seq":11002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.336993+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:33.336993+0000","last_peered":"2024-09-18T11:53:33.336993+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:33.336993+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16898683,"num_objects":2063,"num_object_clones":0,"num_object_copies":8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10959","reported_seq":11084,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:32.625350+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:32.625350+0000","last_peered":"2024-09-18T11:53:32.625350+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:32.625350+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17454326,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10957,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10812","reported_seq":15101,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:34.932875+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:34.932875+0000","last_peered":"2024-09-18T11:53:34.932875+0000","last_clean":"2024-09-18T11:53:34.932875+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:34.932875+0000","last_undegraded":"2024-09-18T11:53:34.932875+0000","last_fullsized":"2024-09-18T11:53:34.932875+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17126646,"num_objects":2091,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10638","reported_seq":10728,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.957525+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:33.957525+0000","last_peered":"2024-09-18T11:53:33.957525+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:33.957525+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16946422,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10636,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":67}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":272868748,"num_objects":33315,"num_object_clones":0,"num_object_copies":133260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173772,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":545103872,"data_stored":545103872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733300,"kb_used_data":134572,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93638540,"statfs":{"total":96636764160,"available":95885864960,"internally_reserved":0,"allocated":137801728,"data_stored":137617907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":11.34},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.56299999999999994}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":16.898}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.214},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.314}]}]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724080,"kb_used_data":134184,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93647760,"statfs":{"total":96636764160,"available":95895306240,"internally_reserved":0,"allocated":137404416,"data_stored":137220595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":12.869999999999999},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":12.903}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.4480000000000004},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":6.2999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443296,"kb_used_data":133008,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93928544,"statfs":{"total":96636764160,"available":96182829056,"internally_reserved":0,"allocated":136200192,"data_stored":136016371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,3,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":11},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.7959999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":5.7149999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":9.9789999999999992}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":9.1609999999999996},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":10.050000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723940,"kb_used_data":133596,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93647900,"statfs":{"total":96636764160,"available":95895449600,"internally_reserved":0,"allocated":136802304,"data_stored":136618483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":1.1379999999999999},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.1080000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.501},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":1.0780000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135999488,"data_stored":135999488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135716864,"data_stored":135716864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136667136,"data_stored":136667136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":136720384,"data_stored":136720384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:36.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:36.890 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:36.890 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:36.909 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-18T11:53:36.045749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":272868748,"num_objects":33315,"num_object_clones":0,"num_object_copies":133260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173772,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2624616,"kb_used_data":535360,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374862744,"statfs":{"total":386547056640,"available":383859449856,"internally_reserved":0,"allocated":548208640,"data_stored":547473356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,11,12,11,14,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19791872,"num_objects":-2416,"num_object_clones":0,"num_object_copies":-9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002856"},"pg_stats":[{"pgid":"1.f","version":"37'11111","reported_seq":11212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.260475+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:33.260475+0000","last_peered":"2024-09-18T11:53:33.260475+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:33.260475+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17522688,"num_objects":2139,"num_object_clones":0,"num_object_copies":8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":36}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10968","reported_seq":15260,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:34.859893+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:34.859893+0000","last_peered":"2024-09-18T11:53:34.859893+0000","last_clean":"2024-09-18T11:53:34.859893+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:34.859893+0000","last_undegraded":"2024-09-18T11:53:34.859893+0000","last_fullsized":"2024-09-18T11:53:34.859893+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17102070,"num_objects":2088,"num_object_clones":0,"num_object_copies":8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10966,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10733","reported_seq":10850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:32.864780+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:32.864780+0000","last_peered":"2024-09-18T11:53:32.864780+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:32.864780+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16997011,"num_objects":2076,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10732,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":36}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10880","reported_seq":10974,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:34.041046+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:34.041046+0000","last_peered":"2024-09-18T11:53:34.041046+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:34.041046+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16758006,"num_objects":2046,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-55}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10799","reported_seq":10920,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.012858+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:33.012858+0000","last_peered":"2024-09-18T11:53:33.012858+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:33.012858+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167606,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10797,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":44}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10738","reported_seq":10857,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.221018+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:33.221018+0000","last_peered":"2024-09-18T11:53:33.221018+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:33.221018+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10738,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-71}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10943","reported_seq":11050,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:34.014253+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:34.014253+0000","last_peered":"2024-09-18T11:53:34.014253+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:34.014253+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16979190,"num_objects":2073,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-43}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10766","reported_seq":11992,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:34.140105+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:34.140105+0000","last_peered":"2024-09-18T11:53:34.140105+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:34.140105+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17274102,"num_objects":2109,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10764,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11079","reported_seq":15431,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:33.246221+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:33.246221+0000","last_peered":"2024-09-18T11:53:33.246221+0000","last_clean":"2024-09-18T11:53:33.246221+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:33.246221+0000","last_undegraded":"2024-09-18T11:53:33.246221+0000","last_fullsized":"2024-09-18T11:53:33.246221+0000","mapping_epoch":36,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17172972,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11076,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10571","reported_seq":10685,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.163892+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:33.163892+0000","last_peered":"2024-09-18T11:53:33.163892+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:33.163892+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16569590,"num_objects":2023,"num_object_clones":0,"num_object_copies":8092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10569,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10908","reported_seq":11029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.198996+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:33.198996+0000","last_peered":"2024-09-18T11:53:33.198996+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:33.198996+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17403761,"num_objects":2125,"num_object_clones":0,"num_object_copies":8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10990","reported_seq":11083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.253083+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:33.253083+0000","last_peered":"2024-09-18T11:53:33.253083+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:33.253083+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16718459,"num_objects":2041,"num_object_clones":0,"num_object_copies":8164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10989,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-43}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10904","reported_seq":11002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.336993+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:33.336993+0000","last_peered":"2024-09-18T11:53:33.336993+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:33.336993+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16898683,"num_objects":2063,"num_object_clones":0,"num_object_copies":8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10959","reported_seq":11084,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:32.625350+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:32.625350+0000","last_peered":"2024-09-18T11:53:32.625350+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:32.625350+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17454326,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10957,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10812","reported_seq":15101,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:34.932875+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:34.932875+0000","last_peered":"2024-09-18T11:53:34.932875+0000","last_clean":"2024-09-18T11:53:34.932875+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:34.932875+0000","last_undegraded":"2024-09-18T11:53:34.932875+0000","last_fullsized":"2024-09-18T11:53:34.932875+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17126646,"num_objects":2091,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10638","reported_seq":10728,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.957525+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:33.957525+0000","last_peered":"2024-09-18T11:53:33.957525+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:33.957525+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16946422,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10636,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":67}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":272868748,"num_objects":33315,"num_object_clones":0,"num_object_copies":133260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173772,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":545103872,"data_stored":545103872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733300,"kb_used_data":134572,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93638540,"statfs":{"total":96636764160,"available":95885864960,"internally_reserved":0,"allocated":137801728,"data_stored":137617907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":11.34},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.56299999999999994}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":16.898}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.214},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.314}]}]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724080,"kb_used_data":134184,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93647760,"statfs":{"total":96636764160,"available":95895306240,"internally_reserved":0,"allocated":137404416,"data_stored":137220595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":12.869999999999999},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":12.903}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.4480000000000004},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":6.2999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443296,"kb_used_data":133008,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93928544,"statfs":{"total":96636764160,"available":96182829056,"internally_reserved":0,"allocated":136200192,"data_stored":136016371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,3,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":11},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.7959999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":5.7149999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":9.9789999999999992}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":9.1609999999999996},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":10.050000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723940,"kb_used_data":133596,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93647900,"statfs":{"total":96636764160,"available":95895449600,"internally_reserved":0,"allocated":136802304,"data_stored":136618483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":1.1379999999999999},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.1080000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.501},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":1.0780000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135999488,"data_stored":135999488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135716864,"data_stored":135716864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136667136,"data_stored":136667136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":136720384,"data_stored":136720384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:36.911 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:36.911 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11111", 'reported_seq': 11212, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:33.260475+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:53:33.260475+0000', 'last_peered': '2024-09-18T11:53:33.260475+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:53:33.260475+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17522688, 'num_objects': 2139, 'num_object_clones': 0, 'num_object_copies': 8556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11111, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 36}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:36.911 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:36.911 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10733", 'reported_seq': 10850, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:32.864780+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:53:32.864780+0000', 'last_peered': '2024-09-18T11:53:32.864780+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:53:32.864780+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16997011, 'num_objects': 2076, 'num_object_clones': 0, 'num_object_copies': 8304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2076, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10732, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 36}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:36.911 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:36.911 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10880", 'reported_seq': 10974, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:34.041046+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:34.041046+0000', 'last_peered': '2024-09-18T11:53:34.041046+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:34.041046+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16758006, 'num_objects': 2046, 'num_object_clones': 0, 'num_object_copies': 8184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10878, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -55}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:36.911 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:36.911 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10799", 'reported_seq': 10920, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:33.012858+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:53:33.012858+0000', 'last_peered': '2024-09-18T11:53:33.012858+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:53:33.012858+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17167606, 'num_objects': 2096, 'num_object_clones': 0, 'num_object_copies': 8384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10797, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 44}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:36.911 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:36.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10738", 'reported_seq': 10857, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:33.221018+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:53:33.221018+0000', 'last_peered': '2024-09-18T11:53:33.221018+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:53:33.221018+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16777216, 'num_objects': 2048, 'num_object_clones': 0, 'num_object_copies': 8192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2048, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10738, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -71}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:36.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:36.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10943", 'reported_seq': 11050, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:34.014253+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:34.014253+0000', 'last_peered': '2024-09-18T11:53:34.014253+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:34.014253+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16979190, 'num_objects': 2073, 'num_object_clones': 0, 'num_object_copies': 8292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10941, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -43}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:36.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:36.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10766", 'reported_seq': 11992, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:34.140105+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:34.140105+0000', 'last_peered': '2024-09-18T11:53:34.140105+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:34.140105+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17274102, 'num_objects': 2109, 'num_object_clones': 0, 'num_object_copies': 8436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1458, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10764, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 580, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:36.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:36.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10571", 'reported_seq': 10685, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:33.163892+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:53:33.163892+0000', 'last_peered': '2024-09-18T11:53:33.163892+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:53:33.163892+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16569590, 'num_objects': 2023, 'num_object_clones': 0, 'num_object_copies': 8092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2023, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10569, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 60}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:36.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:36.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10908", 'reported_seq': 11029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:33.198996+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:53:33.198996+0000', 'last_peered': '2024-09-18T11:53:33.198996+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:53:33.198996+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17403761, 'num_objects': 2125, 'num_object_clones': 0, 'num_object_copies': 8500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10905, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 91}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:36.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:36.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10990", 'reported_seq': 11083, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:33.253083+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:53:33.253083+0000', 'last_peered': '2024-09-18T11:53:33.253083+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:53:33.253083+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16718459, 'num_objects': 2041, 'num_object_clones': 0, 'num_object_copies': 8164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10989, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -43}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:36.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:36.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10904", 'reported_seq': 11002, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:33.336993+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:53:33.336993+0000', 'last_peered': '2024-09-18T11:53:33.336993+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:53:33.336993+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16898683, 'num_objects': 2063, 'num_object_clones': 0, 'num_object_copies': 8252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10903, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 17}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:36.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:36.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10959", 'reported_seq': 11084, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:32.625350+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:53:32.625350+0000', 'last_peered': '2024-09-18T11:53:32.625350+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:53:32.625350+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17454326, 'num_objects': 2131, 'num_object_clones': 0, 'num_object_copies': 8524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10957, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 8}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:36.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:36.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10638", 'reported_seq': 10728, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:33.957525+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:33.957525+0000', 'last_peered': '2024-09-18T11:53:33.957525+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:33.957525+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16946422, 'num_objects': 2069, 'num_object_clones': 0, 'num_object_copies': 8276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10636, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 67}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:36.913 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:53:37.530 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:37.530 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26259/133260 objects degraded (19.705%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":615,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":33315,"data_bytes":272868748,"bytes_used":2687606784,"bytes_avail":383859449856,"bytes_total":386547056640,"degraded_objects":26259,"degraded_total":133260,"degraded_ratio":0.19705087798289062,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":201},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:53:37.542 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:53:37.543 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:38.052 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:38.052 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:38.080 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-09-18T11:53:38.046217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":272868748,"num_objects":33315,"num_object_clones":0,"num_object_copies":133260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173772,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2624616,"kb_used_data":535360,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374862744,"statfs":{"total":386547056640,"available":383859449856,"internally_reserved":0,"allocated":548208640,"data_stored":547473356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,11,12,11,14,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16662528,"num_objects":-2034,"num_object_clones":0,"num_object_copies":-8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2034,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002842"},"pg_stats":[{"pgid":"1.f","version":"37'11111","reported_seq":11212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.260475+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:33.260475+0000","last_peered":"2024-09-18T11:53:33.260475+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:33.260475+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17522688,"num_objects":2139,"num_object_clones":0,"num_object_copies":8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":36}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10968","reported_seq":15260,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:34.859893+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:34.859893+0000","last_peered":"2024-09-18T11:53:34.859893+0000","last_clean":"2024-09-18T11:53:34.859893+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:34.859893+0000","last_undegraded":"2024-09-18T11:53:34.859893+0000","last_fullsized":"2024-09-18T11:53:34.859893+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17102070,"num_objects":2088,"num_object_clones":0,"num_object_copies":8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10966,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10733","reported_seq":10850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:32.864780+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:32.864780+0000","last_peered":"2024-09-18T11:53:32.864780+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:32.864780+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16997011,"num_objects":2076,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10732,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":36}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10880","reported_seq":10974,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:34.041046+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:34.041046+0000","last_peered":"2024-09-18T11:53:34.041046+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:34.041046+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16758006,"num_objects":2046,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-55}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10799","reported_seq":10920,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.012858+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:33.012858+0000","last_peered":"2024-09-18T11:53:33.012858+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:33.012858+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167606,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10797,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":44}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10738","reported_seq":10857,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.221018+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:33.221018+0000","last_peered":"2024-09-18T11:53:33.221018+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:33.221018+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10738,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-71}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10943","reported_seq":11050,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:34.014253+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:34.014253+0000","last_peered":"2024-09-18T11:53:34.014253+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:34.014253+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16979190,"num_objects":2073,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-43}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10766","reported_seq":11992,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:34.140105+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:34.140105+0000","last_peered":"2024-09-18T11:53:34.140105+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:34.140105+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17274102,"num_objects":2109,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10764,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11079","reported_seq":15431,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:33.246221+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:33.246221+0000","last_peered":"2024-09-18T11:53:33.246221+0000","last_clean":"2024-09-18T11:53:33.246221+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:33.246221+0000","last_undegraded":"2024-09-18T11:53:33.246221+0000","last_fullsized":"2024-09-18T11:53:33.246221+0000","mapping_epoch":36,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17172972,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11076,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10571","reported_seq":10685,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.163892+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:33.163892+0000","last_peered":"2024-09-18T11:53:33.163892+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:33.163892+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16569590,"num_objects":2023,"num_object_clones":0,"num_object_copies":8092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10569,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10908","reported_seq":11029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.198996+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:33.198996+0000","last_peered":"2024-09-18T11:53:33.198996+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:33.198996+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17403761,"num_objects":2125,"num_object_clones":0,"num_object_copies":8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10990","reported_seq":11083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.253083+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:33.253083+0000","last_peered":"2024-09-18T11:53:33.253083+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:33.253083+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16718459,"num_objects":2041,"num_object_clones":0,"num_object_copies":8164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10989,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-43}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10904","reported_seq":11002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.336993+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:33.336993+0000","last_peered":"2024-09-18T11:53:33.336993+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:33.336993+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16898683,"num_objects":2063,"num_object_clones":0,"num_object_copies":8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'10959","reported_seq":11084,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:32.625350+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:32.625350+0000","last_peered":"2024-09-18T11:53:32.625350+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:32.625350+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17454326,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10957,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10812","reported_seq":15101,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:34.932875+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:34.932875+0000","last_peered":"2024-09-18T11:53:34.932875+0000","last_clean":"2024-09-18T11:53:34.932875+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:34.932875+0000","last_undegraded":"2024-09-18T11:53:34.932875+0000","last_fullsized":"2024-09-18T11:53:34.932875+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17126646,"num_objects":2091,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10638","reported_seq":10728,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:33.957525+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:33.957525+0000","last_peered":"2024-09-18T11:53:33.957525+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:33.957525+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16946422,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10636,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":67}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":272868748,"num_objects":33315,"num_object_clones":0,"num_object_copies":133260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173772,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7197,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":545103872,"data_stored":545103872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733300,"kb_used_data":134572,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93638540,"statfs":{"total":96636764160,"available":95885864960,"internally_reserved":0,"allocated":137801728,"data_stored":137617907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":11.34},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.56299999999999994}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":16.898}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.214},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.314}]}]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724080,"kb_used_data":134184,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93647760,"statfs":{"total":96636764160,"available":95895306240,"internally_reserved":0,"allocated":137404416,"data_stored":137220595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":12.869999999999999},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":12.903}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":4.4480000000000004},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":6.2999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443296,"kb_used_data":133008,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93928544,"statfs":{"total":96636764160,"available":96182829056,"internally_reserved":0,"allocated":136200192,"data_stored":136016371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,3,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":11},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.7959999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":5.7149999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":9.9789999999999992}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":9.1609999999999996},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":10.050000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723940,"kb_used_data":133596,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93647900,"statfs":{"total":96636764160,"available":95895449600,"internally_reserved":0,"allocated":136802304,"data_stored":136618483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":1.1379999999999999},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.1080000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.501},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":1.0780000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135999488,"data_stored":135999488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135716864,"data_stored":135716864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136667136,"data_stored":136667136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":136720384,"data_stored":136720384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:41.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:41.700 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:41.700 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:41.728 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-18T11:53:40.046663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":265676172,"num_objects":32437,"num_object_clones":0,"num_object_copies":129748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174650,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7260,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160677,"ondisk_log_size":160677,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2608460,"kb_used_data":523352,"kb_used_omap":14,"kb_used_meta":2085041,"kb_avail":374878900,"statfs":{"total":386547056640,"available":383875993600,"internally_reserved":0,"allocated":535912448,"data_stored":535177164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135082011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,5,10,10,24,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23855104,"num_objects":-2912,"num_object_clones":0,"num_object_copies":-11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2912,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002823"},"pg_stats":[{"pgid":"1.f","version":"37'11168","reported_seq":11268,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.189085+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:38.189085+0000","last_peered":"2024-09-18T11:53:38.189085+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:38.189085+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17055744,"num_objects":2082,"num_object_clones":0,"num_object_copies":8328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11168,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-21}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10968","reported_seq":15260,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:34.859893+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:34.859893+0000","last_peered":"2024-09-18T11:53:34.859893+0000","last_clean":"2024-09-18T11:53:34.859893+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:34.859893+0000","last_undegraded":"2024-09-18T11:53:34.859893+0000","last_fullsized":"2024-09-18T11:53:34.859893+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17102070,"num_objects":2088,"num_object_clones":0,"num_object_copies":8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10966,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10802","reported_seq":10918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.199119+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:38.199119+0000","last_peered":"2024-09-18T11:53:38.199119+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:38.199119+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16431763,"num_objects":2007,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10801,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-33}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10927","reported_seq":11020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.085246+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:39.085246+0000","last_peered":"2024-09-18T11:53:39.085246+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:39.085246+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16372982,"num_objects":1999,"num_object_clones":0,"num_object_copies":7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10925,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-102}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10852","reported_seq":10972,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.261254+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:38.261254+0000","last_peered":"2024-09-18T11:53:38.261254+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:38.261254+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16733430,"num_objects":2043,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10850,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10810","reported_seq":10929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.107802+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:38.107802+0000","last_peered":"2024-09-18T11:53:38.107802+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:38.107802+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":1976,"num_object_clones":0,"num_object_copies":7904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11019","reported_seq":11125,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.085180+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:39.085180+0000","last_peered":"2024-09-18T11:53:39.085180+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:39.085180+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16356598,"num_objects":1997,"num_object_clones":0,"num_object_copies":7988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-119}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10822","reported_seq":12173,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.214104+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:39.214104+0000","last_peered":"2024-09-18T11:53:39.214104+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:39.214104+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16815350,"num_objects":2053,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10820,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":643,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11149","reported_seq":15501,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:38.308217+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:38.308217+0000","last_peered":"2024-09-18T11:53:38.308217+0000","last_clean":"2024-09-18T11:53:38.308217+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:38.308217+0000","last_undegraded":"2024-09-18T11:53:38.308217+0000","last_fullsized":"2024-09-18T11:53:38.308217+0000","mapping_epoch":36,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16599532,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11146,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10626","reported_seq":10740,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.398892+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:38.398892+0000","last_peered":"2024-09-18T11:53:38.398892+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:38.398892+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16119030,"num_objects":1968,"num_object_clones":0,"num_object_copies":7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10624,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10973","reported_seq":11094,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.235525+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:38.235525+0000","last_peered":"2024-09-18T11:53:38.235525+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:38.235525+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16871281,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10970,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11059","reported_seq":11152,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.208492+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:38.208492+0000","last_peered":"2024-09-18T11:53:38.208492+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:38.208492+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16153211,"num_objects":1972,"num_object_clones":0,"num_object_copies":7888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11058,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-112}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10983","reported_seq":11079,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.340589+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:38.340589+0000","last_peered":"2024-09-18T11:53:38.340589+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:38.340589+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16251515,"num_objects":1984,"num_object_clones":0,"num_object_copies":7936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-62}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11013","reported_seq":11138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.007629+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:38.007629+0000","last_peered":"2024-09-18T11:53:38.007629+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:38.007629+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17011958,"num_objects":2077,"num_object_clones":0,"num_object_copies":8308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-46}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10812","reported_seq":15101,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:34.932875+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:34.932875+0000","last_peered":"2024-09-18T11:53:34.932875+0000","last_clean":"2024-09-18T11:53:34.932875+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:34.932875+0000","last_undegraded":"2024-09-18T11:53:34.932875+0000","last_fullsized":"2024-09-18T11:53:34.932875+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17126646,"num_objects":2091,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10694","reported_seq":10784,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.067442+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:39.067442+0000","last_peered":"2024-09-18T11:53:39.067442+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:39.067442+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16487670,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":265676172,"num_objects":32437,"num_object_clones":0,"num_object_copies":129748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174650,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7260,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532869120,"data_stored":532869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160677,"ondisk_log_size":160677,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716760,"kb_used_data":130512,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93655080,"statfs":{"total":96636764160,"available":95902801920,"internally_reserved":0,"allocated":133644288,"data_stored":133460467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,0,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":10.468999999999999},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.438000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":10.391},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.1000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.4400000000000004},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.0740000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724280,"kb_used_data":130224,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93647560,"statfs":{"total":96636764160,"available":95895101440,"internally_reserved":0,"allocated":133349376,"data_stored":133165555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":6.1749999999999998},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":6.2030000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":5.8390000000000004},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":11.071}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.625},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":5.4029999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443296,"kb_used_data":133008,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93928544,"statfs":{"total":96636764160,"available":96182829056,"internally_reserved":0,"allocated":136200192,"data_stored":136016371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,3,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":11},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.7959999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":5.7149999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":9.9789999999999992}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":9.1609999999999996},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":10.050000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724124,"kb_used_data":129608,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93647716,"statfs":{"total":96636764160,"available":95895261184,"internally_reserved":0,"allocated":132718592,"data_stored":132534771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":19.170999999999999},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":20.059000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":20.099}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":8.2119999999999997},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":20.134}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":131891200,"data_stored":131891200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135716864,"data_stored":135716864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":132579328,"data_stored":132579328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":132681728,"data_stored":132681728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:41.729 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:42.338 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:42.338 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:42.370 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-18T11:53:42.047145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264660364,"num_objects":32313,"num_object_clones":0,"num_object_copies":129252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174774,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7260,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2591888,"kb_used_data":519232,"kb_used_omap":14,"kb_used_meta":2072561,"kb_avail":374895472,"statfs":{"total":386547056640,"available":383892963328,"internally_reserved":0,"allocated":531693568,"data_stored":530958284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2122302491},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,6,12,21,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16449536,"num_objects":-2008,"num_object_clones":0,"num_object_copies":-8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002868"},"pg_stats":[{"pgid":"1.f","version":"37'11168","reported_seq":11268,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.189085+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:38.189085+0000","last_peered":"2024-09-18T11:53:38.189085+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:38.189085+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17055744,"num_objects":2082,"num_object_clones":0,"num_object_copies":8328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11168,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-21}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11025","reported_seq":15317,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:39.939676+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:39.939676+0000","last_peered":"2024-09-18T11:53:39.939676+0000","last_clean":"2024-09-18T11:53:39.939676+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:39.939676+0000","last_undegraded":"2024-09-18T11:53:39.939676+0000","last_fullsized":"2024-09-18T11:53:39.939676+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16635126,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10802","reported_seq":10918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.199119+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:38.199119+0000","last_peered":"2024-09-18T11:53:38.199119+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:38.199119+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16431763,"num_objects":2007,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10801,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-33}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10927","reported_seq":11020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.085246+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:39.085246+0000","last_peered":"2024-09-18T11:53:39.085246+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:39.085246+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16372982,"num_objects":1999,"num_object_clones":0,"num_object_copies":7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10925,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-102}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10852","reported_seq":10972,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.261254+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:38.261254+0000","last_peered":"2024-09-18T11:53:38.261254+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:38.261254+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16733430,"num_objects":2043,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10850,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10810","reported_seq":10929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.107802+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:38.107802+0000","last_peered":"2024-09-18T11:53:38.107802+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:38.107802+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":1976,"num_object_clones":0,"num_object_copies":7904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11019","reported_seq":11125,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.085180+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:39.085180+0000","last_peered":"2024-09-18T11:53:39.085180+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:39.085180+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16356598,"num_objects":1997,"num_object_clones":0,"num_object_copies":7988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-119}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10822","reported_seq":12173,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.214104+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:39.214104+0000","last_peered":"2024-09-18T11:53:39.214104+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:39.214104+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16815350,"num_objects":2053,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10820,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":643,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11149","reported_seq":15501,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:38.308217+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:38.308217+0000","last_peered":"2024-09-18T11:53:38.308217+0000","last_clean":"2024-09-18T11:53:38.308217+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:38.308217+0000","last_undegraded":"2024-09-18T11:53:38.308217+0000","last_fullsized":"2024-09-18T11:53:38.308217+0000","mapping_epoch":36,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16599532,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11146,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10626","reported_seq":10740,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.398892+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:38.398892+0000","last_peered":"2024-09-18T11:53:38.398892+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:38.398892+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16119030,"num_objects":1968,"num_object_clones":0,"num_object_copies":7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10624,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10973","reported_seq":11094,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.235525+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:38.235525+0000","last_peered":"2024-09-18T11:53:38.235525+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:38.235525+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16871281,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10970,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11059","reported_seq":11152,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.208492+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:38.208492+0000","last_peered":"2024-09-18T11:53:38.208492+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:38.208492+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16153211,"num_objects":1972,"num_object_clones":0,"num_object_copies":7888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11058,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-112}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10983","reported_seq":11079,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.340589+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:38.340589+0000","last_peered":"2024-09-18T11:53:38.340589+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:38.340589+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16251515,"num_objects":1984,"num_object_clones":0,"num_object_copies":7936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-62}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11013","reported_seq":11138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.007629+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:38.007629+0000","last_peered":"2024-09-18T11:53:38.007629+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:38.007629+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17011958,"num_objects":2077,"num_object_clones":0,"num_object_copies":8308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-46}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10879","reported_seq":15168,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:39.632143+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:39.632143+0000","last_peered":"2024-09-18T11:53:39.632143+0000","last_clean":"2024-09-18T11:53:39.632143+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:39.632143+0000","last_undegraded":"2024-09-18T11:53:39.632143+0000","last_fullsized":"2024-09-18T11:53:39.632143+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16577782,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10877,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10694","reported_seq":10784,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.067442+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:39.067442+0000","last_peered":"2024-09-18T11:53:39.067442+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:39.067442+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16487670,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":264660364,"num_objects":32313,"num_object_clones":0,"num_object_copies":129252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174774,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7260,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":528723968,"data_stored":528723968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716760,"kb_used_data":130512,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93655080,"statfs":{"total":96636764160,"available":95902801920,"internally_reserved":0,"allocated":133644288,"data_stored":133460467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,0,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":10.468999999999999},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.438000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":10.391},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.1000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.4400000000000004},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.0740000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724280,"kb_used_data":130224,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93647560,"statfs":{"total":96636764160,"available":95895101440,"internally_reserved":0,"allocated":133349376,"data_stored":133165555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":6.1749999999999998},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":6.2030000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":5.8390000000000004},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":11.071}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.625},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":5.4029999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426724,"kb_used_data":128888,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93945116,"statfs":{"total":96636764160,"available":96199798784,"internally_reserved":0,"allocated":131981312,"data_stored":131797491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.247},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.399}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.5030000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.1829999999999998},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":3.5910000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724124,"kb_used_data":129608,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93647716,"statfs":{"total":96636764160,"available":95895261184,"internally_reserved":0,"allocated":132718592,"data_stored":132534771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":19.170999999999999},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":20.059000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":20.099}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":8.2119999999999997},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":20.134}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":131891200,"data_stored":131891200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131571712,"data_stored":131571712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":132579328,"data_stored":132579328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":132681728,"data_stored":132681728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:42.371 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:42.371 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11168", 'reported_seq': 11268, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:38.189085+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:53:38.189085+0000', 'last_peered': '2024-09-18T11:53:38.189085+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:53:38.189085+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17055744, 'num_objects': 2082, 'num_object_clones': 0, 'num_object_copies': 8328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2082, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11168, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -21}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:42.371 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:42.372 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10802", 'reported_seq': 10918, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:38.199119+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:53:38.199119+0000', 'last_peered': '2024-09-18T11:53:38.199119+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:53:38.199119+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16431763, 'num_objects': 2007, 'num_object_clones': 0, 'num_object_copies': 8028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2007, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10801, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -33}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:42.372 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:42.372 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10927", 'reported_seq': 11020, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:39.085246+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:39.085246+0000', 'last_peered': '2024-09-18T11:53:39.085246+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:39.085246+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16372982, 'num_objects': 1999, 'num_object_clones': 0, 'num_object_copies': 7996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10925, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -102}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:42.372 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:42.372 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10852", 'reported_seq': 10972, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:38.261254+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:53:38.261254+0000', 'last_peered': '2024-09-18T11:53:38.261254+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:53:38.261254+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16733430, 'num_objects': 2043, 'num_object_clones': 0, 'num_object_copies': 8172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10850, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -9}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:42.372 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:42.372 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10810", 'reported_seq': 10929, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:38.107802+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:53:38.107802+0000', 'last_peered': '2024-09-18T11:53:38.107802+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:53:38.107802+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16187392, 'num_objects': 1976, 'num_object_clones': 0, 'num_object_copies': 7904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10810, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:42.372 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:42.372 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11019", 'reported_seq': 11125, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:39.085180+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:39.085180+0000', 'last_peered': '2024-09-18T11:53:39.085180+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:39.085180+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16356598, 'num_objects': 1997, 'num_object_clones': 0, 'num_object_copies': 7988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11017, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -119}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:42.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:42.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10822", 'reported_seq': 12173, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:39.214104+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:39.214104+0000', 'last_peered': '2024-09-18T11:53:39.214104+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:39.214104+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16815350, 'num_objects': 2053, 'num_object_clones': 0, 'num_object_copies': 8212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1395, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10820, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 643, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 658}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:42.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:42.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10626", 'reported_seq': 10740, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:38.398892+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:53:38.398892+0000', 'last_peered': '2024-09-18T11:53:38.398892+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:53:38.398892+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16119030, 'num_objects': 1968, 'num_object_clones': 0, 'num_object_copies': 7872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10624, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 5}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:42.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:42.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10973", 'reported_seq': 11094, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:38.235525+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:53:38.235525+0000', 'last_peered': '2024-09-18T11:53:38.235525+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:53:38.235525+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16871281, 'num_objects': 2060, 'num_object_clones': 0, 'num_object_copies': 8240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10970, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 26}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:42.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:42.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11059", 'reported_seq': 11152, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:38.208492+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:53:38.208492+0000', 'last_peered': '2024-09-18T11:53:38.208492+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:53:38.208492+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16153211, 'num_objects': 1972, 'num_object_clones': 0, 'num_object_copies': 7888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11058, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -112}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:42.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:42.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'10983", 'reported_seq': 11079, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:38.340589+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:53:38.340589+0000', 'last_peered': '2024-09-18T11:53:38.340589+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:53:38.340589+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16251515, 'num_objects': 1984, 'num_object_clones': 0, 'num_object_copies': 7936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10982, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -62}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:42.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:42.374 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11013", 'reported_seq': 11138, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:38.007629+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:53:38.007629+0000', 'last_peered': '2024-09-18T11:53:38.007629+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:53:38.007629+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17011958, 'num_objects': 2077, 'num_object_clones': 0, 'num_object_copies': 8308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11011, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -46}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:42.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:42.374 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10694", 'reported_seq': 10784, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:39.067442+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:39.067442+0000', 'last_peered': '2024-09-18T11:53:39.067442+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:39.067442+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16487670, 'num_objects': 2013, 'num_object_clones': 0, 'num_object_copies': 8052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10692, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 11}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:42.374 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:53:43.059 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:43.059 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26196/129252 objects degraded (20.267%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":621,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32313,"data_bytes":264660364,"bytes_used":2654093312,"bytes_avail":383892963328,"bytes_total":386547056640,"degraded_objects":26196,"degraded_total":129252,"degraded_ratio":0.2026738464395135,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":167},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:53:43.086 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:53:43.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:43.699 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:43.699 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:43.719 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-18T11:53:42.047145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264660364,"num_objects":32313,"num_object_clones":0,"num_object_copies":129252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174774,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7260,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2591888,"kb_used_data":519232,"kb_used_omap":14,"kb_used_meta":2072561,"kb_avail":374895472,"statfs":{"total":386547056640,"available":383892963328,"internally_reserved":0,"allocated":531693568,"data_stored":530958284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2122302491},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,6,12,21,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16449536,"num_objects":-2008,"num_object_clones":0,"num_object_copies":-8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002868"},"pg_stats":[{"pgid":"1.f","version":"37'11168","reported_seq":11268,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.189085+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:38.189085+0000","last_peered":"2024-09-18T11:53:38.189085+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:38.189085+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17055744,"num_objects":2082,"num_object_clones":0,"num_object_copies":8328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11168,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-21}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11025","reported_seq":15317,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:39.939676+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:39.939676+0000","last_peered":"2024-09-18T11:53:39.939676+0000","last_clean":"2024-09-18T11:53:39.939676+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:39.939676+0000","last_undegraded":"2024-09-18T11:53:39.939676+0000","last_fullsized":"2024-09-18T11:53:39.939676+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16635126,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10802","reported_seq":10918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.199119+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:38.199119+0000","last_peered":"2024-09-18T11:53:38.199119+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:38.199119+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16431763,"num_objects":2007,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10801,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-33}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10927","reported_seq":11020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.085246+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:39.085246+0000","last_peered":"2024-09-18T11:53:39.085246+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:39.085246+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16372982,"num_objects":1999,"num_object_clones":0,"num_object_copies":7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10925,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-102}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10852","reported_seq":10972,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.261254+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:38.261254+0000","last_peered":"2024-09-18T11:53:38.261254+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:38.261254+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16733430,"num_objects":2043,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10850,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10810","reported_seq":10929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.107802+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:38.107802+0000","last_peered":"2024-09-18T11:53:38.107802+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:38.107802+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":1976,"num_object_clones":0,"num_object_copies":7904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11019","reported_seq":11125,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.085180+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:39.085180+0000","last_peered":"2024-09-18T11:53:39.085180+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:39.085180+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16356598,"num_objects":1997,"num_object_clones":0,"num_object_copies":7988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-119}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10822","reported_seq":12173,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.214104+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:39.214104+0000","last_peered":"2024-09-18T11:53:39.214104+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:39.214104+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16815350,"num_objects":2053,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10820,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":643,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11149","reported_seq":15501,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:38.308217+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:38.308217+0000","last_peered":"2024-09-18T11:53:38.308217+0000","last_clean":"2024-09-18T11:53:38.308217+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:38.308217+0000","last_undegraded":"2024-09-18T11:53:38.308217+0000","last_fullsized":"2024-09-18T11:53:38.308217+0000","mapping_epoch":36,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16599532,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11146,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10626","reported_seq":10740,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.398892+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:38.398892+0000","last_peered":"2024-09-18T11:53:38.398892+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:38.398892+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16119030,"num_objects":1968,"num_object_clones":0,"num_object_copies":7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10624,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10973","reported_seq":11094,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.235525+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:38.235525+0000","last_peered":"2024-09-18T11:53:38.235525+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:38.235525+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16871281,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10970,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11059","reported_seq":11152,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.208492+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:38.208492+0000","last_peered":"2024-09-18T11:53:38.208492+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:38.208492+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16153211,"num_objects":1972,"num_object_clones":0,"num_object_copies":7888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11058,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-112}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10983","reported_seq":11079,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.340589+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:38.340589+0000","last_peered":"2024-09-18T11:53:38.340589+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:38.340589+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16251515,"num_objects":1984,"num_object_clones":0,"num_object_copies":7936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-62}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11013","reported_seq":11138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:38.007629+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:38.007629+0000","last_peered":"2024-09-18T11:53:38.007629+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:38.007629+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17011958,"num_objects":2077,"num_object_clones":0,"num_object_copies":8308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-46}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10879","reported_seq":15168,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:39.632143+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:39.632143+0000","last_peered":"2024-09-18T11:53:39.632143+0000","last_clean":"2024-09-18T11:53:39.632143+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:39.632143+0000","last_undegraded":"2024-09-18T11:53:39.632143+0000","last_fullsized":"2024-09-18T11:53:39.632143+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16577782,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10877,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10694","reported_seq":10784,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:39.067442+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:39.067442+0000","last_peered":"2024-09-18T11:53:39.067442+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:39.067442+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16487670,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":264660364,"num_objects":32313,"num_object_clones":0,"num_object_copies":129252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174774,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7260,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":528723968,"data_stored":528723968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716760,"kb_used_data":130512,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93655080,"statfs":{"total":96636764160,"available":95902801920,"internally_reserved":0,"allocated":133644288,"data_stored":133460467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,0,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":10.468999999999999},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.438000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":10.391},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.1000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.4400000000000004},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.0740000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724280,"kb_used_data":130224,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93647560,"statfs":{"total":96636764160,"available":95895101440,"internally_reserved":0,"allocated":133349376,"data_stored":133165555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":6.1749999999999998},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":6.2030000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":5.8390000000000004},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":11.071}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":5.625},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":5.4029999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426724,"kb_used_data":128888,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93945116,"statfs":{"total":96636764160,"available":96199798784,"internally_reserved":0,"allocated":131981312,"data_stored":131797491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.247},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.399}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.5030000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.1829999999999998},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":3.5910000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724124,"kb_used_data":129608,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93647716,"statfs":{"total":96636764160,"available":95895261184,"internally_reserved":0,"allocated":132718592,"data_stored":132534771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":19.170999999999999},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":20.059000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":20.099}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":8.2119999999999997},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":20.134}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":131891200,"data_stored":131891200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131571712,"data_stored":131571712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":132579328,"data_stored":132579328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":132681728,"data_stored":132681728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:46.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:47.300 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:47.300 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:47.330 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-18T11:53:46.048098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":256304524,"num_objects":31293,"num_object_clones":0,"num_object_copies":125172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175794,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7307,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2592312,"kb_used_data":503064,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374895048,"statfs":{"total":386547056640,"available":383892529152,"internally_reserved":0,"allocated":515137536,"data_stored":514402252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,11,14,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19685376,"num_objects":-2403,"num_object_clones":0,"num_object_copies":-9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2403,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002827"},"pg_stats":[{"pgid":"1.f","version":"37'11235","reported_seq":11336,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.311949+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:43.311949+0000","last_peered":"2024-09-18T11:53:43.311949+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:43.311949+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16506880,"num_objects":2015,"num_object_clones":0,"num_object_copies":8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-88}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11091","reported_seq":15383,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.819949+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:44.819949+0000","last_peered":"2024-09-18T11:53:44.819949+0000","last_clean":"2024-09-18T11:53:44.819949+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:44.819949+0000","last_undegraded":"2024-09-18T11:53:44.819949+0000","last_fullsized":"2024-09-18T11:53:44.819949+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16094454,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11089,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10864","reported_seq":10981,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.378664+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:43.378664+0000","last_peered":"2024-09-18T11:53:43.378664+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:43.378664+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15923859,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10863,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-95}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10978","reported_seq":11071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.176455+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:44.176455+0000","last_peered":"2024-09-18T11:53:44.176455+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:44.176455+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15955190,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10976,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10921","reported_seq":11041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.232838+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:43.232838+0000","last_peered":"2024-09-18T11:53:43.232838+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:43.232838+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16168182,"num_objects":1974,"num_object_clones":0,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10919,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-78}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10883","reported_seq":11002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.285792+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:43.285792+0000","last_peered":"2024-09-18T11:53:43.285792+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:43.285792+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15589376,"num_objects":1903,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10883,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11084","reported_seq":11191,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.155370+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:44.155370+0000","last_peered":"2024-09-18T11:53:44.155370+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:44.155370+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15824118,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11082,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-184}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10884","reported_seq":12330,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.234519+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:44.234519+0000","last_peered":"2024-09-18T11:53:44.234519+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:44.234519+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16307446,"num_objects":1991,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10882,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11216","reported_seq":15569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:43.332798+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:43.332798+0000","last_peered":"2024-09-18T11:53:43.332798+0000","last_clean":"2024-09-18T11:53:43.332798+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:43.332798+0000","last_undegraded":"2024-09-18T11:53:43.332798+0000","last_fullsized":"2024-09-18T11:53:43.332798+0000","mapping_epoch":36,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16050668,"num_objects":1960,"num_object_clones":0,"num_object_copies":7840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10686","reported_seq":10799,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.491837+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:43.491837+0000","last_peered":"2024-09-18T11:53:43.491837+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:43.491837+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15627510,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10684,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-55}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11038","reported_seq":11159,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.270292+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:43.270292+0000","last_peered":"2024-09-18T11:53:43.270292+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:43.270292+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16338801,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11035,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-39}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11123","reported_seq":11216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.371622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:43.371622+0000","last_peered":"2024-09-18T11:53:43.371622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:43.371622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15628923,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11122,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-176}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11052","reported_seq":11150,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.479134+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:43.479134+0000","last_peered":"2024-09-18T11:53:43.479134+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:43.479134+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15686267,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11051,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11071","reported_seq":11196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.154547+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:43.154547+0000","last_peered":"2024-09-18T11:53:43.154547+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:43.154547+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16536822,"num_objects":2019,"num_object_clones":0,"num_object_copies":8076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11069,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10938","reported_seq":15226,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.013243+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:45.013243+0000","last_peered":"2024-09-18T11:53:45.013243+0000","last_clean":"2024-09-18T11:53:45.013243+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:45.013243+0000","last_undegraded":"2024-09-18T11:53:45.013243+0000","last_fullsized":"2024-09-18T11:53:45.013243+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16094454,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10936,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10757","reported_seq":10847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.035938+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:44.035938+0000","last_peered":"2024-09-18T11:53:44.035938+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:44.035938+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15971574,"num_objects":1950,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10755,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-52}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":256304524,"num_objects":31293,"num_object_clones":0,"num_object_copies":125172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175794,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7307,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512057344,"data_stored":512057344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716888,"kb_used_data":126480,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93654952,"statfs":{"total":96636764160,"available":95902670848,"internally_reserved":0,"allocated":129515520,"data_stored":129331699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":11.368},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":11.401}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":33.281999999999996},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.891}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.452},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.657}]}]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724412,"kb_used_data":126196,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93647428,"statfs":{"total":96636764160,"available":95894966272,"internally_reserved":0,"allocated":129224704,"data_stored":129040883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":8.4920000000000009},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":8.5229999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":10.007999999999999},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":11.377000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":1.353},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":10.917999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426840,"kb_used_data":124880,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93945000,"statfs":{"total":96636764160,"available":96199680000,"internally_reserved":0,"allocated":127877120,"data_stored":127693299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.247},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.399}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.5030000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.1829999999999998},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":3.5910000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724172,"kb_used_data":125508,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93647668,"statfs":{"total":96636764160,"available":95895212032,"internally_reserved":0,"allocated":128520192,"data_stored":128336371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.7490000000000001},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.9700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":2.0449999999999999},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":20.018000000000001},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":13.198}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":127778816,"data_stored":127778816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127422464,"data_stored":127422464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128380928,"data_stored":128380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128475136,"data_stored":128475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:47.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:47.850 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:48.238 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:48.240 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-18T11:53:46.048098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":256304524,"num_objects":31293,"num_object_clones":0,"num_object_copies":125172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175794,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7307,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2592312,"kb_used_data":503064,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374895048,"statfs":{"total":386547056640,"available":383892529152,"internally_reserved":0,"allocated":515137536,"data_stored":514402252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,11,14,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19685376,"num_objects":-2403,"num_object_clones":0,"num_object_copies":-9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2403,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002827"},"pg_stats":[{"pgid":"1.f","version":"37'11235","reported_seq":11336,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.311949+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:43.311949+0000","last_peered":"2024-09-18T11:53:43.311949+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:43.311949+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16506880,"num_objects":2015,"num_object_clones":0,"num_object_copies":8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-88}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11091","reported_seq":15383,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.819949+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:44.819949+0000","last_peered":"2024-09-18T11:53:44.819949+0000","last_clean":"2024-09-18T11:53:44.819949+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:44.819949+0000","last_undegraded":"2024-09-18T11:53:44.819949+0000","last_fullsized":"2024-09-18T11:53:44.819949+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16094454,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11089,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10864","reported_seq":10981,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.378664+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:43.378664+0000","last_peered":"2024-09-18T11:53:43.378664+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:43.378664+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15923859,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10863,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-95}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10978","reported_seq":11071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.176455+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:44.176455+0000","last_peered":"2024-09-18T11:53:44.176455+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:44.176455+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15955190,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10976,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10921","reported_seq":11041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.232838+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:43.232838+0000","last_peered":"2024-09-18T11:53:43.232838+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:43.232838+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16168182,"num_objects":1974,"num_object_clones":0,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10919,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-78}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10883","reported_seq":11002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.285792+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:43.285792+0000","last_peered":"2024-09-18T11:53:43.285792+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:43.285792+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15589376,"num_objects":1903,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10883,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11084","reported_seq":11191,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.155370+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:44.155370+0000","last_peered":"2024-09-18T11:53:44.155370+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:44.155370+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15824118,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11082,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-184}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10884","reported_seq":12330,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.234519+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:44.234519+0000","last_peered":"2024-09-18T11:53:44.234519+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:44.234519+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16307446,"num_objects":1991,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10882,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11216","reported_seq":15569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:43.332798+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:43.332798+0000","last_peered":"2024-09-18T11:53:43.332798+0000","last_clean":"2024-09-18T11:53:43.332798+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:43.332798+0000","last_undegraded":"2024-09-18T11:53:43.332798+0000","last_fullsized":"2024-09-18T11:53:43.332798+0000","mapping_epoch":36,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16050668,"num_objects":1960,"num_object_clones":0,"num_object_copies":7840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10686","reported_seq":10799,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.491837+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:43.491837+0000","last_peered":"2024-09-18T11:53:43.491837+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:43.491837+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15627510,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10684,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-55}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11038","reported_seq":11159,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.270292+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:43.270292+0000","last_peered":"2024-09-18T11:53:43.270292+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:43.270292+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16338801,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11035,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-39}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11123","reported_seq":11216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.371622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:43.371622+0000","last_peered":"2024-09-18T11:53:43.371622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:43.371622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15628923,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11122,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-176}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11052","reported_seq":11150,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.479134+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:43.479134+0000","last_peered":"2024-09-18T11:53:43.479134+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:43.479134+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15686267,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11051,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11071","reported_seq":11196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.154547+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:43.154547+0000","last_peered":"2024-09-18T11:53:43.154547+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:43.154547+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16536822,"num_objects":2019,"num_object_clones":0,"num_object_copies":8076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11069,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10938","reported_seq":15226,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.013243+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:45.013243+0000","last_peered":"2024-09-18T11:53:45.013243+0000","last_clean":"2024-09-18T11:53:45.013243+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:45.013243+0000","last_undegraded":"2024-09-18T11:53:45.013243+0000","last_fullsized":"2024-09-18T11:53:45.013243+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16094454,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10936,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10757","reported_seq":10847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.035938+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:44.035938+0000","last_peered":"2024-09-18T11:53:44.035938+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:44.035938+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15971574,"num_objects":1950,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10755,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-52}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":256304524,"num_objects":31293,"num_object_clones":0,"num_object_copies":125172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175794,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7307,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512057344,"data_stored":512057344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716888,"kb_used_data":126480,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93654952,"statfs":{"total":96636764160,"available":95902670848,"internally_reserved":0,"allocated":129515520,"data_stored":129331699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":11.368},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":11.401}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":33.281999999999996},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.891}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.452},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.657}]}]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724412,"kb_used_data":126196,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93647428,"statfs":{"total":96636764160,"available":95894966272,"internally_reserved":0,"allocated":129224704,"data_stored":129040883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":8.4920000000000009},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":8.5229999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":10.007999999999999},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":11.377000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":1.353},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":10.917999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426840,"kb_used_data":124880,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93945000,"statfs":{"total":96636764160,"available":96199680000,"internally_reserved":0,"allocated":127877120,"data_stored":127693299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.247},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.399}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.5030000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.1829999999999998},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":3.5910000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724172,"kb_used_data":125508,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93647668,"statfs":{"total":96636764160,"available":95895212032,"internally_reserved":0,"allocated":128520192,"data_stored":128336371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.7490000000000001},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.9700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":2.0449999999999999},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":20.018000000000001},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":13.198}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":127778816,"data_stored":127778816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127422464,"data_stored":127422464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128380928,"data_stored":128380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128475136,"data_stored":128475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:48.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:48.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11235", 'reported_seq': 11336, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:43.311949+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:53:43.311949+0000', 'last_peered': '2024-09-18T11:53:43.311949+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:53:43.311949+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16506880, 'num_objects': 2015, 'num_object_clones': 0, 'num_object_copies': 8060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11235, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -88}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:48.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:48.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10864", 'reported_seq': 10981, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:43.378664+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:53:43.378664+0000', 'last_peered': '2024-09-18T11:53:43.378664+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:53:43.378664+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15923859, 'num_objects': 1945, 'num_object_clones': 0, 'num_object_copies': 7780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1945, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10863, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -95}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:48.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:48.242 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'10978", 'reported_seq': 11071, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:44.176455+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:44.176455+0000', 'last_peered': '2024-09-18T11:53:44.176455+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:44.176455+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15955190, 'num_objects': 1948, 'num_object_clones': 0, 'num_object_copies': 7792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10976, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -153}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:48.242 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:48.242 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10921", 'reported_seq': 11041, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:43.232838+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:53:43.232838+0000', 'last_peered': '2024-09-18T11:53:43.232838+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:53:43.232838+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16168182, 'num_objects': 1974, 'num_object_clones': 0, 'num_object_copies': 7896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10919, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -78}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:48.242 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:48.242 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10883", 'reported_seq': 11002, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:43.285792+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:53:43.285792+0000', 'last_peered': '2024-09-18T11:53:43.285792+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:53:43.285792+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15589376, 'num_objects': 1903, 'num_object_clones': 0, 'num_object_copies': 7612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10883, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -216}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:48.242 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:48.242 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11084", 'reported_seq': 11191, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:44.155370+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:44.155370+0000', 'last_peered': '2024-09-18T11:53:44.155370+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:44.155370+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15824118, 'num_objects': 1932, 'num_object_clones': 0, 'num_object_copies': 7728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11082, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -184}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:48.242 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:48.242 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10884", 'reported_seq': 12330, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:44.234519+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:44.234519+0000', 'last_peered': '2024-09-18T11:53:44.234519+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:44.234519+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16307446, 'num_objects': 1991, 'num_object_clones': 0, 'num_object_copies': 7964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1348, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10882, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 690, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 643}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:48.242 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:48.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10686", 'reported_seq': 10799, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:43.491837+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:53:43.491837+0000', 'last_peered': '2024-09-18T11:53:43.491837+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:53:43.491837+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15627510, 'num_objects': 1908, 'num_object_clones': 0, 'num_object_copies': 7632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10684, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -55}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:48.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:48.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11038", 'reported_seq': 11159, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:43.270292+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:53:43.270292+0000', 'last_peered': '2024-09-18T11:53:43.270292+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:53:43.270292+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16338801, 'num_objects': 1995, 'num_object_clones': 0, 'num_object_copies': 7980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11035, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -39}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:48.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:48.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11123", 'reported_seq': 11216, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:43.371622+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:53:43.371622+0000', 'last_peered': '2024-09-18T11:53:43.371622+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:53:43.371622+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15628923, 'num_objects': 1908, 'num_object_clones': 0, 'num_object_copies': 7632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11122, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:48.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:48.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11052", 'reported_seq': 11150, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:43.479134+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:53:43.479134+0000', 'last_peered': '2024-09-18T11:53:43.479134+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:53:43.479134+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15686267, 'num_objects': 1915, 'num_object_clones': 0, 'num_object_copies': 7660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11051, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -131}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:48.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:48.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11071", 'reported_seq': 11196, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:43.154547+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:53:43.154547+0000', 'last_peered': '2024-09-18T11:53:43.154547+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:53:43.154547+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16536822, 'num_objects': 2019, 'num_object_clones': 0, 'num_object_copies': 8076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11069, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:48.244 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:48.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10757", 'reported_seq': 10847, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:44.035938+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:44.035938+0000', 'last_peered': '2024-09-18T11:53:44.035938+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:44.035938+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15971574, 'num_objects': 1950, 'num_object_clones': 0, 'num_object_copies': 7800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10755, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -52}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:48.244 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:53:48.968 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:48.969 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26149/125172 objects degraded (20.890%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":627,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":31293,"data_bytes":256304524,"bytes_used":2654527488,"bytes_avail":383892529152,"bytes_total":386547056640,"degraded_objects":26149,"degraded_total":125172,"degraded_ratio":0.20890454734285624,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":168},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:53:49.001 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:53:49.001 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:49.541 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:49.542 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:49.568 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-18T11:53:48.048553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":256304524,"num_objects":31293,"num_object_clones":0,"num_object_copies":125172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175794,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7307,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2592312,"kb_used_data":503064,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374895048,"statfs":{"total":386547056640,"available":383892529152,"internally_reserved":0,"allocated":515137536,"data_stored":514402252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,11,14,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16564224,"num_objects":-2022,"num_object_clones":0,"num_object_copies":-8088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2022,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002804"},"pg_stats":[{"pgid":"1.f","version":"37'11235","reported_seq":11336,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.311949+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:43.311949+0000","last_peered":"2024-09-18T11:53:43.311949+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:43.311949+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16506880,"num_objects":2015,"num_object_clones":0,"num_object_copies":8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-88}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11091","reported_seq":15383,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.819949+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:44.819949+0000","last_peered":"2024-09-18T11:53:44.819949+0000","last_clean":"2024-09-18T11:53:44.819949+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:44.819949+0000","last_undegraded":"2024-09-18T11:53:44.819949+0000","last_fullsized":"2024-09-18T11:53:44.819949+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16094454,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11089,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10864","reported_seq":10981,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.378664+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:43.378664+0000","last_peered":"2024-09-18T11:53:43.378664+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:43.378664+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15923859,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10863,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-95}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10978","reported_seq":11071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.176455+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:44.176455+0000","last_peered":"2024-09-18T11:53:44.176455+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:44.176455+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15955190,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10976,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'10921","reported_seq":11041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.232838+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:43.232838+0000","last_peered":"2024-09-18T11:53:43.232838+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:43.232838+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16168182,"num_objects":1974,"num_object_clones":0,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10919,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-78}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10883","reported_seq":11002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.285792+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:43.285792+0000","last_peered":"2024-09-18T11:53:43.285792+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:43.285792+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15589376,"num_objects":1903,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10883,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11084","reported_seq":11191,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.155370+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:44.155370+0000","last_peered":"2024-09-18T11:53:44.155370+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:44.155370+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15824118,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11082,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-184}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10884","reported_seq":12330,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.234519+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:44.234519+0000","last_peered":"2024-09-18T11:53:44.234519+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:44.234519+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16307446,"num_objects":1991,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10882,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11216","reported_seq":15569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:43.332798+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:43.332798+0000","last_peered":"2024-09-18T11:53:43.332798+0000","last_clean":"2024-09-18T11:53:43.332798+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:43.332798+0000","last_undegraded":"2024-09-18T11:53:43.332798+0000","last_fullsized":"2024-09-18T11:53:43.332798+0000","mapping_epoch":36,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16050668,"num_objects":1960,"num_object_clones":0,"num_object_copies":7840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10686","reported_seq":10799,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.491837+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:43.491837+0000","last_peered":"2024-09-18T11:53:43.491837+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:43.491837+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15627510,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10684,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-55}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11038","reported_seq":11159,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.270292+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:43.270292+0000","last_peered":"2024-09-18T11:53:43.270292+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:43.270292+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16338801,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11035,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-39}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11123","reported_seq":11216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.371622+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:43.371622+0000","last_peered":"2024-09-18T11:53:43.371622+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:43.371622+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15628923,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11122,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-176}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11052","reported_seq":11150,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.479134+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:43.479134+0000","last_peered":"2024-09-18T11:53:43.479134+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:43.479134+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15686267,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11051,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11071","reported_seq":11196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:43.154547+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:43.154547+0000","last_peered":"2024-09-18T11:53:43.154547+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:43.154547+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16536822,"num_objects":2019,"num_object_clones":0,"num_object_copies":8076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11069,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10938","reported_seq":15226,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.013243+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:45.013243+0000","last_peered":"2024-09-18T11:53:45.013243+0000","last_clean":"2024-09-18T11:53:45.013243+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:45.013243+0000","last_undegraded":"2024-09-18T11:53:45.013243+0000","last_fullsized":"2024-09-18T11:53:45.013243+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16094454,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10936,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10757","reported_seq":10847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:44.035938+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:44.035938+0000","last_peered":"2024-09-18T11:53:44.035938+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:44.035938+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15971574,"num_objects":1950,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10755,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-52}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":256304524,"num_objects":31293,"num_object_clones":0,"num_object_copies":125172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175794,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7307,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512057344,"data_stored":512057344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716888,"kb_used_data":126480,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93654952,"statfs":{"total":96636764160,"available":95902670848,"internally_reserved":0,"allocated":129515520,"data_stored":129331699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":11.368},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":11.401}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":33.281999999999996},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":1.891}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.452},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":5.657}]}]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724412,"kb_used_data":126196,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93647428,"statfs":{"total":96636764160,"available":95894966272,"internally_reserved":0,"allocated":129224704,"data_stored":129040883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":8.4920000000000009},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":8.5229999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":10.007999999999999},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":11.377000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":1.353},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":10.917999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426840,"kb_used_data":124880,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93945000,"statfs":{"total":96636764160,"available":96199680000,"internally_reserved":0,"allocated":127877120,"data_stored":127693299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.247},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.399}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":1.7809999999999999},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.5030000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":3.1829999999999998},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":3.5910000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724172,"kb_used_data":125508,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93647668,"statfs":{"total":96636764160,"available":95895212032,"internally_reserved":0,"allocated":128520192,"data_stored":128336371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.7490000000000001},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.9700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":2.0449999999999999},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":20.018000000000001},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":13.198}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":127778816,"data_stored":127778816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127422464,"data_stored":127422464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128380928,"data_stored":128380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128475136,"data_stored":128475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:52.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:53.103 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:53.103 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:53.123 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-18T11:53:52.051542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247801228,"num_objects":30255,"num_object_clones":0,"num_object_copies":121020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176832,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7363,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160559,"ondisk_log_size":160559,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2576140,"kb_used_data":486864,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374911220,"statfs":{"total":386547056640,"available":383909089280,"internally_reserved":0,"allocated":498548736,"data_stored":497813452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,4,7,10,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17874944,"num_objects":-2182,"num_object_clones":0,"num_object_copies":-8728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2182,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004879"},"pg_stats":[{"pgid":"1.f","version":"37'11303","reported_seq":11404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.417286+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:48.417286+0000","last_peered":"2024-09-18T11:53:48.417286+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:48.417286+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15949824,"num_objects":1947,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11303,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11160","reported_seq":15452,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:50.038701+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:50.038701+0000","last_peered":"2024-09-18T11:53:50.038701+0000","last_clean":"2024-09-18T11:53:50.038701+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:50.038701+0000","last_undegraded":"2024-09-18T11:53:50.038701+0000","last_fullsized":"2024-09-18T11:53:50.038701+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15529206,"num_objects":1896,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10909","reported_seq":11026,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.481120+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:48.481120+0000","last_peered":"2024-09-18T11:53:48.481120+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:48.481120+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15555219,"num_objects":1900,"num_object_clones":0,"num_object_copies":7600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10908,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11045","reported_seq":11139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.208419+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:49.208419+0000","last_peered":"2024-09-18T11:53:49.208419+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:49.208419+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15406326,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11001","reported_seq":11122,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.497433+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:48.497433+0000","last_peered":"2024-09-18T11:53:48.497433+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:48.497433+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15512822,"num_objects":1894,"num_object_clones":0,"num_object_copies":7576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10999,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10948","reported_seq":11067,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.226284+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:48.226284+0000","last_peered":"2024-09-18T11:53:48.226284+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:48.226284+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15056896,"num_objects":1838,"num_object_clones":0,"num_object_copies":7352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11144","reported_seq":11251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.109443+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:49.109443+0000","last_peered":"2024-09-18T11:53:49.109443+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:49.109443+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15332598,"num_objects":1872,"num_object_clones":0,"num_object_copies":7488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-244}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10944","reported_seq":12502,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.240560+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:49.240560+0000","last_peered":"2024-09-18T11:53:49.240560+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:49.240560+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15815926,"num_objects":1931,"num_object_clones":0,"num_object_copies":7724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10942,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":746,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11283","reported_seq":15635,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:48.336209+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:48.336209+0000","last_peered":"2024-09-18T11:53:48.336209+0000","last_clean":"2024-09-18T11:53:48.336209+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:48.336209+0000","last_undegraded":"2024-09-18T11:53:48.336209+0000","last_fullsized":"2024-09-18T11:53:48.336209+0000","mapping_epoch":36,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15501804,"num_objects":1893,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11280,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10751","reported_seq":10865,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.405342+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:48.405342+0000","last_peered":"2024-09-18T11:53:48.405342+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:48.405342+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15095030,"num_objects":1843,"num_object_clones":0,"num_object_copies":7372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10749,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11095","reported_seq":11216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.423657+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:48.423657+0000","last_peered":"2024-09-18T11:53:48.423657+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:48.423657+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15871857,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11092,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-96}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11201","reported_seq":11292,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.445583+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:48.445583+0000","last_peered":"2024-09-18T11:53:48.445583+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:48.445583+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14989947,"num_objects":1830,"num_object_clones":0,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11200,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-254}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11115","reported_seq":11213,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.381683+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:48.381683+0000","last_peered":"2024-09-18T11:53:48.381683+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:48.381683+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15170171,"num_objects":1852,"num_object_clones":0,"num_object_copies":7408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11128","reported_seq":11253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.211031+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:48.211031+0000","last_peered":"2024-09-18T11:53:48.211031+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:48.211031+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16069878,"num_objects":1962,"num_object_clones":0,"num_object_copies":7848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11126,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11013","reported_seq":15302,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:50.037032+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:50.037032+0000","last_peered":"2024-09-18T11:53:50.037032+0000","last_clean":"2024-09-18T11:53:50.037032+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:50.037032+0000","last_undegraded":"2024-09-18T11:53:50.037032+0000","last_fullsized":"2024-09-18T11:53:50.037032+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15480054,"num_objects":1890,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10819","reported_seq":10909,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.161009+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:49.161009+0000","last_peered":"2024-09-18T11:53:49.161009+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:49.161009+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15463670,"num_objects":1888,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10817,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":247801228,"num_objects":30255,"num_object_clones":0,"num_object_copies":121020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176832,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7363,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495308800,"data_stored":495308800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160559,"ondisk_log_size":160559,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716948,"kb_used_data":122380,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93654892,"statfs":{"total":96636764160,"available":95902609408,"internally_reserved":0,"allocated":125317120,"data_stored":125133299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":9.016},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":3.1280000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":2.2999999999999998},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":2.1699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":22.190999999999999},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.093}]}]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707900,"kb_used_data":122152,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93663940,"statfs":{"total":96636764160,"available":95911874560,"internally_reserved":0,"allocated":125083648,"data_stored":124899827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":9.5530000000000008},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":9.5749999999999993}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":46.793999999999997},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":9.5120000000000005},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":7.7949999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427056,"kb_used_data":120932,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93944784,"statfs":{"total":96636764160,"available":96199458816,"internally_reserved":0,"allocated":123834368,"data_stored":123650547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,0,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":1.587},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":6.9130000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":18.512},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":25.481000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":7.032},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":7.484}]}]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724236,"kb_used_data":121400,"kb_used_omap":4,"kb_used_meta":602811,"kb_avail":93647604,"statfs":{"total":96636764160,"available":95895146496,"internally_reserved":0,"allocated":124313600,"data_stored":124129779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617279041},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":18.587},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":15.654}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":12.541},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":9.7189999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.1230000000000002},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":9.5380000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":123633664,"data_stored":123633664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123322368,"data_stored":123322368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124166144,"data_stored":124166144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124186624,"data_stored":124186624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:53.124 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:53.723 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:53.723 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:53.753 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-18T11:53:52.051542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247801228,"num_objects":30255,"num_object_clones":0,"num_object_copies":121020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176832,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7363,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160559,"ondisk_log_size":160559,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2576140,"kb_used_data":486864,"kb_used_omap":14,"kb_used_meta":2089201,"kb_avail":374911220,"statfs":{"total":386547056640,"available":383909089280,"internally_reserved":0,"allocated":498548736,"data_stored":497813452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139341851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,4,7,10,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17874944,"num_objects":-2182,"num_object_clones":0,"num_object_copies":-8728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2182,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004879"},"pg_stats":[{"pgid":"1.f","version":"37'11303","reported_seq":11404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.417286+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:48.417286+0000","last_peered":"2024-09-18T11:53:48.417286+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:48.417286+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15949824,"num_objects":1947,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11303,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11160","reported_seq":15452,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:50.038701+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:50.038701+0000","last_peered":"2024-09-18T11:53:50.038701+0000","last_clean":"2024-09-18T11:53:50.038701+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:50.038701+0000","last_undegraded":"2024-09-18T11:53:50.038701+0000","last_fullsized":"2024-09-18T11:53:50.038701+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15529206,"num_objects":1896,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10909","reported_seq":11026,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.481120+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:48.481120+0000","last_peered":"2024-09-18T11:53:48.481120+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:48.481120+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15555219,"num_objects":1900,"num_object_clones":0,"num_object_copies":7600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10908,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11045","reported_seq":11139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.208419+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:49.208419+0000","last_peered":"2024-09-18T11:53:49.208419+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:49.208419+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15406326,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11001","reported_seq":11122,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.497433+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:48.497433+0000","last_peered":"2024-09-18T11:53:48.497433+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:48.497433+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15512822,"num_objects":1894,"num_object_clones":0,"num_object_copies":7576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10999,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10948","reported_seq":11067,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.226284+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:48.226284+0000","last_peered":"2024-09-18T11:53:48.226284+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:48.226284+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15056896,"num_objects":1838,"num_object_clones":0,"num_object_copies":7352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11144","reported_seq":11251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.109443+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:49.109443+0000","last_peered":"2024-09-18T11:53:49.109443+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:49.109443+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15332598,"num_objects":1872,"num_object_clones":0,"num_object_copies":7488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-244}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10944","reported_seq":12502,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.240560+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:49.240560+0000","last_peered":"2024-09-18T11:53:49.240560+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:49.240560+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15815926,"num_objects":1931,"num_object_clones":0,"num_object_copies":7724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10942,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":746,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11283","reported_seq":15635,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:48.336209+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:48.336209+0000","last_peered":"2024-09-18T11:53:48.336209+0000","last_clean":"2024-09-18T11:53:48.336209+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:48.336209+0000","last_undegraded":"2024-09-18T11:53:48.336209+0000","last_fullsized":"2024-09-18T11:53:48.336209+0000","mapping_epoch":36,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15501804,"num_objects":1893,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11280,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10751","reported_seq":10865,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.405342+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:48.405342+0000","last_peered":"2024-09-18T11:53:48.405342+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:48.405342+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15095030,"num_objects":1843,"num_object_clones":0,"num_object_copies":7372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10749,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11095","reported_seq":11216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.423657+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:48.423657+0000","last_peered":"2024-09-18T11:53:48.423657+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:48.423657+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15871857,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11092,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-96}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11201","reported_seq":11292,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.445583+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:48.445583+0000","last_peered":"2024-09-18T11:53:48.445583+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:48.445583+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14989947,"num_objects":1830,"num_object_clones":0,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11200,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-254}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11115","reported_seq":11213,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.381683+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:48.381683+0000","last_peered":"2024-09-18T11:53:48.381683+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:48.381683+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15170171,"num_objects":1852,"num_object_clones":0,"num_object_copies":7408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11128","reported_seq":11253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:48.211031+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:48.211031+0000","last_peered":"2024-09-18T11:53:48.211031+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:48.211031+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16069878,"num_objects":1962,"num_object_clones":0,"num_object_copies":7848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11126,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11013","reported_seq":15302,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:50.037032+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:50.037032+0000","last_peered":"2024-09-18T11:53:50.037032+0000","last_clean":"2024-09-18T11:53:50.037032+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:50.037032+0000","last_undegraded":"2024-09-18T11:53:50.037032+0000","last_fullsized":"2024-09-18T11:53:50.037032+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15480054,"num_objects":1890,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10819","reported_seq":10909,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.161009+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:49.161009+0000","last_peered":"2024-09-18T11:53:49.161009+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:49.161009+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15463670,"num_objects":1888,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10817,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":247801228,"num_objects":30255,"num_object_clones":0,"num_object_copies":121020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176832,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7363,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495308800,"data_stored":495308800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160559,"ondisk_log_size":160559,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716948,"kb_used_data":122380,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93654892,"statfs":{"total":96636764160,"available":95902609408,"internally_reserved":0,"allocated":125317120,"data_stored":125133299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":9.016},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":3.1280000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":2.2999999999999998},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":2.1699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":22.190999999999999},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.093}]}]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707900,"kb_used_data":122152,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93663940,"statfs":{"total":96636764160,"available":95911874560,"internally_reserved":0,"allocated":125083648,"data_stored":124899827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":9.5530000000000008},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":9.5749999999999993}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":46.793999999999997},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":9.5120000000000005},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":7.7949999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427056,"kb_used_data":120932,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93944784,"statfs":{"total":96636764160,"available":96199458816,"internally_reserved":0,"allocated":123834368,"data_stored":123650547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,0,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":1.587},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":6.9130000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":18.512},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":25.481000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":7.032},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":7.484}]}]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724236,"kb_used_data":121400,"kb_used_omap":4,"kb_used_meta":602811,"kb_avail":93647604,"statfs":{"total":96636764160,"available":95895146496,"internally_reserved":0,"allocated":124313600,"data_stored":124129779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617279041},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":18.587},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":15.654}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":12.541},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":9.7189999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.1230000000000002},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":9.5380000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":123633664,"data_stored":123633664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123322368,"data_stored":123322368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124166144,"data_stored":124166144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124186624,"data_stored":124186624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:53.754 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:53.754 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11303", 'reported_seq': 11404, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:48.417286+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:53:48.417286+0000', 'last_peered': '2024-09-18T11:53:48.417286+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:53:48.417286+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15949824, 'num_objects': 1947, 'num_object_clones': 0, 'num_object_copies': 7788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11303, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -156}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:53.754 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:53.754 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10909", 'reported_seq': 11026, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:48.481120+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:53:48.481120+0000', 'last_peered': '2024-09-18T11:53:48.481120+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:53:48.481120+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15555219, 'num_objects': 1900, 'num_object_clones': 0, 'num_object_copies': 7600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1900, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10908, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -140}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:53.755 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:53.755 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11045", 'reported_seq': 11139, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:49.208419+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:49.208419+0000', 'last_peered': '2024-09-18T11:53:49.208419+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:49.208419+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15406326, 'num_objects': 1881, 'num_object_clones': 0, 'num_object_copies': 7524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11043, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:53.755 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:53.755 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11001", 'reported_seq': 11122, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:48.497433+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:53:48.497433+0000', 'last_peered': '2024-09-18T11:53:48.497433+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:53:48.497433+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15512822, 'num_objects': 1894, 'num_object_clones': 0, 'num_object_copies': 7576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1894, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10999, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -158}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:53.755 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:53.755 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10948", 'reported_seq': 11067, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:48.226284+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:53:48.226284+0000', 'last_peered': '2024-09-18T11:53:48.226284+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:53:48.226284+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15056896, 'num_objects': 1838, 'num_object_clones': 0, 'num_object_copies': 7352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1838, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10948, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -281}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:53.755 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:53.755 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11144", 'reported_seq': 11251, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:49.109443+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:49.109443+0000', 'last_peered': '2024-09-18T11:53:49.109443+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:49.109443+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15332598, 'num_objects': 1872, 'num_object_clones': 0, 'num_object_copies': 7488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11142, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -244}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:53.755 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:53.755 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10944", 'reported_seq': 12502, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:49.240560+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:49.240560+0000', 'last_peered': '2024-09-18T11:53:49.240560+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:49.240560+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15815926, 'num_objects': 1931, 'num_object_clones': 0, 'num_object_copies': 7724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10942, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 746, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:53.756 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:53.756 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10751", 'reported_seq': 10865, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:48.405342+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:53:48.405342+0000', 'last_peered': '2024-09-18T11:53:48.405342+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:53:48.405342+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15095030, 'num_objects': 1843, 'num_object_clones': 0, 'num_object_copies': 7372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10749, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -120}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:53.756 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:53.756 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11095", 'reported_seq': 11216, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:48.423657+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:53:48.423657+0000', 'last_peered': '2024-09-18T11:53:48.423657+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:53:48.423657+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15871857, 'num_objects': 1938, 'num_object_clones': 0, 'num_object_copies': 7752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11092, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -96}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:53.756 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:53.756 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11201", 'reported_seq': 11292, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:48.445583+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:53:48.445583+0000', 'last_peered': '2024-09-18T11:53:48.445583+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:53:48.445583+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14989947, 'num_objects': 1830, 'num_object_clones': 0, 'num_object_copies': 7320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1830, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11200, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -254}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:53.756 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:53.756 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11115", 'reported_seq': 11213, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:48.381683+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:53:48.381683+0000', 'last_peered': '2024-09-18T11:53:48.381683+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:53:48.381683+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15170171, 'num_objects': 1852, 'num_object_clones': 0, 'num_object_copies': 7408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11114, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:53.756 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:53.756 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11128", 'reported_seq': 11253, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:48.211031+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:53:48.211031+0000', 'last_peered': '2024-09-18T11:53:48.211031+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:53:48.211031+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16069878, 'num_objects': 1962, 'num_object_clones': 0, 'num_object_copies': 7848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1962, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11126, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:53.757 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:53.757 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10819", 'reported_seq': 10909, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:49.161009+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:49.161009+0000', 'last_peered': '2024-09-18T11:53:49.161009+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:49.161009+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15463670, 'num_objects': 1888, 'num_object_clones': 0, 'num_object_copies': 7552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10817, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -114}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:53.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:53:54.469 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:54.469 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26093/118468 objects degraded (22.025%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":632,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":29617,"data_bytes":242574732,"bytes_used":2638053376,"bytes_avail":383909003264,"bytes_total":386547056640,"degraded_objects":26093,"degraded_total":118468,"degraded_ratio":0.2202535705844616,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":224},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:53:54.502 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:53:54.503 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:55.114 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:55.116 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:55.140 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-09-18T11:53:54.051999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242574732,"num_objects":29617,"num_object_clones":0,"num_object_copies":118468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29617,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177470,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7363,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160797,"ondisk_log_size":160797,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2576224,"kb_used_data":478640,"kb_used_omap":14,"kb_used_meta":2097521,"kb_avail":374911136,"statfs":{"total":386547056640,"available":383909003264,"internally_reserved":0,"allocated":490127360,"data_stored":489392076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2147861531},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,6,10,6,5,13,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22085632,"num_objects":-2696,"num_object_clones":0,"num_object_copies":-10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004854"},"pg_stats":[{"pgid":"1.f","version":"37'11376","reported_seq":11477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.590639+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:53.590639+0000","last_peered":"2024-09-18T11:53:53.590639+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:53.590639+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15351808,"num_objects":1874,"num_object_clones":0,"num_object_copies":7496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11160","reported_seq":15452,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:50.038701+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:50.038701+0000","last_peered":"2024-09-18T11:53:50.038701+0000","last_clean":"2024-09-18T11:53:50.038701+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:50.038701+0000","last_undegraded":"2024-09-18T11:53:50.038701+0000","last_fullsized":"2024-09-18T11:53:50.038701+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15529206,"num_objects":1896,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10967","reported_seq":11083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.512392+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:53.512392+0000","last_peered":"2024-09-18T11:53:53.512392+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:53.512392+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15080083,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10966,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-198}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11045","reported_seq":11139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.208419+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:49.208419+0000","last_peered":"2024-09-18T11:53:49.208419+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:49.208419+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15406326,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11061","reported_seq":11182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.371268+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:53.371268+0000","last_peered":"2024-09-18T11:53:53.371268+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:53.371268+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15021302,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11059,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11020","reported_seq":11139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.424330+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:53.424330+0000","last_peered":"2024-09-18T11:53:53.424330+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:53.424330+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14467072,"num_objects":1766,"num_object_clones":0,"num_object_copies":7064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11020,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11144","reported_seq":11251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.109443+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:49.109443+0000","last_peered":"2024-09-18T11:53:49.109443+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:49.109443+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15332598,"num_objects":1872,"num_object_clones":0,"num_object_copies":7488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-244}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10944","reported_seq":12502,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.240560+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:49.240560+0000","last_peered":"2024-09-18T11:53:49.240560+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:49.240560+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15815926,"num_objects":1931,"num_object_clones":0,"num_object_copies":7724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10942,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":746,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11342","reported_seq":15695,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:53.620102+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:53.620102+0000","last_peered":"2024-09-18T11:53:53.620102+0000","last_clean":"2024-09-18T11:53:53.620102+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:53.620102+0000","last_undegraded":"2024-09-18T11:53:53.620102+0000","last_fullsized":"2024-09-18T11:53:53.620102+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15018476,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11339,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10812","reported_seq":10926,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.507588+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:53.507588+0000","last_peered":"2024-09-18T11:53:53.507588+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:53.507588+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14595318,"num_objects":1782,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-181}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11166","reported_seq":11287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.476523+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:53.476523+0000","last_peered":"2024-09-18T11:53:53.476523+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:53.476523+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15290225,"num_objects":1867,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11163,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11253","reported_seq":11346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.268915+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:53.268915+0000","last_peered":"2024-09-18T11:53:53.268915+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:53.268915+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14563963,"num_objects":1778,"num_object_clones":0,"num_object_copies":7112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11252,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-306}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11180","reported_seq":11278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.329587+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:53.329587+0000","last_peered":"2024-09-18T11:53:53.329587+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:53.329587+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14637691,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11195","reported_seq":11320,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.557102+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:53.557102+0000","last_peered":"2024-09-18T11:53:53.557102+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:53.557102+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15521014,"num_objects":1895,"num_object_clones":0,"num_object_copies":7580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-228}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11013","reported_seq":15302,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:50.037032+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:50.037032+0000","last_peered":"2024-09-18T11:53:50.037032+0000","last_clean":"2024-09-18T11:53:50.037032+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:50.037032+0000","last_undegraded":"2024-09-18T11:53:50.037032+0000","last_fullsized":"2024-09-18T11:53:50.037032+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15480054,"num_objects":1890,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10819","reported_seq":10909,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:49.161009+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:49.161009+0000","last_peered":"2024-09-18T11:53:49.161009+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:49.161009+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15463670,"num_objects":1888,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10817,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":242574732,"num_objects":29617,"num_object_clones":0,"num_object_copies":118468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29617,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177470,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7363,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487022592,"data_stored":487022592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160797,"ondisk_log_size":160797,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716940,"kb_used_data":118212,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93654900,"statfs":{"total":96636764160,"available":95902617600,"internally_reserved":0,"allocated":121049088,"data_stored":120865267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,3,4,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":11.880000000000001},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.569}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.983000000000001},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":8.7810000000000006}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":8.8559999999999999},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":8.8170000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707992,"kb_used_data":118096,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93663848,"statfs":{"total":96636764160,"available":95911780352,"internally_reserved":0,"allocated":120930304,"data_stored":120746483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":1.02},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.054}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":7.9690000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":0.71599999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427056,"kb_used_data":120932,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93944784,"statfs":{"total":96636764160,"available":96199458816,"internally_reserved":0,"allocated":123834368,"data_stored":123650547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,0,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":1.587},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":6.9130000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":18.512},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":25.481000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":7.032},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":7.484}]}]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724236,"kb_used_data":121400,"kb_used_omap":4,"kb_used_meta":602811,"kb_avail":93647604,"statfs":{"total":96636764160,"available":95895146496,"internally_reserved":0,"allocated":124313600,"data_stored":124129779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617279041},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":18.587},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":15.654}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":12.541},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":9.7189999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":7.1230000000000002},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":9.5380000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":123633664,"data_stored":123633664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123322368,"data_stored":123322368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":120045568,"data_stored":120045568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120020992,"data_stored":120020992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:58.142 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:58.702 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:58.703 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:58.728 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-18T11:53:58.052978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":239584652,"num_objects":29252,"num_object_clones":0,"num_object_copies":117008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177835,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7426,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2847960,"kb_used_data":470320,"kb_used_omap":14,"kb_used_meta":2377585,"kb_avail":374639400,"statfs":{"total":386547056640,"available":383630745600,"internally_reserved":0,"allocated":481607680,"data_stored":480872396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2434647067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,2,7,9,12,12,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16719872,"num_objects":-2041,"num_object_clones":0,"num_object_copies":-8164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2041,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004880"},"pg_stats":[{"pgid":"1.f","version":"37'11376","reported_seq":11477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.590639+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:53.590639+0000","last_peered":"2024-09-18T11:53:53.590639+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:53.590639+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15351808,"num_objects":1874,"num_object_clones":0,"num_object_copies":7496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11233","reported_seq":15525,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:54.971454+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:54.971454+0000","last_peered":"2024-09-18T11:53:54.971454+0000","last_clean":"2024-09-18T11:53:54.971454+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:54.971454+0000","last_undegraded":"2024-09-18T11:53:54.971454+0000","last_fullsized":"2024-09-18T11:53:54.971454+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14931190,"num_objects":1823,"num_object_clones":0,"num_object_copies":7292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11231,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10967","reported_seq":11083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.512392+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:53.512392+0000","last_peered":"2024-09-18T11:53:53.512392+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:53.512392+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15080083,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10966,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-198}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11102","reported_seq":11196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:54.251651+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:54.251651+0000","last_peered":"2024-09-18T11:53:54.251651+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:54.251651+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14939382,"num_objects":1824,"num_object_clones":0,"num_object_copies":7296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11100,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-277}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11061","reported_seq":11182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.371268+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:53.371268+0000","last_peered":"2024-09-18T11:53:53.371268+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:53.371268+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15021302,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11059,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11020","reported_seq":11139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.424330+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:53.424330+0000","last_peered":"2024-09-18T11:53:53.424330+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:53.424330+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14467072,"num_objects":1766,"num_object_clones":0,"num_object_copies":7064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11020,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11208","reported_seq":11315,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:54.184722+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:54.184722+0000","last_peered":"2024-09-18T11:53:54.184722+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:54.184722+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14808310,"num_objects":1808,"num_object_clones":0,"num_object_copies":7232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11206,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-308}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11019","reported_seq":12703,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:54.379331+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:54.379331+0000","last_peered":"2024-09-18T11:53:54.379331+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:54.379331+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15201526,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":809,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":627}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11342","reported_seq":15695,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:53.620102+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:53.620102+0000","last_peered":"2024-09-18T11:53:53.620102+0000","last_clean":"2024-09-18T11:53:53.620102+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:53.620102+0000","last_undegraded":"2024-09-18T11:53:53.620102+0000","last_fullsized":"2024-09-18T11:53:53.620102+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15018476,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11339,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10812","reported_seq":10926,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.507588+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:53.507588+0000","last_peered":"2024-09-18T11:53:53.507588+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:53.507588+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14595318,"num_objects":1782,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-181}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11166","reported_seq":11287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.476523+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:53.476523+0000","last_peered":"2024-09-18T11:53:53.476523+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:53.476523+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15290225,"num_objects":1867,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11163,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11253","reported_seq":11346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.268915+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:53.268915+0000","last_peered":"2024-09-18T11:53:53.268915+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:53.268915+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14563963,"num_objects":1778,"num_object_clones":0,"num_object_copies":7112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11252,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-306}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11180","reported_seq":11278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.329587+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:53.329587+0000","last_peered":"2024-09-18T11:53:53.329587+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:53.329587+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14637691,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11195","reported_seq":11320,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.557102+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:53.557102+0000","last_peered":"2024-09-18T11:53:53.557102+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:53.557102+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15521014,"num_objects":1895,"num_object_clones":0,"num_object_copies":7580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-228}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11072","reported_seq":15361,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:55.162368+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:55.162368+0000","last_peered":"2024-09-18T11:53:55.162368+0000","last_clean":"2024-09-18T11:53:55.162368+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:55.162368+0000","last_undegraded":"2024-09-18T11:53:55.162368+0000","last_fullsized":"2024-09-18T11:53:55.162368+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14996726,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11070,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10856","reported_seq":10946,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:54.124452+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:54.124452+0000","last_peered":"2024-09-18T11:53:54.124452+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:54.124452+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15160566,"num_objects":1851,"num_object_clones":0,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10854,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-151}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":239584652,"num_objects":29252,"num_object_clones":0,"num_object_copies":117008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177835,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7426,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":478633984,"data_stored":478633984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716940,"kb_used_data":118212,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93654900,"statfs":{"total":96636764160,"available":95902617600,"internally_reserved":0,"allocated":121049088,"data_stored":120865267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,3,4,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":11.880000000000001},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.569}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.983000000000001},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":8.7810000000000006}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":8.8559999999999999},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":8.8170000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707992,"kb_used_data":118096,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93663848,"statfs":{"total":96636764160,"available":95911780352,"internally_reserved":0,"allocated":120930304,"data_stored":120746483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":1.02},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.054}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":7.9690000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":0.71599999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715460,"kb_used_data":116788,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93656380,"statfs":{"total":96636764160,"available":95904133120,"internally_reserved":0,"allocated":119590912,"data_stored":119407091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":7.1779999999999999},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":12.771000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":2.7530000000000001},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":7.2130000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.375},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":2.8839999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707568,"kb_used_data":117224,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93664272,"statfs":{"total":96636764160,"available":95912214528,"internally_reserved":0,"allocated":120037376,"data_stored":119853555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":8.0549999999999997},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.0449999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":3.8330000000000002},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":4.4800000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":1.78},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":1.879}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119435264,"data_stored":119435264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119132160,"data_stored":119132160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":120045568,"data_stored":120045568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120020992,"data_stored":120020992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:58.730 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:59.307 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:53:59.308 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:53:59.335 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-18T11:53:58.052978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":239584652,"num_objects":29252,"num_object_clones":0,"num_object_copies":117008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177835,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7426,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2847960,"kb_used_data":470320,"kb_used_omap":14,"kb_used_meta":2377585,"kb_avail":374639400,"statfs":{"total":386547056640,"available":383630745600,"internally_reserved":0,"allocated":481607680,"data_stored":480872396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2434647067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,2,7,9,12,12,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16719872,"num_objects":-2041,"num_object_clones":0,"num_object_copies":-8164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2041,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004880"},"pg_stats":[{"pgid":"1.f","version":"37'11376","reported_seq":11477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.590639+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:53.590639+0000","last_peered":"2024-09-18T11:53:53.590639+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:53.590639+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15351808,"num_objects":1874,"num_object_clones":0,"num_object_copies":7496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11233","reported_seq":15525,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:54.971454+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:54.971454+0000","last_peered":"2024-09-18T11:53:54.971454+0000","last_clean":"2024-09-18T11:53:54.971454+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:54.971454+0000","last_undegraded":"2024-09-18T11:53:54.971454+0000","last_fullsized":"2024-09-18T11:53:54.971454+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14931190,"num_objects":1823,"num_object_clones":0,"num_object_copies":7292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11231,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'10967","reported_seq":11083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.512392+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:53.512392+0000","last_peered":"2024-09-18T11:53:53.512392+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:53.512392+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15080083,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10966,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-198}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11102","reported_seq":11196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:54.251651+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:54.251651+0000","last_peered":"2024-09-18T11:53:54.251651+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:54.251651+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14939382,"num_objects":1824,"num_object_clones":0,"num_object_copies":7296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11100,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-277}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11061","reported_seq":11182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.371268+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:53.371268+0000","last_peered":"2024-09-18T11:53:53.371268+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:53.371268+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15021302,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11059,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11020","reported_seq":11139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.424330+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:53.424330+0000","last_peered":"2024-09-18T11:53:53.424330+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:53.424330+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14467072,"num_objects":1766,"num_object_clones":0,"num_object_copies":7064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11020,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11208","reported_seq":11315,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:54.184722+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:54.184722+0000","last_peered":"2024-09-18T11:53:54.184722+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:54.184722+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14808310,"num_objects":1808,"num_object_clones":0,"num_object_copies":7232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11206,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-308}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11019","reported_seq":12703,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:54.379331+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:54.379331+0000","last_peered":"2024-09-18T11:53:54.379331+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:54.379331+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15201526,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":809,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":627}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11342","reported_seq":15695,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:53.620102+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:53.620102+0000","last_peered":"2024-09-18T11:53:53.620102+0000","last_clean":"2024-09-18T11:53:53.620102+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:53.620102+0000","last_undegraded":"2024-09-18T11:53:53.620102+0000","last_fullsized":"2024-09-18T11:53:53.620102+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15018476,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11339,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10812","reported_seq":10926,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.507588+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:53.507588+0000","last_peered":"2024-09-18T11:53:53.507588+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:53.507588+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14595318,"num_objects":1782,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-181}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11166","reported_seq":11287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.476523+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:53.476523+0000","last_peered":"2024-09-18T11:53:53.476523+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:53.476523+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15290225,"num_objects":1867,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11163,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11253","reported_seq":11346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.268915+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:53.268915+0000","last_peered":"2024-09-18T11:53:53.268915+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:53.268915+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14563963,"num_objects":1778,"num_object_clones":0,"num_object_copies":7112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11252,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-306}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11180","reported_seq":11278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.329587+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:53.329587+0000","last_peered":"2024-09-18T11:53:53.329587+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:53.329587+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14637691,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11195","reported_seq":11320,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:53.557102+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:53.557102+0000","last_peered":"2024-09-18T11:53:53.557102+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:53.557102+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15521014,"num_objects":1895,"num_object_clones":0,"num_object_copies":7580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-228}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11072","reported_seq":15361,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:55.162368+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:55.162368+0000","last_peered":"2024-09-18T11:53:55.162368+0000","last_clean":"2024-09-18T11:53:55.162368+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:55.162368+0000","last_undegraded":"2024-09-18T11:53:55.162368+0000","last_fullsized":"2024-09-18T11:53:55.162368+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14996726,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11070,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10856","reported_seq":10946,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:54.124452+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:54.124452+0000","last_peered":"2024-09-18T11:53:54.124452+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:54.124452+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15160566,"num_objects":1851,"num_object_clones":0,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10854,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-151}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":239584652,"num_objects":29252,"num_object_clones":0,"num_object_copies":117008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177835,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7426,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":478633984,"data_stored":478633984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716940,"kb_used_data":118212,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93654900,"statfs":{"total":96636764160,"available":95902617600,"internally_reserved":0,"allocated":121049088,"data_stored":120865267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,3,4,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":11.880000000000001},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.569}]},{"osd":1,"last update":"Wed Sep 18 11:52:55 2024","interfaces":[{"interface":"back","average":{"1min":8.8710000000000004,"5min":7.1879999999999997,"15min":6.0940000000000003},"min":{"1min":0.749,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":23.948,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.983000000000001},{"interface":"front","average":{"1min":7.3150000000000004,"5min":6.6520000000000001,"15min":4.9859999999999998},"min":{"1min":0.32100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":26.242999999999999,"5min":57.372,"15min":57.372},"last":8.7810000000000006}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":8.8559999999999999},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":8.8170000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707992,"kb_used_data":118096,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93663848,"statfs":{"total":96636764160,"available":95911780352,"internally_reserved":0,"allocated":120930304,"data_stored":120746483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":1.02},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.054}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":7.9690000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":0.71599999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715460,"kb_used_data":116788,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93656380,"statfs":{"total":96636764160,"available":95904133120,"internally_reserved":0,"allocated":119590912,"data_stored":119407091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":7.1779999999999999},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":12.771000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":2.7530000000000001},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":7.2130000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.375},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":2.8839999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707568,"kb_used_data":117224,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93664272,"statfs":{"total":96636764160,"available":95912214528,"internally_reserved":0,"allocated":120037376,"data_stored":119853555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":8.0549999999999997},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.0449999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":3.8330000000000002},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":4.4800000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":1.78},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":1.879}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119435264,"data_stored":119435264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119132160,"data_stored":119132160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":120045568,"data_stored":120045568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120020992,"data_stored":120020992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:59.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:53:59.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11376", 'reported_seq': 11477, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:53.590639+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:53:53.590639+0000', 'last_peered': '2024-09-18T11:53:53.590639+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:53:53.590639+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15351808, 'num_objects': 1874, 'num_object_clones': 0, 'num_object_copies': 7496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11376, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -229}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:59.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:53:59.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10967", 'reported_seq': 11083, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:53.512392+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:53:53.512392+0000', 'last_peered': '2024-09-18T11:53:53.512392+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:53:53.512392+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15080083, 'num_objects': 1842, 'num_object_clones': 0, 'num_object_copies': 7368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1842, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10966, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -198}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:59.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:53:59.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11102", 'reported_seq': 11196, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:54.251651+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:54.251651+0000', 'last_peered': '2024-09-18T11:53:54.251651+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:54.251651+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14939382, 'num_objects': 1824, 'num_object_clones': 0, 'num_object_copies': 7296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1824, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11100, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:59.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:53:59.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11061", 'reported_seq': 11182, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:53.371268+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:53:53.371268+0000', 'last_peered': '2024-09-18T11:53:53.371268+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:53:53.371268+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15021302, 'num_objects': 1834, 'num_object_clones': 0, 'num_object_copies': 7336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11059, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -218}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:59.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:53:59.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11020", 'reported_seq': 11139, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:53.424330+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:53:53.424330+0000', 'last_peered': '2024-09-18T11:53:53.424330+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:53:53.424330+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14467072, 'num_objects': 1766, 'num_object_clones': 0, 'num_object_copies': 7064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11020, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -353}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:59.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:53:59.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11208", 'reported_seq': 11315, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:54.184722+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:54.184722+0000', 'last_peered': '2024-09-18T11:53:54.184722+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:54.184722+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14808310, 'num_objects': 1808, 'num_object_clones': 0, 'num_object_copies': 7232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1808, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11206, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -308}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:59.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:53:59.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11019", 'reported_seq': 12703, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:54.379331+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:54.379331+0000', 'last_peered': '2024-09-18T11:53:54.379331+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:54.379331+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15201526, 'num_objects': 1856, 'num_object_clones': 0, 'num_object_copies': 7424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11017, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 809, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 627}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10812", 'reported_seq': 10926, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:53.507588+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:53:53.507588+0000', 'last_peered': '2024-09-18T11:53:53.507588+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:53:53.507588+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14595318, 'num_objects': 1782, 'num_object_clones': 0, 'num_object_copies': 7128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10810, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -181}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11166", 'reported_seq': 11287, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:53.476523+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:53:53.476523+0000', 'last_peered': '2024-09-18T11:53:53.476523+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:53:53.476523+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15290225, 'num_objects': 1867, 'num_object_clones': 0, 'num_object_copies': 7468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11163, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -167}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11253", 'reported_seq': 11346, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:53.268915+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:53:53.268915+0000', 'last_peered': '2024-09-18T11:53:53.268915+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:53:53.268915+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14563963, 'num_objects': 1778, 'num_object_clones': 0, 'num_object_copies': 7112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11252, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11180", 'reported_seq': 11278, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:53.329587+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:53:53.329587+0000', 'last_peered': '2024-09-18T11:53:53.329587+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:53:53.329587+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14637691, 'num_objects': 1787, 'num_object_clones': 0, 'num_object_copies': 7148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11179, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -259}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11195", 'reported_seq': 11320, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:53.557102+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:53:53.557102+0000', 'last_peered': '2024-09-18T11:53:53.557102+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:53:53.557102+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15521014, 'num_objects': 1895, 'num_object_clones': 0, 'num_object_copies': 7580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1895, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11193, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -228}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:53:59.338 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:53:59.339 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10856", 'reported_seq': 10946, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:54.124452+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:54.124452+0000', 'last_peered': '2024-09-18T11:53:54.124452+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:54.124452+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15160566, 'num_objects': 1851, 'num_object_clones': 0, 'num_object_copies': 7404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10854, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -151}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:53:59.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:00.055 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:00.056 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26030/117008 objects degraded (22.246%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":638,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":29252,"data_bytes":239584652,"bytes_used":2916311040,"bytes_avail":383630745600,"bytes_total":386547056640,"degraded_objects":26030,"degraded_total":117008,"degraded_ratio":0.22246342130452618,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":170},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:00.080 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:00.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:00.608 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:00.609 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:00.626 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-18T11:54:00.055345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":232179084,"num_objects":28348,"num_object_clones":0,"num_object_copies":113392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28348,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178739,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7472,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2830840,"kb_used_data":457356,"kb_used_omap":14,"kb_used_meta":2373425,"kb_avail":374656520,"statfs":{"total":386547056640,"available":383648276480,"internally_reserved":0,"allocated":468332544,"data_stored":467597260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430387227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,11,11,11,11,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24125440,"num_objects":-2945,"num_object_clones":0,"num_object_copies":-11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006791"},"pg_stats":[{"pgid":"1.f","version":"37'11450","reported_seq":11551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:58.438059+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:53:58.438059+0000","last_peered":"2024-09-18T11:53:58.438059+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:53:58.438059+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14745600,"num_objects":1800,"num_object_clones":0,"num_object_copies":7200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11233","reported_seq":15525,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:54.971454+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:53:54.971454+0000","last_peered":"2024-09-18T11:53:54.971454+0000","last_clean":"2024-09-18T11:53:54.971454+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:53:54.971454+0000","last_undegraded":"2024-09-18T11:53:54.971454+0000","last_fullsized":"2024-09-18T11:53:54.971454+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14931190,"num_objects":1823,"num_object_clones":0,"num_object_copies":7292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11231,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11035","reported_seq":11152,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:58.507339+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:53:58.507339+0000","last_peered":"2024-09-18T11:53:58.507339+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:53:58.507339+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14523027,"num_objects":1774,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11034,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-266}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11167","reported_seq":11261,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.442738+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:59.442738+0000","last_peered":"2024-09-18T11:53:59.442738+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:59.442738+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14406902,"num_objects":1759,"num_object_clones":0,"num_object_copies":7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11118","reported_seq":11238,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:58.652790+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:53:58.652790+0000","last_peered":"2024-09-18T11:53:58.652790+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:53:58.652790+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14554358,"num_objects":1777,"num_object_clones":0,"num_object_copies":7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11116,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11094","reported_seq":11213,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:58.531218+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:53:58.531218+0000","last_peered":"2024-09-18T11:53:58.531218+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:53:58.531218+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13860864,"num_objects":1692,"num_object_clones":0,"num_object_copies":6768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11270","reported_seq":11377,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.421717+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:59.421717+0000","last_peered":"2024-09-18T11:53:59.421717+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:59.421717+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14300406,"num_objects":1746,"num_object_clones":0,"num_object_copies":6984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11268,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-370}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11075","reported_seq":12851,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.471853+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:59.471853+0000","last_peered":"2024-09-18T11:53:59.471853+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:59.471853+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14742774,"num_objects":1800,"num_object_clones":0,"num_object_copies":7200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11073,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":855,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11406","reported_seq":15758,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:58.538524+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:53:58.538524+0000","last_peered":"2024-09-18T11:53:58.538524+0000","last_clean":"2024-09-18T11:53:58.538524+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:53:58.538524+0000","last_undegraded":"2024-09-18T11:53:58.538524+0000","last_fullsized":"2024-09-18T11:53:58.538524+0000","mapping_epoch":36,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14494188,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10875","reported_seq":10989,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:58.589940+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:53:58.589940+0000","last_peered":"2024-09-18T11:53:58.589940+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:53:58.589940+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14079222,"num_objects":1719,"num_object_clones":0,"num_object_copies":6876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10873,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-244}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11229","reported_seq":11350,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:58.419712+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:53:58.419712+0000","last_peered":"2024-09-18T11:53:58.419712+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:53:58.419712+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14774129,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11226,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11313","reported_seq":11406,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:58.533281+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:53:58.533281+0000","last_peered":"2024-09-18T11:53:58.533281+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:53:58.533281+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14072443,"num_objects":1718,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11312,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-366}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11245","reported_seq":11343,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:58.641087+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:53:58.641087+0000","last_peered":"2024-09-18T11:53:58.641087+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:53:58.641087+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14105211,"num_objects":1722,"num_object_clones":0,"num_object_copies":6888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-324}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11258","reported_seq":11383,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:58.585033+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:53:58.585033+0000","last_peered":"2024-09-18T11:53:58.585033+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:53:58.585033+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15004918,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11256,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-291}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11072","reported_seq":15361,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:53:55.162368+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:53:55.162368+0000","last_peered":"2024-09-18T11:53:55.162368+0000","last_clean":"2024-09-18T11:53:55.162368+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:53:55.162368+0000","last_undegraded":"2024-09-18T11:53:55.162368+0000","last_fullsized":"2024-09-18T11:53:55.162368+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14996726,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11070,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10926","reported_seq":11016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.473316+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:59.473316+0000","last_peered":"2024-09-18T11:53:59.473316+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:59.473316+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14587126,"num_objects":1781,"num_object_clones":0,"num_object_copies":7124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":232179084,"num_objects":28348,"num_object_clones":0,"num_object_copies":113392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28348,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178739,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7472,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":465973248,"data_stored":465973248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699660,"kb_used_data":113412,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93672180,"statfs":{"total":96636764160,"available":95920312320,"internally_reserved":0,"allocated":116133888,"data_stored":115950067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":3.27},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":5.0880000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.3049999999999999},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":1.264}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":6.1130000000000004},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":1.236}]}]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707996,"kb_used_data":113936,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93663844,"statfs":{"total":96636764160,"available":95911776256,"internally_reserved":0,"allocated":116670464,"data_stored":116486643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":0.24299999999999999},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":4.1890000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:52:58 2024","interfaces":[{"interface":"back","average":{"1min":7.5700000000000003,"5min":7.0960000000000001,"15min":6.4029999999999996},"min":{"1min":0.33700000000000002,"5min":0.216,"15min":0.216},"max":{"1min":24,"5min":26.530000000000001,"15min":26.530000000000001},"last":14.367000000000001},{"interface":"front","average":{"1min":5.468,"5min":4.9550000000000001,"15min":4.125},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":26.545000000000002,"5min":26.545000000000002,"15min":26.545000000000002},"last":4.1609999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":17.402000000000001},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":4.6079999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715460,"kb_used_data":116788,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93656380,"statfs":{"total":96636764160,"available":95904133120,"internally_reserved":0,"allocated":119590912,"data_stored":119407091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":7.1779999999999999},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":12.771000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":2.7530000000000001},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":7.2130000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":0.375},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":2.8839999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707724,"kb_used_data":113220,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93664116,"statfs":{"total":96636764160,"available":95912054784,"internally_reserved":0,"allocated":115937280,"data_stored":115753459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,2,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":3.996},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":4.9240000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":3.4039999999999999},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.1269999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":3.8969999999999998},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":8.0050000000000008}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115175424,"data_stored":115175424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119132160,"data_stored":119132160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":115855360,"data_stored":115855360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":115810304,"data_stored":115810304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:03.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:04.219 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:04.240 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-18T11:54:04.056281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":225904012,"num_objects":27582,"num_object_clones":0,"num_object_copies":110328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179505,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7472,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815352,"kb_used_data":446016,"kb_used_omap":14,"kb_used_meta":2369265,"kb_avail":374672008,"statfs":{"total":386547056640,"available":383664136192,"internally_reserved":0,"allocated":456720384,"data_stored":455985100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2426127387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,8,6,8,12,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21897216,"num_objects":-2673,"num_object_clones":0,"num_object_copies":-10692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2673,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004738"},"pg_stats":[{"pgid":"1.f","version":"37'11514","reported_seq":11615,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.531209+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:03.531209+0000","last_peered":"2024-09-18T11:54:03.531209+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:03.531209+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":1736,"num_object_clones":0,"num_object_copies":6944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11295","reported_seq":15587,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:00.171236+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:00.171236+0000","last_peered":"2024-09-18T11:54:00.171236+0000","last_clean":"2024-09-18T11:54:00.171236+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:00.171236+0000","last_undegraded":"2024-09-18T11:54:00.171236+0000","last_fullsized":"2024-09-18T11:54:00.171236+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14423286,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11293,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11107","reported_seq":11224,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.624382+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:03.624382+0000","last_peered":"2024-09-18T11:54:03.624382+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:03.624382+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13933203,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11106,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11167","reported_seq":11261,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.442738+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:59.442738+0000","last_peered":"2024-09-18T11:53:59.442738+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:59.442738+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14406902,"num_objects":1759,"num_object_clones":0,"num_object_copies":7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11174","reported_seq":11295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.619042+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:03.619042+0000","last_peered":"2024-09-18T11:54:03.619042+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:03.619042+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14095606,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11154","reported_seq":11273,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.756760+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:03.756760+0000","last_peered":"2024-09-18T11:54:03.756760+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:03.756760+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13369344,"num_objects":1632,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11154,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11270","reported_seq":11377,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.421717+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:59.421717+0000","last_peered":"2024-09-18T11:53:59.421717+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:59.421717+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14300406,"num_objects":1746,"num_object_clones":0,"num_object_copies":6984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11268,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-370}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11075","reported_seq":12851,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.471853+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:59.471853+0000","last_peered":"2024-09-18T11:53:59.471853+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:59.471853+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14742774,"num_objects":1800,"num_object_clones":0,"num_object_copies":7200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11073,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":855,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11466","reported_seq":15819,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.756366+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:03.756366+0000","last_peered":"2024-09-18T11:54:03.756366+0000","last_clean":"2024-09-18T11:54:03.756366+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:03.756366+0000","last_undegraded":"2024-09-18T11:54:03.756366+0000","last_fullsized":"2024-09-18T11:54:03.756366+0000","mapping_epoch":36,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14002668,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11463,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10951","reported_seq":11065,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.609778+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:03.609778+0000","last_peered":"2024-09-18T11:54:03.609778+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:03.609778+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13456630,"num_objects":1643,"num_object_clones":0,"num_object_copies":6572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-320}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11290","reported_seq":11411,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.329596+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:03.329596+0000","last_peered":"2024-09-18T11:54:03.329596+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:03.329596+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14274417,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11377","reported_seq":11470,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.655056+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:03.655056+0000","last_peered":"2024-09-18T11:54:03.655056+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:03.655056+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13548155,"num_objects":1654,"num_object_clones":0,"num_object_copies":6616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-430}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11299","reported_seq":11397,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.417453+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:03.417453+0000","last_peered":"2024-09-18T11:54:03.417453+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:03.417453+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13662843,"num_objects":1668,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-378}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11326","reported_seq":11451,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.620745+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:03.620745+0000","last_peered":"2024-09-18T11:54:03.620745+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:03.620745+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14447862,"num_objects":1764,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11324,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-359}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11141","reported_seq":15430,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:00.352596+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:00.352596+0000","last_peered":"2024-09-18T11:54:00.352596+0000","last_clean":"2024-09-18T11:54:00.352596+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:00.352596+0000","last_undegraded":"2024-09-18T11:54:00.352596+0000","last_fullsized":"2024-09-18T11:54:00.352596+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431478,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10926","reported_seq":11016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.473316+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:59.473316+0000","last_peered":"2024-09-18T11:53:59.473316+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:59.473316+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14587126,"num_objects":1781,"num_object_clones":0,"num_object_copies":7124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":225904012,"num_objects":27582,"num_object_clones":0,"num_object_copies":110328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179505,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7472,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":453373952,"data_stored":453373952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700496,"kb_used_data":110088,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93671344,"statfs":{"total":96636764160,"available":95919456256,"internally_reserved":0,"allocated":112730112,"data_stored":112546291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":21.527999999999999},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":7.319}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":7.3470000000000004},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":3.6749999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":9.3480000000000008},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":3.706}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708228,"kb_used_data":109988,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93663612,"statfs":{"total":96636764160,"available":95911538688,"internally_reserved":0,"allocated":112627712,"data_stored":112443891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":10.77},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":10.804}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":4.6020000000000003},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":10.834}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":10.856999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":1.897}]}]},{"osd":1,"up_from":29,"seq":124554051695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698904,"kb_used_data":112720,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93672936,"statfs":{"total":96636764160,"available":95921086464,"internally_reserved":0,"allocated":115425280,"data_stored":115241459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":5.4800000000000004},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":27.09}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":26.802},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.4450000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":27.032},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":26.984999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707724,"kb_used_data":113220,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93664116,"statfs":{"total":96636764160,"available":95912054784,"internally_reserved":0,"allocated":115937280,"data_stored":115753459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,2,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":3.996},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":4.9240000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":3.4039999999999999},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.1269999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":3.8969999999999998},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":8.0050000000000008}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115175424,"data_stored":115175424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114896896,"data_stored":114896896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":111706112,"data_stored":111706112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":111595520,"data_stored":111595520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:04.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:04.802 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:04.803 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:04.830 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-18T11:54:04.056281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":225904012,"num_objects":27582,"num_object_clones":0,"num_object_copies":110328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179505,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7472,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815352,"kb_used_data":446016,"kb_used_omap":14,"kb_used_meta":2369265,"kb_avail":374672008,"statfs":{"total":386547056640,"available":383664136192,"internally_reserved":0,"allocated":456720384,"data_stored":455985100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2426127387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,8,6,8,12,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21897216,"num_objects":-2673,"num_object_clones":0,"num_object_copies":-10692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2673,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004738"},"pg_stats":[{"pgid":"1.f","version":"37'11514","reported_seq":11615,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.531209+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:03.531209+0000","last_peered":"2024-09-18T11:54:03.531209+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:03.531209+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":1736,"num_object_clones":0,"num_object_copies":6944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11295","reported_seq":15587,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:00.171236+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:00.171236+0000","last_peered":"2024-09-18T11:54:00.171236+0000","last_clean":"2024-09-18T11:54:00.171236+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:00.171236+0000","last_undegraded":"2024-09-18T11:54:00.171236+0000","last_fullsized":"2024-09-18T11:54:00.171236+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14423286,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11293,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11107","reported_seq":11224,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.624382+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:03.624382+0000","last_peered":"2024-09-18T11:54:03.624382+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:03.624382+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13933203,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11106,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11167","reported_seq":11261,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.442738+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:53:59.442738+0000","last_peered":"2024-09-18T11:53:59.442738+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:53:59.442738+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14406902,"num_objects":1759,"num_object_clones":0,"num_object_copies":7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11174","reported_seq":11295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.619042+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:03.619042+0000","last_peered":"2024-09-18T11:54:03.619042+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:03.619042+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14095606,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11154","reported_seq":11273,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.756760+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:03.756760+0000","last_peered":"2024-09-18T11:54:03.756760+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:03.756760+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13369344,"num_objects":1632,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11154,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11270","reported_seq":11377,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.421717+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:53:59.421717+0000","last_peered":"2024-09-18T11:53:59.421717+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:53:59.421717+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14300406,"num_objects":1746,"num_object_clones":0,"num_object_copies":6984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11268,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-370}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11075","reported_seq":12851,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.471853+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:53:59.471853+0000","last_peered":"2024-09-18T11:53:59.471853+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:53:59.471853+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14742774,"num_objects":1800,"num_object_clones":0,"num_object_copies":7200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11073,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":855,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11466","reported_seq":15819,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.756366+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:03.756366+0000","last_peered":"2024-09-18T11:54:03.756366+0000","last_clean":"2024-09-18T11:54:03.756366+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:03.756366+0000","last_undegraded":"2024-09-18T11:54:03.756366+0000","last_fullsized":"2024-09-18T11:54:03.756366+0000","mapping_epoch":36,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14002668,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11463,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10951","reported_seq":11065,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.609778+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:03.609778+0000","last_peered":"2024-09-18T11:54:03.609778+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:03.609778+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13456630,"num_objects":1643,"num_object_clones":0,"num_object_copies":6572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-320}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11290","reported_seq":11411,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.329596+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:03.329596+0000","last_peered":"2024-09-18T11:54:03.329596+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:03.329596+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14274417,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11377","reported_seq":11470,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.655056+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:03.655056+0000","last_peered":"2024-09-18T11:54:03.655056+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:03.655056+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13548155,"num_objects":1654,"num_object_clones":0,"num_object_copies":6616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-430}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11299","reported_seq":11397,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.417453+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:03.417453+0000","last_peered":"2024-09-18T11:54:03.417453+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:03.417453+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13662843,"num_objects":1668,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-378}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11326","reported_seq":11451,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.620745+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:03.620745+0000","last_peered":"2024-09-18T11:54:03.620745+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:03.620745+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14447862,"num_objects":1764,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11324,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-359}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11141","reported_seq":15430,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:00.352596+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:00.352596+0000","last_peered":"2024-09-18T11:54:00.352596+0000","last_clean":"2024-09-18T11:54:00.352596+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:00.352596+0000","last_undegraded":"2024-09-18T11:54:00.352596+0000","last_fullsized":"2024-09-18T11:54:00.352596+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431478,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10926","reported_seq":11016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:53:59.473316+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:53:59.473316+0000","last_peered":"2024-09-18T11:53:59.473316+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:53:59.473316+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14587126,"num_objects":1781,"num_object_clones":0,"num_object_copies":7124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":225904012,"num_objects":27582,"num_object_clones":0,"num_object_copies":110328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179505,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7472,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":453373952,"data_stored":453373952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700496,"kb_used_data":110088,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93671344,"statfs":{"total":96636764160,"available":95919456256,"internally_reserved":0,"allocated":112730112,"data_stored":112546291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":21.527999999999999},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":7.319}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":7.3470000000000004},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":3.6749999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":9.3480000000000008},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":3.706}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708228,"kb_used_data":109988,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93663612,"statfs":{"total":96636764160,"available":95911538688,"internally_reserved":0,"allocated":112627712,"data_stored":112443891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":10.77},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":10.804}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":4.6020000000000003},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":10.834}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":10.856999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":1.897}]}]},{"osd":1,"up_from":29,"seq":124554051695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698904,"kb_used_data":112720,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93672936,"statfs":{"total":96636764160,"available":95921086464,"internally_reserved":0,"allocated":115425280,"data_stored":115241459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":5.944,"15min":4.3940000000000001},"min":{"1min":0.443,"5min":0.24299999999999999,"15min":0.159},"max":{"1min":28.128,"5min":32.371000000000002,"15min":45.462000000000003},"last":5.4800000000000004},{"interface":"front","average":{"1min":10.472,"5min":6.5579999999999998,"15min":5.4710000000000001},"min":{"1min":0.27600000000000002,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":54.710999999999999,"5min":54.710999999999999,"15min":54.710999999999999},"last":27.09}]},{"osd":2,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":9.0980000000000008,"5min":6.7930000000000001,"15min":5.1260000000000003},"min":{"1min":0.39400000000000002,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":27.425000000000001,"5min":31.306000000000001,"15min":31.306000000000001},"last":26.802},{"interface":"front","average":{"1min":8.7780000000000005,"5min":5.3360000000000003,"15min":4.431},"min":{"1min":0.40899999999999997,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":28.094999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.4450000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:53:04 2024","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":5.6459999999999999,"15min":4.625},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":18.765999999999998,"5min":20.373999999999999,"15min":20.373999999999999},"last":27.032},{"interface":"front","average":{"1min":7.2699999999999996,"5min":6.79,"15min":5.4640000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":25.158000000000001,"5min":25.158000000000001,"15min":25.440000000000001},"last":26.984999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707724,"kb_used_data":113220,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93664116,"statfs":{"total":96636764160,"available":95912054784,"internally_reserved":0,"allocated":115937280,"data_stored":115753459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,2,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:52:59 2024","interfaces":[{"interface":"back","average":{"1min":10.148999999999999,"5min":6.8449999999999998,"15min":4.8090000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.14399999999999999},"max":{"1min":27.419,"5min":28.059000000000001,"15min":28.059000000000001},"last":3.996},{"interface":"front","average":{"1min":6.218,"5min":5.6239999999999997,"15min":4.4950000000000001},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":14.849,"5min":34.786999999999999,"15min":34.786999999999999},"last":4.9240000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":3.4039999999999999},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.1269999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":3.8969999999999998},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":8.0050000000000008}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115175424,"data_stored":115175424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114896896,"data_stored":114896896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":111706112,"data_stored":111706112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":111595520,"data_stored":111595520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:04.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:54:04.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11514", 'reported_seq': 11615, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:03.531209+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:03.531209+0000', 'last_peered': '2024-09-18T11:54:03.531209+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:03.531209+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14221312, 'num_objects': 1736, 'num_object_clones': 0, 'num_object_copies': 6944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11514, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:04.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:54:04.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11107", 'reported_seq': 11224, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:03.624382+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:03.624382+0000', 'last_peered': '2024-09-18T11:54:03.624382+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:03.624382+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13933203, 'num_objects': 1702, 'num_object_clones': 0, 'num_object_copies': 6808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1702, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11106, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -338}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:04.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:54:04.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11167", 'reported_seq': 11261, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:59.442738+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:53:59.442738+0000', 'last_peered': '2024-09-18T11:53:59.442738+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:53:59.442738+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14406902, 'num_objects': 1759, 'num_object_clones': 0, 'num_object_copies': 7036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11165, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -342}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:04.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:54:04.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11174", 'reported_seq': 11295, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:03.619042+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:03.619042+0000', 'last_peered': '2024-09-18T11:54:03.619042+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:03.619042+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14095606, 'num_objects': 1721, 'num_object_clones': 0, 'num_object_copies': 6884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11172, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -331}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:04.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:54:04.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11154", 'reported_seq': 11273, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:03.756760+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:03.756760+0000', 'last_peered': '2024-09-18T11:54:03.756760+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:03.756760+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13369344, 'num_objects': 1632, 'num_object_clones': 0, 'num_object_copies': 6528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1632, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11154, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -487}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:04.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:54:04.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11270", 'reported_seq': 11377, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:59.421717+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:53:59.421717+0000', 'last_peered': '2024-09-18T11:53:59.421717+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:53:59.421717+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14300406, 'num_objects': 1746, 'num_object_clones': 0, 'num_object_copies': 6984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1746, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11268, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -370}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:04.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:54:04.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11075", 'reported_seq': 12851, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:59.471853+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:53:59.471853+0000', 'last_peered': '2024-09-18T11:53:59.471853+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:53:59.471853+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14742774, 'num_objects': 1800, 'num_object_clones': 0, 'num_object_copies': 7200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1183, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11073, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 855, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:04.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10951", 'reported_seq': 11065, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:03.609778+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:03.609778+0000', 'last_peered': '2024-09-18T11:54:03.609778+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:03.609778+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13456630, 'num_objects': 1643, 'num_object_clones': 0, 'num_object_copies': 6572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10949, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -320}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11290", 'reported_seq': 11411, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:03.329596+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:03.329596+0000', 'last_peered': '2024-09-18T11:54:03.329596+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:03.329596+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14274417, 'num_objects': 1743, 'num_object_clones': 0, 'num_object_copies': 6972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11287, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11377", 'reported_seq': 11470, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:03.655056+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:03.655056+0000', 'last_peered': '2024-09-18T11:54:03.655056+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:03.655056+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13548155, 'num_objects': 1654, 'num_object_clones': 0, 'num_object_copies': 6616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1654, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11376, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -430}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11299", 'reported_seq': 11397, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:03.417453+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:03.417453+0000', 'last_peered': '2024-09-18T11:54:03.417453+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:03.417453+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13662843, 'num_objects': 1668, 'num_object_clones': 0, 'num_object_copies': 6672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11298, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -378}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11326", 'reported_seq': 11451, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:03.620745+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:03.620745+0000', 'last_peered': '2024-09-18T11:54:03.620745+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:03.620745+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14447862, 'num_objects': 1764, 'num_object_clones': 0, 'num_object_copies': 7056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11324, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -359}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:54:04.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10926", 'reported_seq': 11016, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:53:59.473316+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:53:59.473316+0000', 'last_peered': '2024-09-18T11:53:59.473316+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:53:59.473316+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14587126, 'num_objects': 1781, 'num_object_clones': 0, 'num_object_copies': 7124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10924, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -221}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:04.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:05.547 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:05.547 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25984/110328 objects degraded (23.552%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":643,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":27582,"data_bytes":225904012,"bytes_used":2882920448,"bytes_avail":383664136192,"bytes_total":386547056640,"degraded_objects":25984,"degraded_total":110328,"degraded_ratio":0.23551591617721704,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":222},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:05.569 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:05.569 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:06.133 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:06.133 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:06.150 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-09-18T11:54:06.056789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222684556,"num_objects":27189,"num_object_clones":0,"num_object_copies":108756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179898,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7517,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811280,"kb_used_data":437776,"kb_used_omap":14,"kb_used_meta":2373425,"kb_avail":374676080,"statfs":{"total":386547056640,"available":383668305920,"internally_reserved":0,"allocated":448282624,"data_stored":447547340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430387227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,4,7,11,13,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19890176,"num_objects":-2428,"num_object_clones":0,"num_object_copies":-9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2428,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004789"},"pg_stats":[{"pgid":"1.f","version":"37'11514","reported_seq":11615,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.531209+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:03.531209+0000","last_peered":"2024-09-18T11:54:03.531209+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:03.531209+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":1736,"num_object_clones":0,"num_object_copies":6944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11354","reported_seq":15646,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:05.385853+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:05.385853+0000","last_peered":"2024-09-18T11:54:05.385853+0000","last_clean":"2024-09-18T11:54:05.385853+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:05.385853+0000","last_undegraded":"2024-09-18T11:54:05.385853+0000","last_fullsized":"2024-09-18T11:54:05.385853+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939958,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11352,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11107","reported_seq":11224,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.624382+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:03.624382+0000","last_peered":"2024-09-18T11:54:03.624382+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:03.624382+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13933203,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11106,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11227","reported_seq":11316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:04.478694+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:04.478694+0000","last_peered":"2024-09-18T11:54:04.478694+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:04.478694+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13915382,"num_objects":1699,"num_object_clones":0,"num_object_copies":6796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11225,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11174","reported_seq":11295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.619042+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:03.619042+0000","last_peered":"2024-09-18T11:54:03.619042+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:03.619042+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14095606,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11154","reported_seq":11273,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.756760+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:03.756760+0000","last_peered":"2024-09-18T11:54:03.756760+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:03.756760+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13369344,"num_objects":1632,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11154,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11337","reported_seq":11444,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:04.401323+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:04.401323+0000","last_peered":"2024-09-18T11:54:04.401323+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:04.401323+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13751542,"num_objects":1679,"num_object_clones":0,"num_object_copies":6716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11335,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-437}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11145","reported_seq":13011,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:04.457156+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:04.457156+0000","last_peered":"2024-09-18T11:54:04.457156+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:04.457156+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14169334,"num_objects":1730,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11143,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":900,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11466","reported_seq":15819,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.756366+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:03.756366+0000","last_peered":"2024-09-18T11:54:03.756366+0000","last_clean":"2024-09-18T11:54:03.756366+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:03.756366+0000","last_undegraded":"2024-09-18T11:54:03.756366+0000","last_fullsized":"2024-09-18T11:54:03.756366+0000","mapping_epoch":36,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14002668,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11463,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10951","reported_seq":11065,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.609778+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:03.609778+0000","last_peered":"2024-09-18T11:54:03.609778+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:03.609778+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13456630,"num_objects":1643,"num_object_clones":0,"num_object_copies":6572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-320}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11290","reported_seq":11411,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.329596+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:03.329596+0000","last_peered":"2024-09-18T11:54:03.329596+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:03.329596+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14274417,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11377","reported_seq":11470,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.655056+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:03.655056+0000","last_peered":"2024-09-18T11:54:03.655056+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:03.655056+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13548155,"num_objects":1654,"num_object_clones":0,"num_object_copies":6616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-430}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11299","reported_seq":11397,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.417453+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:03.417453+0000","last_peered":"2024-09-18T11:54:03.417453+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:03.417453+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13662843,"num_objects":1668,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-378}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11326","reported_seq":11451,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.620745+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:03.620745+0000","last_peered":"2024-09-18T11:54:03.620745+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:03.620745+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14447862,"num_objects":1764,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11324,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-359}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11218","reported_seq":15505,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:05.445910+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:05.445910+0000","last_peered":"2024-09-18T11:54:05.445910+0000","last_clean":"2024-09-18T11:54:05.445910+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:05.445910+0000","last_undegraded":"2024-09-18T11:54:05.445910+0000","last_fullsized":"2024-09-18T11:54:05.445910+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13800694,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11216,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10986","reported_seq":11076,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:04.391381+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:04.391381+0000","last_peered":"2024-09-18T11:54:04.391381+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:04.391381+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14095606,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10984,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":222684556,"num_objects":27189,"num_object_clones":0,"num_object_copies":108756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179898,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7517,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":444903424,"data_stored":444903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700496,"kb_used_data":110088,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93671344,"statfs":{"total":96636764160,"available":95919456256,"internally_reserved":0,"allocated":112730112,"data_stored":112546291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":21.527999999999999},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":7.319}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":7.3470000000000004},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":3.6749999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":9.3480000000000008},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":3.706}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708228,"kb_used_data":109988,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93663612,"statfs":{"total":96636764160,"available":95911538688,"internally_reserved":0,"allocated":112627712,"data_stored":112443891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":10.77},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":10.804}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":4.6020000000000003},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":10.834}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":10.856999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":1.897}]}]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698932,"kb_used_data":108580,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93672908,"statfs":{"total":96636764160,"available":95921057792,"internally_reserved":0,"allocated":111185920,"data_stored":111002099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,4,5,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.34000000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.32900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.59199999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703624,"kb_used_data":109120,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93668216,"statfs":{"total":96636764160,"available":95916253184,"internally_reserved":0,"allocated":111738880,"data_stored":111555059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.153},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.78300000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":1.1870000000000001},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.35699999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":10.586}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110997504,"data_stored":110997504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110604288,"data_stored":110604288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":111706112,"data_stored":111706112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":111595520,"data_stored":111595520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:09.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:09.712 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:09.713 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:09.742 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-18T11:54:08.057284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222684556,"num_objects":27189,"num_object_clones":0,"num_object_copies":108756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179898,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7517,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811280,"kb_used_data":437776,"kb_used_omap":14,"kb_used_meta":2373425,"kb_avail":374676080,"statfs":{"total":386547056640,"available":383668305920,"internally_reserved":0,"allocated":448282624,"data_stored":447547340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430387227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,4,7,11,13,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16900096,"num_objects":-2063,"num_object_clones":0,"num_object_copies":-8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2063,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004762"},"pg_stats":[{"pgid":"1.f","version":"37'11514","reported_seq":11615,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.531209+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:03.531209+0000","last_peered":"2024-09-18T11:54:03.531209+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:03.531209+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":1736,"num_object_clones":0,"num_object_copies":6944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11354","reported_seq":15646,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:05.385853+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:05.385853+0000","last_peered":"2024-09-18T11:54:05.385853+0000","last_clean":"2024-09-18T11:54:05.385853+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:05.385853+0000","last_undegraded":"2024-09-18T11:54:05.385853+0000","last_fullsized":"2024-09-18T11:54:05.385853+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939958,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11352,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11107","reported_seq":11224,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.624382+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:03.624382+0000","last_peered":"2024-09-18T11:54:03.624382+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:03.624382+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13933203,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11106,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11227","reported_seq":11316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:04.478694+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:04.478694+0000","last_peered":"2024-09-18T11:54:04.478694+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:04.478694+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13915382,"num_objects":1699,"num_object_clones":0,"num_object_copies":6796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11225,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11174","reported_seq":11295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.619042+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:03.619042+0000","last_peered":"2024-09-18T11:54:03.619042+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:03.619042+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14095606,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11154","reported_seq":11273,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.756760+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:03.756760+0000","last_peered":"2024-09-18T11:54:03.756760+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:03.756760+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13369344,"num_objects":1632,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11154,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11337","reported_seq":11444,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:04.401323+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:04.401323+0000","last_peered":"2024-09-18T11:54:04.401323+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:04.401323+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13751542,"num_objects":1679,"num_object_clones":0,"num_object_copies":6716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11335,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-437}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11145","reported_seq":13011,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:04.457156+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:04.457156+0000","last_peered":"2024-09-18T11:54:04.457156+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:04.457156+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14169334,"num_objects":1730,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11143,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":900,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11466","reported_seq":15819,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.756366+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:03.756366+0000","last_peered":"2024-09-18T11:54:03.756366+0000","last_clean":"2024-09-18T11:54:03.756366+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:03.756366+0000","last_undegraded":"2024-09-18T11:54:03.756366+0000","last_fullsized":"2024-09-18T11:54:03.756366+0000","mapping_epoch":36,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14002668,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11463,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10951","reported_seq":11065,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.609778+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:03.609778+0000","last_peered":"2024-09-18T11:54:03.609778+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:03.609778+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13456630,"num_objects":1643,"num_object_clones":0,"num_object_copies":6572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-320}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11290","reported_seq":11411,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.329596+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:03.329596+0000","last_peered":"2024-09-18T11:54:03.329596+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:03.329596+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14274417,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11377","reported_seq":11470,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.655056+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:03.655056+0000","last_peered":"2024-09-18T11:54:03.655056+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:03.655056+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13548155,"num_objects":1654,"num_object_clones":0,"num_object_copies":6616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-430}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11299","reported_seq":11397,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.417453+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:03.417453+0000","last_peered":"2024-09-18T11:54:03.417453+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:03.417453+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13662843,"num_objects":1668,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-378}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11326","reported_seq":11451,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:03.620745+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:03.620745+0000","last_peered":"2024-09-18T11:54:03.620745+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:03.620745+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14447862,"num_objects":1764,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11324,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-359}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11218","reported_seq":15505,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:05.445910+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:05.445910+0000","last_peered":"2024-09-18T11:54:05.445910+0000","last_clean":"2024-09-18T11:54:05.445910+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:05.445910+0000","last_undegraded":"2024-09-18T11:54:05.445910+0000","last_fullsized":"2024-09-18T11:54:05.445910+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13800694,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11216,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'10986","reported_seq":11076,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:04.391381+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:04.391381+0000","last_peered":"2024-09-18T11:54:04.391381+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:04.391381+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14095606,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10984,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":222684556,"num_objects":27189,"num_object_clones":0,"num_object_copies":108756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179898,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7517,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":444903424,"data_stored":444903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700496,"kb_used_data":110088,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93671344,"statfs":{"total":96636764160,"available":95919456256,"internally_reserved":0,"allocated":112730112,"data_stored":112546291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":21.527999999999999},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":7.319}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":7.3470000000000004},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":3.6749999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":9.3480000000000008},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":3.706}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708228,"kb_used_data":109988,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93663612,"statfs":{"total":96636764160,"available":95911538688,"internally_reserved":0,"allocated":112627712,"data_stored":112443891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":10.77},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":10.804}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":4.6020000000000003},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":10.834}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":10.856999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":1.897}]}]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698932,"kb_used_data":108580,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93672908,"statfs":{"total":96636764160,"available":95921057792,"internally_reserved":0,"allocated":111185920,"data_stored":111002099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,4,5,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.34000000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.32900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.59199999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703624,"kb_used_data":109120,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93668216,"statfs":{"total":96636764160,"available":95916253184,"internally_reserved":0,"allocated":111738880,"data_stored":111555059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.153},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.78300000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":1.1870000000000001},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.35699999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":10.586}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110997504,"data_stored":110997504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110604288,"data_stored":110604288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":111706112,"data_stored":111706112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":111595520,"data_stored":111595520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:09.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:10.327 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:10.328 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:10.350 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-18T11:54:10.057717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215631244,"num_objects":26328,"num_object_clones":0,"num_object_copies":105312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180759,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7579,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2794288,"kb_used_data":424960,"kb_used_omap":14,"kb_used_meta":2369265,"kb_avail":374693072,"statfs":{"total":386547056640,"available":383685705728,"internally_reserved":0,"allocated":435159040,"data_stored":434423756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2426127387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,12,16,12,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23953408,"num_objects":-2924,"num_object_clones":0,"num_object_copies":-11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004738"},"pg_stats":[{"pgid":"1.f","version":"37'11573","reported_seq":11674,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.740971+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:08.740971+0000","last_peered":"2024-09-18T11:54:08.740971+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:08.740971+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13737984,"num_objects":1677,"num_object_clones":0,"num_object_copies":6708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11573,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-426}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11354","reported_seq":15646,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:05.385853+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:05.385853+0000","last_peered":"2024-09-18T11:54:05.385853+0000","last_clean":"2024-09-18T11:54:05.385853+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:05.385853+0000","last_undegraded":"2024-09-18T11:54:05.385853+0000","last_fullsized":"2024-09-18T11:54:05.385853+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939958,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11352,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11172","reported_seq":11288,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.612266+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:08.612266+0000","last_peered":"2024-09-18T11:54:08.612266+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:08.612266+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13400723,"num_objects":1637,"num_object_clones":0,"num_object_copies":6548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11171,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-403}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11290","reported_seq":11384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.404069+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:09.404069+0000","last_peered":"2024-09-18T11:54:09.404069+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:09.404069+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13399286,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11247","reported_seq":11368,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.744381+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:08.744381+0000","last_peered":"2024-09-18T11:54:08.744381+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:08.744381+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13497590,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11245,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11225","reported_seq":11344,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.821280+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:08.821280+0000","last_peered":"2024-09-18T11:54:08.821280+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:08.821280+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12787712,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11225,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11399","reported_seq":11506,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.366827+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:09.366827+0000","last_peered":"2024-09-18T11:54:09.366827+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:09.366827+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13243638,"num_objects":1617,"num_object_clones":0,"num_object_copies":6468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-499}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11195","reported_seq":13185,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.522150+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:09.522150+0000","last_peered":"2024-09-18T11:54:09.522150+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:09.522150+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13759734,"num_objects":1680,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11529","reported_seq":15882,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:08.822688+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:08.822688+0000","last_peered":"2024-09-18T11:54:08.822688+0000","last_clean":"2024-09-18T11:54:08.822688+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:08.822688+0000","last_undegraded":"2024-09-18T11:54:08.822688+0000","last_fullsized":"2024-09-18T11:54:08.822688+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13486572,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11526,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10993","reported_seq":11107,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.404178+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:08.404178+0000","last_peered":"2024-09-18T11:54:08.404178+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:08.404178+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13112566,"num_objects":1601,"num_object_clones":0,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10991,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11347","reported_seq":11468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.665437+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:08.665437+0000","last_peered":"2024-09-18T11:54:08.665437+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:08.665437+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13807473,"num_objects":1686,"num_object_clones":0,"num_object_copies":6744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11344,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-348}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11436","reported_seq":11529,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.598218+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:08.598218+0000","last_peered":"2024-09-18T11:54:08.598218+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:08.598218+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13064827,"num_objects":1595,"num_object_clones":0,"num_object_copies":6380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11435,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11366","reported_seq":11464,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.503755+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:08.503755+0000","last_peered":"2024-09-18T11:54:08.503755+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:08.503755+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13113979,"num_objects":1601,"num_object_clones":0,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11365,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11397","reported_seq":11521,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.794218+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:08.794218+0000","last_peered":"2024-09-18T11:54:08.794218+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:08.794218+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13866230,"num_objects":1693,"num_object_clones":0,"num_object_copies":6772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-430}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11218","reported_seq":15505,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:05.445910+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:05.445910+0000","last_peered":"2024-09-18T11:54:05.445910+0000","last_clean":"2024-09-18T11:54:05.445910+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:05.445910+0000","last_undegraded":"2024-09-18T11:54:05.445910+0000","last_fullsized":"2024-09-18T11:54:05.445910+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13800694,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11216,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11045","reported_seq":11135,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.515906+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:09.515906+0000","last_peered":"2024-09-18T11:54:09.515906+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:09.515906+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13612278,"num_objects":1662,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":215631244,"num_objects":26328,"num_object_clones":0,"num_object_copies":105312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180759,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7579,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":432414720,"data_stored":432414720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699872,"kb_used_data":105304,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93671968,"statfs":{"total":96636764160,"available":95920095232,"internally_reserved":0,"allocated":107831296,"data_stored":107647475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":7.6200000000000001},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":7.5899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":7.6479999999999997},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":1.859}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.7000000000000002},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":7.9779999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691668,"kb_used_data":105924,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93680172,"statfs":{"total":96636764160,"available":95928496128,"internally_reserved":0,"allocated":108466176,"data_stored":108282355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":11.602},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":11.640000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":20.073},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":2.036}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.008},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.2570000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698932,"kb_used_data":108580,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93672908,"statfs":{"total":96636764160,"available":95921057792,"internally_reserved":0,"allocated":111185920,"data_stored":111002099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,4,5,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.34000000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.32900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.59199999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703816,"kb_used_data":105152,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93668024,"statfs":{"total":96636764160,"available":95916056576,"internally_reserved":0,"allocated":107675648,"data_stored":107491827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":18.478999999999999},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.911}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":13.265000000000001},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":16.981999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.6509999999999998},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":27.015999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106799104,"data_stored":106799104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110604288,"data_stored":110604288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107593728,"data_stored":107593728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":107417600,"data_stored":107417600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:10.351 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:54:10.351 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11573", 'reported_seq': 11674, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:08.740971+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:08.740971+0000', 'last_peered': '2024-09-18T11:54:08.740971+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:08.740971+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13737984, 'num_objects': 1677, 'num_object_clones': 0, 'num_object_copies': 6708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11573, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -426}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:10.352 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:54:10.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11172", 'reported_seq': 11288, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:08.612266+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:08.612266+0000', 'last_peered': '2024-09-18T11:54:08.612266+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:08.612266+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13400723, 'num_objects': 1637, 'num_object_clones': 0, 'num_object_copies': 6548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1637, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11171, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -403}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:10.352 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:54:10.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11290", 'reported_seq': 11384, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:09.404069+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:54:09.404069+0000', 'last_peered': '2024-09-18T11:54:09.404069+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:54:09.404069+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13399286, 'num_objects': 1636, 'num_object_clones': 0, 'num_object_copies': 6544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11288, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -465}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:10.352 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:54:10.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11247", 'reported_seq': 11368, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:08.744381+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:08.744381+0000', 'last_peered': '2024-09-18T11:54:08.744381+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:08.744381+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13497590, 'num_objects': 1648, 'num_object_clones': 0, 'num_object_copies': 6592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1648, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11245, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -404}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:10.352 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:54:10.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11225", 'reported_seq': 11344, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:08.821280+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:08.821280+0000', 'last_peered': '2024-09-18T11:54:08.821280+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:08.821280+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12787712, 'num_objects': 1561, 'num_object_clones': 0, 'num_object_copies': 6244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11225, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -558}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:10.352 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:54:10.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11399", 'reported_seq': 11506, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:09.366827+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:09.366827+0000', 'last_peered': '2024-09-18T11:54:09.366827+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:09.366827+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13243638, 'num_objects': 1617, 'num_object_clones': 0, 'num_object_copies': 6468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11397, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -499}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:10.353 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:54:10.353 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11195", 'reported_seq': 13185, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:09.522150+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:54:09.522150+0000', 'last_peered': '2024-09-18T11:54:09.522150+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:54:09.522150+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13759734, 'num_objects': 1680, 'num_object_clones': 0, 'num_object_copies': 6720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11193, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 962, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:10.353 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:54:10.353 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10993", 'reported_seq': 11107, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:08.404178+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:08.404178+0000', 'last_peered': '2024-09-18T11:54:08.404178+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:08.404178+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13112566, 'num_objects': 1601, 'num_object_clones': 0, 'num_object_copies': 6404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10991, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -362}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:10.353 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:54:10.353 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11347", 'reported_seq': 11468, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:08.665437+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:08.665437+0000', 'last_peered': '2024-09-18T11:54:08.665437+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:08.665437+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13807473, 'num_objects': 1686, 'num_object_clones': 0, 'num_object_copies': 6744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11344, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -348}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:10.353 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:54:10.353 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11436", 'reported_seq': 11529, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:08.598218+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:08.598218+0000', 'last_peered': '2024-09-18T11:54:08.598218+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:08.598218+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13064827, 'num_objects': 1595, 'num_object_clones': 0, 'num_object_copies': 6380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11435, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:10.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:54:10.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11366", 'reported_seq': 11464, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:08.503755+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:08.503755+0000', 'last_peered': '2024-09-18T11:54:08.503755+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:08.503755+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13113979, 'num_objects': 1601, 'num_object_clones': 0, 'num_object_copies': 6404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11365, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:10.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:54:10.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11397", 'reported_seq': 11521, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:08.794218+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:08.794218+0000', 'last_peered': '2024-09-18T11:54:08.794218+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:08.794218+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13866230, 'num_objects': 1693, 'num_object_clones': 0, 'num_object_copies': 6772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11395, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -430}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:10.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:54:10.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11045", 'reported_seq': 11135, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:09.515906+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:09.515906+0000', 'last_peered': '2024-09-18T11:54:09.515906+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:09.515906+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13612278, 'num_objects': 1662, 'num_object_clones': 0, 'num_object_copies': 6648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11043, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:10.354 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:11.046 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:11.046 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25877/105312 objects degraded (24.572%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":649,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26328,"data_bytes":215631244,"bytes_used":2861350912,"bytes_avail":383685705728,"bytes_total":386547056640,"degraded_objects":25877,"degraded_total":105312,"degraded_ratio":0.24571748708599209,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":243},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:11.072 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:11.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:11.645 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:11.645 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:11.663 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-18T11:54:10.057717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215631244,"num_objects":26328,"num_object_clones":0,"num_object_copies":105312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180759,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7579,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2794288,"kb_used_data":424960,"kb_used_omap":14,"kb_used_meta":2369265,"kb_avail":374693072,"statfs":{"total":386547056640,"available":383685705728,"internally_reserved":0,"allocated":435159040,"data_stored":434423756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2426127387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,12,16,12,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23953408,"num_objects":-2924,"num_object_clones":0,"num_object_copies":-11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004738"},"pg_stats":[{"pgid":"1.f","version":"37'11573","reported_seq":11674,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.740971+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:08.740971+0000","last_peered":"2024-09-18T11:54:08.740971+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:08.740971+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13737984,"num_objects":1677,"num_object_clones":0,"num_object_copies":6708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11573,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-426}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11354","reported_seq":15646,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:05.385853+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:05.385853+0000","last_peered":"2024-09-18T11:54:05.385853+0000","last_clean":"2024-09-18T11:54:05.385853+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:05.385853+0000","last_undegraded":"2024-09-18T11:54:05.385853+0000","last_fullsized":"2024-09-18T11:54:05.385853+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939958,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11352,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11172","reported_seq":11288,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.612266+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:08.612266+0000","last_peered":"2024-09-18T11:54:08.612266+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:08.612266+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13400723,"num_objects":1637,"num_object_clones":0,"num_object_copies":6548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11171,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-403}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11290","reported_seq":11384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.404069+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:09.404069+0000","last_peered":"2024-09-18T11:54:09.404069+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:09.404069+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13399286,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11247","reported_seq":11368,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.744381+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:08.744381+0000","last_peered":"2024-09-18T11:54:08.744381+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:08.744381+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13497590,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11245,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11225","reported_seq":11344,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.821280+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:08.821280+0000","last_peered":"2024-09-18T11:54:08.821280+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:08.821280+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12787712,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11225,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11399","reported_seq":11506,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.366827+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:09.366827+0000","last_peered":"2024-09-18T11:54:09.366827+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:09.366827+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13243638,"num_objects":1617,"num_object_clones":0,"num_object_copies":6468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-499}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11195","reported_seq":13185,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.522150+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:09.522150+0000","last_peered":"2024-09-18T11:54:09.522150+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:09.522150+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13759734,"num_objects":1680,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11529","reported_seq":15882,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:08.822688+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:08.822688+0000","last_peered":"2024-09-18T11:54:08.822688+0000","last_clean":"2024-09-18T11:54:08.822688+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:08.822688+0000","last_undegraded":"2024-09-18T11:54:08.822688+0000","last_fullsized":"2024-09-18T11:54:08.822688+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13486572,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11526,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10993","reported_seq":11107,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.404178+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:08.404178+0000","last_peered":"2024-09-18T11:54:08.404178+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:08.404178+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13112566,"num_objects":1601,"num_object_clones":0,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10991,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11347","reported_seq":11468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.665437+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:08.665437+0000","last_peered":"2024-09-18T11:54:08.665437+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:08.665437+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13807473,"num_objects":1686,"num_object_clones":0,"num_object_copies":6744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11344,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-348}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11436","reported_seq":11529,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.598218+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:08.598218+0000","last_peered":"2024-09-18T11:54:08.598218+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:08.598218+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13064827,"num_objects":1595,"num_object_clones":0,"num_object_copies":6380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11435,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11366","reported_seq":11464,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.503755+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:08.503755+0000","last_peered":"2024-09-18T11:54:08.503755+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:08.503755+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13113979,"num_objects":1601,"num_object_clones":0,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11365,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11397","reported_seq":11521,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:08.794218+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:08.794218+0000","last_peered":"2024-09-18T11:54:08.794218+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:08.794218+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13866230,"num_objects":1693,"num_object_clones":0,"num_object_copies":6772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-430}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11218","reported_seq":15505,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:05.445910+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:05.445910+0000","last_peered":"2024-09-18T11:54:05.445910+0000","last_clean":"2024-09-18T11:54:05.445910+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:05.445910+0000","last_undegraded":"2024-09-18T11:54:05.445910+0000","last_fullsized":"2024-09-18T11:54:05.445910+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13800694,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11216,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11045","reported_seq":11135,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.515906+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:09.515906+0000","last_peered":"2024-09-18T11:54:09.515906+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:09.515906+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13612278,"num_objects":1662,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":215631244,"num_objects":26328,"num_object_clones":0,"num_object_copies":105312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180759,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7579,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":432414720,"data_stored":432414720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699872,"kb_used_data":105304,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93671968,"statfs":{"total":96636764160,"available":95920095232,"internally_reserved":0,"allocated":107831296,"data_stored":107647475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":7.6200000000000001},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":7.5899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":7.6479999999999997},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":1.859}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.7000000000000002},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":7.9779999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691668,"kb_used_data":105924,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93680172,"statfs":{"total":96636764160,"available":95928496128,"internally_reserved":0,"allocated":108466176,"data_stored":108282355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":11.602},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":11.640000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":20.073},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":2.036}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":2.008},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.2570000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698932,"kb_used_data":108580,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93672908,"statfs":{"total":96636764160,"available":95921057792,"internally_reserved":0,"allocated":111185920,"data_stored":111002099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,4,5,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.34000000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.32900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.59199999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703816,"kb_used_data":105152,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93668024,"statfs":{"total":96636764160,"available":95916056576,"internally_reserved":0,"allocated":107675648,"data_stored":107491827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":18.478999999999999},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.911}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":13.265000000000001},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":16.981999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.6509999999999998},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":27.015999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106799104,"data_stored":106799104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110604288,"data_stored":110604288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107593728,"data_stored":107593728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":107417600,"data_stored":107417600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:14.665 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:15.298 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:15.299 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:15.317 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-18T11:54:14.058730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":209388940,"num_objects":25566,"num_object_clones":0,"num_object_copies":102264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7579,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2794712,"kb_used_data":412884,"kb_used_omap":14,"kb_used_meta":2381745,"kb_avail":374692648,"statfs":{"total":386547056640,"available":383685271552,"internally_reserved":0,"allocated":422793216,"data_stored":422057932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2438906907},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,7,9,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21716992,"num_objects":-2651,"num_object_clones":0,"num_object_copies":-10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2651,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"1.f","version":"37'11640","reported_seq":11741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.781231+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:13.781231+0000","last_peered":"2024-09-18T11:54:13.781231+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:13.781231+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13189120,"num_objects":1610,"num_object_clones":0,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11640,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-493}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11416","reported_seq":15708,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:10.461854+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:10.461854+0000","last_peered":"2024-09-18T11:54:10.461854+0000","last_clean":"2024-09-18T11:54:10.461854+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:10.461854+0000","last_undegraded":"2024-09-18T11:54:10.461854+0000","last_fullsized":"2024-09-18T11:54:10.461854+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13432054,"num_objects":1640,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11230","reported_seq":11347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.252130+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:13.252130+0000","last_peered":"2024-09-18T11:54:13.252130+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:13.252130+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12925587,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11229,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11290","reported_seq":11384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.404069+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:09.404069+0000","last_peered":"2024-09-18T11:54:09.404069+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:09.404069+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13399286,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11307","reported_seq":11428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.824624+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:13.824624+0000","last_peered":"2024-09-18T11:54:13.824624+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:13.824624+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13006070,"num_objects":1588,"num_object_clones":0,"num_object_copies":6352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11305,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11289","reported_seq":11408,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.732606+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:13.732606+0000","last_peered":"2024-09-18T11:54:13.732606+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:13.732606+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12263424,"num_objects":1497,"num_object_clones":0,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-622}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11399","reported_seq":11506,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.366827+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:09.366827+0000","last_peered":"2024-09-18T11:54:09.366827+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:09.366827+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13243638,"num_objects":1617,"num_object_clones":0,"num_object_copies":6468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-499}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11195","reported_seq":13185,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.522150+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:09.522150+0000","last_peered":"2024-09-18T11:54:09.522150+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:09.522150+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13759734,"num_objects":1680,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11595","reported_seq":15948,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:13.668645+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:13.668645+0000","last_peered":"2024-09-18T11:54:13.668645+0000","last_clean":"2024-09-18T11:54:13.668645+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:13.668645+0000","last_undegraded":"2024-09-18T11:54:13.668645+0000","last_fullsized":"2024-09-18T11:54:13.668645+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12945900,"num_objects":1581,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11592,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11050","reported_seq":11164,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.472461+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:13.472461+0000","last_peered":"2024-09-18T11:54:13.472461+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:13.472461+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12645622,"num_objects":1544,"num_object_clones":0,"num_object_copies":6176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11048,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11397","reported_seq":11518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.822475+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:13.822475+0000","last_peered":"2024-09-18T11:54:13.822475+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:13.822475+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13397873,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11394,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11510","reported_seq":11603,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.564521+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:13.564521+0000","last_peered":"2024-09-18T11:54:13.564521+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:13.564521+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12458619,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-563}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11422","reported_seq":11520,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.579577+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:13.579577+0000","last_peered":"2024-09-18T11:54:13.579577+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:13.579577+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12655227,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11421,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11459","reported_seq":11583,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.837651+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:13.837651+0000","last_peered":"2024-09-18T11:54:13.837651+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:13.837651+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13358326,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-492}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11304","reported_seq":15592,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:10.629205+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:10.629205+0000","last_peered":"2024-09-18T11:54:10.629205+0000","last_clean":"2024-09-18T11:54:10.629205+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:10.629205+0000","last_undegraded":"2024-09-18T11:54:10.629205+0000","last_fullsized":"2024-09-18T11:54:10.629205+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13096182,"num_objects":1599,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11302,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11045","reported_seq":11135,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.515906+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:09.515906+0000","last_peered":"2024-09-18T11:54:09.515906+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:09.515906+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13612278,"num_objects":1662,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":209388940,"num_objects":25566,"num_object_clones":0,"num_object_copies":102264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7579,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":419905536,"data_stored":419905536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700036,"kb_used_data":101308,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93671804,"statfs":{"total":96636764160,"available":95919927296,"internally_reserved":0,"allocated":103739392,"data_stored":103555571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":4.8840000000000003},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.7670000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":16.151},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":8.9969999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.62}]}]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691852,"kb_used_data":101944,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93679988,"statfs":{"total":96636764160,"available":95928307712,"internally_reserved":0,"allocated":104390656,"data_stored":104206835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":8.2680000000000007},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":8.2959999999999994}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.499},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":8.2219999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.3290000000000006},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":4.2160000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699008,"kb_used_data":104480,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93672832,"statfs":{"total":96636764160,"available":95920979968,"internally_reserved":0,"allocated":106987520,"data_stored":106803699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":5.9290000000000003},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":6.7759999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":7.7679999999999998},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.79500000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":2.7549999999999999},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":3.2839999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703816,"kb_used_data":105152,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93668024,"statfs":{"total":96636764160,"available":95916056576,"internally_reserved":0,"allocated":107675648,"data_stored":107491827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":18.478999999999999},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.911}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":13.265000000000001},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":16.981999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.6509999999999998},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":27.015999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106799104,"data_stored":106799104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106389504,"data_stored":106389504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":103436288,"data_stored":103436288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":103280640,"data_stored":103280640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:15.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:15.856 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:15.857 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:15.884 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-18T11:54:14.058730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":209388940,"num_objects":25566,"num_object_clones":0,"num_object_copies":102264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7579,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2794712,"kb_used_data":412884,"kb_used_omap":14,"kb_used_meta":2381745,"kb_avail":374692648,"statfs":{"total":386547056640,"available":383685271552,"internally_reserved":0,"allocated":422793216,"data_stored":422057932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2438906907},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,7,9,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21716992,"num_objects":-2651,"num_object_clones":0,"num_object_copies":-10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2651,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"1.f","version":"37'11640","reported_seq":11741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.781231+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:13.781231+0000","last_peered":"2024-09-18T11:54:13.781231+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:13.781231+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13189120,"num_objects":1610,"num_object_clones":0,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11640,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-493}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11416","reported_seq":15708,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:10.461854+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:10.461854+0000","last_peered":"2024-09-18T11:54:10.461854+0000","last_clean":"2024-09-18T11:54:10.461854+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:10.461854+0000","last_undegraded":"2024-09-18T11:54:10.461854+0000","last_fullsized":"2024-09-18T11:54:10.461854+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13432054,"num_objects":1640,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11230","reported_seq":11347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.252130+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:13.252130+0000","last_peered":"2024-09-18T11:54:13.252130+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:13.252130+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12925587,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11229,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11290","reported_seq":11384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.404069+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:09.404069+0000","last_peered":"2024-09-18T11:54:09.404069+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:09.404069+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13399286,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11307","reported_seq":11428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.824624+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:13.824624+0000","last_peered":"2024-09-18T11:54:13.824624+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:13.824624+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13006070,"num_objects":1588,"num_object_clones":0,"num_object_copies":6352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11305,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11289","reported_seq":11408,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.732606+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:13.732606+0000","last_peered":"2024-09-18T11:54:13.732606+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:13.732606+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12263424,"num_objects":1497,"num_object_clones":0,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-622}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11399","reported_seq":11506,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.366827+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:09.366827+0000","last_peered":"2024-09-18T11:54:09.366827+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:09.366827+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13243638,"num_objects":1617,"num_object_clones":0,"num_object_copies":6468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-499}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11195","reported_seq":13185,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.522150+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:09.522150+0000","last_peered":"2024-09-18T11:54:09.522150+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:09.522150+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13759734,"num_objects":1680,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11595","reported_seq":15948,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:13.668645+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:13.668645+0000","last_peered":"2024-09-18T11:54:13.668645+0000","last_clean":"2024-09-18T11:54:13.668645+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:13.668645+0000","last_undegraded":"2024-09-18T11:54:13.668645+0000","last_fullsized":"2024-09-18T11:54:13.668645+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12945900,"num_objects":1581,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11592,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11050","reported_seq":11164,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.472461+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:13.472461+0000","last_peered":"2024-09-18T11:54:13.472461+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:13.472461+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12645622,"num_objects":1544,"num_object_clones":0,"num_object_copies":6176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11048,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11397","reported_seq":11518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.822475+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:13.822475+0000","last_peered":"2024-09-18T11:54:13.822475+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:13.822475+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13397873,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11394,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11510","reported_seq":11603,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.564521+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:13.564521+0000","last_peered":"2024-09-18T11:54:13.564521+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:13.564521+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12458619,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-563}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11422","reported_seq":11520,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.579577+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:13.579577+0000","last_peered":"2024-09-18T11:54:13.579577+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:13.579577+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12655227,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11421,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11459","reported_seq":11583,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.837651+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:13.837651+0000","last_peered":"2024-09-18T11:54:13.837651+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:13.837651+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13358326,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-492}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11304","reported_seq":15592,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:10.629205+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:10.629205+0000","last_peered":"2024-09-18T11:54:10.629205+0000","last_clean":"2024-09-18T11:54:10.629205+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:10.629205+0000","last_undegraded":"2024-09-18T11:54:10.629205+0000","last_fullsized":"2024-09-18T11:54:10.629205+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13096182,"num_objects":1599,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11302,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11045","reported_seq":11135,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:09.515906+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:09.515906+0000","last_peered":"2024-09-18T11:54:09.515906+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:09.515906+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13612278,"num_objects":1662,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":209388940,"num_objects":25566,"num_object_clones":0,"num_object_copies":102264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181521,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7579,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":419905536,"data_stored":419905536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700036,"kb_used_data":101308,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93671804,"statfs":{"total":96636764160,"available":95919927296,"internally_reserved":0,"allocated":103739392,"data_stored":103555571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":4.8840000000000003},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.7670000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":16.151},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":8.9969999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.62}]}]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691852,"kb_used_data":101944,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93679988,"statfs":{"total":96636764160,"available":95928307712,"internally_reserved":0,"allocated":104390656,"data_stored":104206835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":8.2680000000000007},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":8.2959999999999994}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.499},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":8.2219999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.3290000000000006},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":4.2160000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699008,"kb_used_data":104480,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93672832,"statfs":{"total":96636764160,"available":95920979968,"internally_reserved":0,"allocated":106987520,"data_stored":106803699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":5.9290000000000003},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":6.7759999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":7.7679999999999998},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.79500000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":2.7549999999999999},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":3.2839999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703816,"kb_used_data":105152,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93668024,"statfs":{"total":96636764160,"available":95916056576,"internally_reserved":0,"allocated":107675648,"data_stored":107491827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":18.478999999999999},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.911}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":13.265000000000001},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":16.981999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":5.6509999999999998},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":27.015999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106799104,"data_stored":106799104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106389504,"data_stored":106389504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":103436288,"data_stored":103436288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":103280640,"data_stored":103280640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:15.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:54:15.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11640", 'reported_seq': 11741, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:13.781231+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:13.781231+0000', 'last_peered': '2024-09-18T11:54:13.781231+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:13.781231+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13189120, 'num_objects': 1610, 'num_object_clones': 0, 'num_object_copies': 6440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11640, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -493}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:15.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:54:15.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11230", 'reported_seq': 11347, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:13.252130+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:13.252130+0000', 'last_peered': '2024-09-18T11:54:13.252130+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:13.252130+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12925587, 'num_objects': 1579, 'num_object_clones': 0, 'num_object_copies': 6316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1579, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11229, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -461}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:15.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:54:15.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11290", 'reported_seq': 11384, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:09.404069+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:54:09.404069+0000', 'last_peered': '2024-09-18T11:54:09.404069+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:54:09.404069+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13399286, 'num_objects': 1636, 'num_object_clones': 0, 'num_object_copies': 6544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11288, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -465}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:15.886 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:54:15.886 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11307", 'reported_seq': 11428, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:13.824624+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:13.824624+0000', 'last_peered': '2024-09-18T11:54:13.824624+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:13.824624+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13006070, 'num_objects': 1588, 'num_object_clones': 0, 'num_object_copies': 6352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11305, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -464}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:15.886 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:54:15.886 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11289", 'reported_seq': 11408, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:13.732606+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:13.732606+0000', 'last_peered': '2024-09-18T11:54:13.732606+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:13.732606+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12263424, 'num_objects': 1497, 'num_object_clones': 0, 'num_object_copies': 5988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11289, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -622}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:15.886 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:54:15.886 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11399", 'reported_seq': 11506, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:09.366827+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:09.366827+0000', 'last_peered': '2024-09-18T11:54:09.366827+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:09.366827+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13243638, 'num_objects': 1617, 'num_object_clones': 0, 'num_object_copies': 6468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11397, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -499}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:15.886 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:54:15.886 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11195", 'reported_seq': 13185, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:09.522150+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:54:09.522150+0000', 'last_peered': '2024-09-18T11:54:09.522150+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:54:09.522150+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13759734, 'num_objects': 1680, 'num_object_clones': 0, 'num_object_copies': 6720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11193, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 962, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:15.886 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:54:15.886 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11050", 'reported_seq': 11164, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:13.472461+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:13.472461+0000', 'last_peered': '2024-09-18T11:54:13.472461+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:13.472461+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12645622, 'num_objects': 1544, 'num_object_clones': 0, 'num_object_copies': 6176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1544, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11048, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -419}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:54:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11397", 'reported_seq': 11518, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:13.822475+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:13.822475+0000', 'last_peered': '2024-09-18T11:54:13.822475+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:13.822475+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13397873, 'num_objects': 1636, 'num_object_clones': 0, 'num_object_copies': 6544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11394, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -398}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:54:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11510", 'reported_seq': 11603, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:13.564521+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:13.564521+0000', 'last_peered': '2024-09-18T11:54:13.564521+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:13.564521+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12458619, 'num_objects': 1521, 'num_object_clones': 0, 'num_object_copies': 6084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11509, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -563}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:54:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11422", 'reported_seq': 11520, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:13.579577+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:13.579577+0000', 'last_peered': '2024-09-18T11:54:13.579577+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:13.579577+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12655227, 'num_objects': 1545, 'num_object_clones': 0, 'num_object_copies': 6180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11421, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:54:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11459", 'reported_seq': 11583, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:13.837651+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:13.837651+0000', 'last_peered': '2024-09-18T11:54:13.837651+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:13.837651+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13358326, 'num_objects': 1631, 'num_object_clones': 0, 'num_object_copies': 6524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11457, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -492}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:54:15.887 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11045", 'reported_seq': 11135, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:09.515906+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:09.515906+0000', 'last_peered': '2024-09-18T11:54:09.515906+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:09.515906+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13612278, 'num_objects': 1662, 'num_object_clones': 0, 'num_object_copies': 6648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11043, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:15.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:16.724 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:16.724 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25823/100692 objects degraded (25.646%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":654,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":25173,"data_bytes":206169484,"bytes_used":2845134848,"bytes_avail":383701921792,"bytes_total":386547056640,"degraded_objects":25823,"degraded_total":100692,"degraded_ratio":0.25645532912247249,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":200},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:16.750 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:16.750 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:17.295 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:17.296 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:17.321 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-09-18T11:54:16.059358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":206169484,"num_objects":25173,"num_object_clones":0,"num_object_copies":100692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25173,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181914,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7633,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2778452,"kb_used_data":404916,"kb_used_omap":14,"kb_used_meta":2373425,"kb_avail":374708908,"statfs":{"total":386547056640,"available":383701921792,"internally_reserved":0,"allocated":414633984,"data_stored":413898700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430387227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,10,5,13,17,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19734528,"num_objects":-2409,"num_object_clones":0,"num_object_copies":-9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2409,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003077"},"pg_stats":[{"pgid":"1.f","version":"37'11640","reported_seq":11741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.781231+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:13.781231+0000","last_peered":"2024-09-18T11:54:13.781231+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:13.781231+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13189120,"num_objects":1610,"num_object_clones":0,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11640,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-493}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11481","reported_seq":15772,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:15.606220+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:15.606220+0000","last_peered":"2024-09-18T11:54:15.606220+0000","last_clean":"2024-09-18T11:54:15.606220+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:15.606220+0000","last_undegraded":"2024-09-18T11:54:15.606220+0000","last_fullsized":"2024-09-18T11:54:15.606220+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12899574,"num_objects":1575,"num_object_clones":0,"num_object_copies":6300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11230","reported_seq":11347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.252130+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:13.252130+0000","last_peered":"2024-09-18T11:54:13.252130+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:13.252130+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12925587,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11229,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11353","reported_seq":11447,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:14.505358+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:14.505358+0000","last_peered":"2024-09-18T11:54:14.505358+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:14.505358+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12883190,"num_objects":1573,"num_object_clones":0,"num_object_copies":6292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11351,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11307","reported_seq":11428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.824624+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:13.824624+0000","last_peered":"2024-09-18T11:54:13.824624+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:13.824624+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13006070,"num_objects":1588,"num_object_clones":0,"num_object_copies":6352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11305,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11289","reported_seq":11408,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.732606+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:13.732606+0000","last_peered":"2024-09-18T11:54:13.732606+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:13.732606+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12263424,"num_objects":1497,"num_object_clones":0,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-622}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11449","reported_seq":11556,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:14.437579+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:14.437579+0000","last_peered":"2024-09-18T11:54:14.437579+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:14.437579+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12834038,"num_objects":1567,"num_object_clones":0,"num_object_copies":6268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-549}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11266","reported_seq":13363,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:14.650848+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:14.650848+0000","last_peered":"2024-09-18T11:54:14.650848+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:14.650848+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13178102,"num_objects":1609,"num_object_clones":0,"num_object_copies":6436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11264,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11595","reported_seq":15948,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:13.668645+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:13.668645+0000","last_peered":"2024-09-18T11:54:13.668645+0000","last_clean":"2024-09-18T11:54:13.668645+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:13.668645+0000","last_undegraded":"2024-09-18T11:54:13.668645+0000","last_fullsized":"2024-09-18T11:54:13.668645+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12945900,"num_objects":1581,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11592,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11050","reported_seq":11164,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.472461+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:13.472461+0000","last_peered":"2024-09-18T11:54:13.472461+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:13.472461+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12645622,"num_objects":1544,"num_object_clones":0,"num_object_copies":6176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11048,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11397","reported_seq":11518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.822475+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:13.822475+0000","last_peered":"2024-09-18T11:54:13.822475+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:13.822475+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13397873,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11394,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11510","reported_seq":11603,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.564521+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:13.564521+0000","last_peered":"2024-09-18T11:54:13.564521+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:13.564521+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12458619,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-563}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11422","reported_seq":11520,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.579577+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:13.579577+0000","last_peered":"2024-09-18T11:54:13.579577+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:13.579577+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12655227,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11421,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11459","reported_seq":11583,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:13.837651+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:13.837651+0000","last_peered":"2024-09-18T11:54:13.837651+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:13.837651+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13358326,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-492}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11366","reported_seq":15655,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:15.586212+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:15.586212+0000","last_peered":"2024-09-18T11:54:15.586212+0000","last_clean":"2024-09-18T11:54:15.586212+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:15.586212+0000","last_undegraded":"2024-09-18T11:54:15.586212+0000","last_fullsized":"2024-09-18T11:54:15.586212+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12588278,"num_objects":1537,"num_object_clones":0,"num_object_copies":6148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11364,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11127","reported_seq":11217,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:14.075443+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:14.075443+0000","last_peered":"2024-09-18T11:54:14.075443+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:14.075443+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12940534,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11125,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-422}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":206169484,"num_objects":25173,"num_object_clones":0,"num_object_copies":100692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25173,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181914,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7633,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":411586560,"data_stored":411586560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700036,"kb_used_data":101308,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93671804,"statfs":{"total":96636764160,"available":95919927296,"internally_reserved":0,"allocated":103739392,"data_stored":103555571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":4.8840000000000003},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.7670000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":16.151},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":8.9969999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":0.62}]}]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691852,"kb_used_data":101944,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93679988,"statfs":{"total":96636764160,"available":95928307712,"internally_reserved":0,"allocated":104390656,"data_stored":104206835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2119999999999997,"5min":4.8479999999999999,"15min":3.347},"min":{"1min":0.48599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":13.747999999999999,"5min":24.965,"15min":24.965},"last":8.2680000000000007},{"interface":"front","average":{"1min":4.3040000000000003,"5min":4.9000000000000004,"15min":2.9569999999999999},"min":{"1min":0.61299999999999999,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":13.785,"5min":24.995000000000001,"15min":24.995000000000001},"last":8.2959999999999994}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.499},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":8.2219999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":8.3290000000000006},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":4.2160000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699204,"kb_used_data":100492,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93672636,"statfs":{"total":96636764160,"available":95920779264,"internally_reserved":0,"allocated":102903808,"data_stored":102719987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":21.033000000000001},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":20.965}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":2.8620000000000001},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":21.065999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":4.9889999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687360,"kb_used_data":101172,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93684480,"statfs":{"total":96636764160,"available":95932907520,"internally_reserved":0,"allocated":103600128,"data_stored":103416307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,0,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":3.9580000000000002},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":27.445}]},{"osd":2,"last update":"Wed Sep 18 11:53:14 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":6.4290000000000003,"15min":3.6469999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":28.969000000000001,"5min":29.437000000000001,"15min":32.795000000000002},"last":9.3149999999999995},{"interface":"front","average":{"1min":6.2560000000000002,"5min":5.3490000000000002,"15min":3.2530000000000001},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":36.697000000000003,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.96399999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":3.9889999999999999},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":4.9219999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102641664,"data_stored":102641664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102227968,"data_stored":102227968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":103436288,"data_stored":103436288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":103280640,"data_stored":103280640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:20.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:20.950 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:20.952 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:20.983 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-18T11:54:20.060733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":199091596,"num_objects":24309,"num_object_clones":0,"num_object_copies":97236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182778,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7689,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160905,"ondisk_log_size":160905,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779120,"kb_used_data":393112,"kb_used_omap":14,"kb_used_meta":2385905,"kb_avail":374708240,"statfs":{"total":386547056640,"available":383701237760,"internally_reserved":0,"allocated":402546688,"data_stored":401811404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2443166747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,11,8,12,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23592960,"num_objects":-2880,"num_object_clones":0,"num_object_copies":-11520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2880,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003449"},"pg_stats":[{"pgid":"1.f","version":"37'11716","reported_seq":11816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.874658+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:18.874658+0000","last_peered":"2024-09-18T11:54:18.874658+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:18.874658+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12566528,"num_objects":1534,"num_object_clones":0,"num_object_copies":6136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11716,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11481","reported_seq":15772,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:15.606220+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:15.606220+0000","last_peered":"2024-09-18T11:54:15.606220+0000","last_clean":"2024-09-18T11:54:15.606220+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:15.606220+0000","last_undegraded":"2024-09-18T11:54:15.606220+0000","last_fullsized":"2024-09-18T11:54:15.606220+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12899574,"num_objects":1575,"num_object_clones":0,"num_object_copies":6300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11290","reported_seq":11407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.524112+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:18.524112+0000","last_peered":"2024-09-18T11:54:18.524112+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:18.524112+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12434067,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11289,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11397","reported_seq":11491,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.590285+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:19.590285+0000","last_peered":"2024-09-18T11:54:19.590285+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:19.590285+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12522742,"num_objects":1529,"num_object_clones":0,"num_object_copies":6116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11372","reported_seq":11493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.858177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:18.858177+0000","last_peered":"2024-09-18T11:54:18.858177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:18.858177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12473590,"num_objects":1523,"num_object_clones":0,"num_object_copies":6092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11370,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11359","reported_seq":11478,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.777524+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:18.777524+0000","last_peered":"2024-09-18T11:54:18.777524+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:18.777524+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11689984,"num_objects":1427,"num_object_clones":0,"num_object_copies":5708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11514","reported_seq":11621,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.649398+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:19.649398+0000","last_peered":"2024-09-18T11:54:19.649398+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:19.649398+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12301558,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11512,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-614}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11325","reported_seq":13535,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.747216+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:19.747216+0000","last_peered":"2024-09-18T11:54:19.747216+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:19.747216+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12694774,"num_objects":1550,"num_object_clones":0,"num_object_copies":6200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11323,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1072,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11658","reported_seq":16010,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.775066+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:18.775066+0000","last_peered":"2024-09-18T11:54:18.775066+0000","last_clean":"2024-09-18T11:54:18.775066+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:18.775066+0000","last_undegraded":"2024-09-18T11:54:18.775066+0000","last_fullsized":"2024-09-18T11:54:18.775066+0000","mapping_epoch":36,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12429804,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11655,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11109","reported_seq":11223,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.613828+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:18.613828+0000","last_peered":"2024-09-18T11:54:18.613828+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:18.613828+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12162294,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11107,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11457","reported_seq":11578,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.753375+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:18.753375+0000","last_peered":"2024-09-18T11:54:18.753375+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:18.753375+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12906353,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11581","reported_seq":11674,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.878506+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:18.878506+0000","last_peered":"2024-09-18T11:54:18.878506+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:18.878506+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11876987,"num_objects":1450,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-634}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11478","reported_seq":11576,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.497082+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:18.497082+0000","last_peered":"2024-09-18T11:54:18.497082+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:18.497082+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12196475,"num_objects":1489,"num_object_clones":0,"num_object_copies":5956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11510","reported_seq":11635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.526005+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:18.526005+0000","last_peered":"2024-09-18T11:54:18.526005+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:18.526005+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12940534,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11508,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-543}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11366","reported_seq":15655,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:15.586212+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:15.586212+0000","last_peered":"2024-09-18T11:54:15.586212+0000","last_clean":"2024-09-18T11:54:15.586212+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:15.586212+0000","last_undegraded":"2024-09-18T11:54:15.586212+0000","last_fullsized":"2024-09-18T11:54:15.586212+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12588278,"num_objects":1537,"num_object_clones":0,"num_object_copies":6148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11364,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11192","reported_seq":11281,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.694987+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:19.694987+0000","last_peered":"2024-09-18T11:54:19.694987+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:19.694987+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12408054,"num_objects":1515,"num_object_clones":0,"num_object_copies":6060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11190,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":199091596,"num_objects":24309,"num_object_clones":0,"num_object_copies":97236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182778,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7689,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":399446016,"data_stored":399446016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160905,"ondisk_log_size":160905,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700296,"kb_used_data":97400,"kb_used_omap":4,"kb_used_meta":602875,"kb_avail":93671544,"statfs":{"total":96636764160,"available":95919661056,"internally_reserved":0,"allocated":99737600,"data_stored":99553779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617344577},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":17.937999999999999},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.3680000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":18.143000000000001},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":7.7039999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":17.045000000000002},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":9.452}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692104,"kb_used_data":98048,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93679736,"statfs":{"total":96636764160,"available":95928049664,"internally_reserved":0,"allocated":100401152,"data_stored":100217331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.681},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.7039999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":5.6139999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":5.6440000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":17.547999999999998},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":5.7359999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699204,"kb_used_data":100492,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93672636,"statfs":{"total":96636764160,"available":95920779264,"internally_reserved":0,"allocated":102903808,"data_stored":102719987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":21.033000000000001},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":20.965}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":2.8620000000000001},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":21.065999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":4.9889999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687516,"kb_used_data":97172,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93684324,"statfs":{"total":96636764160,"available":95932747776,"internally_reserved":0,"allocated":99504128,"data_stored":99320307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.89500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.431},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.6599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":2.069}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98541568,"data_stored":98541568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102227968,"data_stored":102227968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":99414016,"data_stored":99414016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99262464,"data_stored":99262464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:20.985 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:21.489 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:21.490 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:21.517 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-18T11:54:20.060733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":199091596,"num_objects":24309,"num_object_clones":0,"num_object_copies":97236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182778,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7689,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160905,"ondisk_log_size":160905,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779120,"kb_used_data":393112,"kb_used_omap":14,"kb_used_meta":2385905,"kb_avail":374708240,"statfs":{"total":386547056640,"available":383701237760,"internally_reserved":0,"allocated":402546688,"data_stored":401811404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2443166747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,11,8,12,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23592960,"num_objects":-2880,"num_object_clones":0,"num_object_copies":-11520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2880,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003449"},"pg_stats":[{"pgid":"1.f","version":"37'11716","reported_seq":11816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.874658+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:18.874658+0000","last_peered":"2024-09-18T11:54:18.874658+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:18.874658+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12566528,"num_objects":1534,"num_object_clones":0,"num_object_copies":6136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11716,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11481","reported_seq":15772,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:15.606220+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:15.606220+0000","last_peered":"2024-09-18T11:54:15.606220+0000","last_clean":"2024-09-18T11:54:15.606220+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:15.606220+0000","last_undegraded":"2024-09-18T11:54:15.606220+0000","last_fullsized":"2024-09-18T11:54:15.606220+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12899574,"num_objects":1575,"num_object_clones":0,"num_object_copies":6300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11290","reported_seq":11407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.524112+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:18.524112+0000","last_peered":"2024-09-18T11:54:18.524112+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:18.524112+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12434067,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11289,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11397","reported_seq":11491,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.590285+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:19.590285+0000","last_peered":"2024-09-18T11:54:19.590285+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:19.590285+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12522742,"num_objects":1529,"num_object_clones":0,"num_object_copies":6116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11372","reported_seq":11493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.858177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:18.858177+0000","last_peered":"2024-09-18T11:54:18.858177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:18.858177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12473590,"num_objects":1523,"num_object_clones":0,"num_object_copies":6092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11370,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11359","reported_seq":11478,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.777524+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:18.777524+0000","last_peered":"2024-09-18T11:54:18.777524+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:18.777524+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11689984,"num_objects":1427,"num_object_clones":0,"num_object_copies":5708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11514","reported_seq":11621,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.649398+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:19.649398+0000","last_peered":"2024-09-18T11:54:19.649398+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:19.649398+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12301558,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11512,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-614}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11325","reported_seq":13535,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.747216+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:19.747216+0000","last_peered":"2024-09-18T11:54:19.747216+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:19.747216+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12694774,"num_objects":1550,"num_object_clones":0,"num_object_copies":6200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11323,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1072,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11658","reported_seq":16010,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.775066+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:18.775066+0000","last_peered":"2024-09-18T11:54:18.775066+0000","last_clean":"2024-09-18T11:54:18.775066+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:18.775066+0000","last_undegraded":"2024-09-18T11:54:18.775066+0000","last_fullsized":"2024-09-18T11:54:18.775066+0000","mapping_epoch":36,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12429804,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11655,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11109","reported_seq":11223,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.613828+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:18.613828+0000","last_peered":"2024-09-18T11:54:18.613828+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:18.613828+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12162294,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11107,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11457","reported_seq":11578,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.753375+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:18.753375+0000","last_peered":"2024-09-18T11:54:18.753375+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:18.753375+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12906353,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11581","reported_seq":11674,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.878506+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:18.878506+0000","last_peered":"2024-09-18T11:54:18.878506+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:18.878506+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11876987,"num_objects":1450,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-634}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11478","reported_seq":11576,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.497082+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:18.497082+0000","last_peered":"2024-09-18T11:54:18.497082+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:18.497082+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12196475,"num_objects":1489,"num_object_clones":0,"num_object_copies":5956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11510","reported_seq":11635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.526005+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:18.526005+0000","last_peered":"2024-09-18T11:54:18.526005+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:18.526005+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12940534,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11508,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-543}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11366","reported_seq":15655,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:15.586212+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:15.586212+0000","last_peered":"2024-09-18T11:54:15.586212+0000","last_clean":"2024-09-18T11:54:15.586212+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:15.586212+0000","last_undegraded":"2024-09-18T11:54:15.586212+0000","last_fullsized":"2024-09-18T11:54:15.586212+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12588278,"num_objects":1537,"num_object_clones":0,"num_object_copies":6148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11364,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11192","reported_seq":11281,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.694987+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:19.694987+0000","last_peered":"2024-09-18T11:54:19.694987+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:19.694987+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12408054,"num_objects":1515,"num_object_clones":0,"num_object_copies":6060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11190,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":199091596,"num_objects":24309,"num_object_clones":0,"num_object_copies":97236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182778,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7689,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":399446016,"data_stored":399446016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160905,"ondisk_log_size":160905,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700296,"kb_used_data":97400,"kb_used_omap":4,"kb_used_meta":602875,"kb_avail":93671544,"statfs":{"total":96636764160,"available":95919661056,"internally_reserved":0,"allocated":99737600,"data_stored":99553779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617344577},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":17.937999999999999},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.3680000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":18.143000000000001},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":7.7039999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":17.045000000000002},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":9.452}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692104,"kb_used_data":98048,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93679736,"statfs":{"total":96636764160,"available":95928049664,"internally_reserved":0,"allocated":100401152,"data_stored":100217331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.681},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.7039999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":5.6139999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":5.6440000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":17.547999999999998},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":5.7359999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699204,"kb_used_data":100492,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93672636,"statfs":{"total":96636764160,"available":95920779264,"internally_reserved":0,"allocated":102903808,"data_stored":102719987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":21.033000000000001},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":20.965}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":2.8620000000000001},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":21.065999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":4.9889999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687516,"kb_used_data":97172,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93684324,"statfs":{"total":96636764160,"available":95932747776,"internally_reserved":0,"allocated":99504128,"data_stored":99320307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.89500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.431},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.6599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":2.069}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98541568,"data_stored":98541568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102227968,"data_stored":102227968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":99414016,"data_stored":99414016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99262464,"data_stored":99262464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:21.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:54:21.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11716", 'reported_seq': 11816, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:18.874658+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:18.874658+0000', 'last_peered': '2024-09-18T11:54:18.874658+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:18.874658+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12566528, 'num_objects': 1534, 'num_object_clones': 0, 'num_object_copies': 6136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11716, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -569}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:21.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:54:21.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11290", 'reported_seq': 11407, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:18.524112+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:18.524112+0000', 'last_peered': '2024-09-18T11:54:18.524112+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:18.524112+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12434067, 'num_objects': 1519, 'num_object_clones': 0, 'num_object_copies': 6076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1519, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11289, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -521}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:21.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:54:21.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11397", 'reported_seq': 11491, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:19.590285+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:54:19.590285+0000', 'last_peered': '2024-09-18T11:54:19.590285+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:54:19.590285+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12522742, 'num_objects': 1529, 'num_object_clones': 0, 'num_object_copies': 6116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11395, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:21.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:54:21.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11372", 'reported_seq': 11493, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:18.858177+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:18.858177+0000', 'last_peered': '2024-09-18T11:54:18.858177+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:18.858177+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12473590, 'num_objects': 1523, 'num_object_clones': 0, 'num_object_copies': 6092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11370, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:21.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:54:21.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11359", 'reported_seq': 11478, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:18.777524+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:18.777524+0000', 'last_peered': '2024-09-18T11:54:18.777524+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:18.777524+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11689984, 'num_objects': 1427, 'num_object_clones': 0, 'num_object_copies': 5708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11359, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -692}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:21.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:54:21.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11514", 'reported_seq': 11621, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:19.649398+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:19.649398+0000', 'last_peered': '2024-09-18T11:54:19.649398+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:19.649398+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12301558, 'num_objects': 1502, 'num_object_clones': 0, 'num_object_copies': 6008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11512, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -614}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:21.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:54:21.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11325", 'reported_seq': 13535, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:19.747216+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:54:19.747216+0000', 'last_peered': '2024-09-18T11:54:19.747216+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:54:19.747216+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12694774, 'num_objects': 1550, 'num_object_clones': 0, 'num_object_copies': 6200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11323, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1072, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 584}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:21.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11109", 'reported_seq': 11223, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:18.613828+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:18.613828+0000', 'last_peered': '2024-09-18T11:54:18.613828+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:18.613828+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12162294, 'num_objects': 1485, 'num_object_clones': 0, 'num_object_copies': 5940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11107, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -478}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11457", 'reported_seq': 11578, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:18.753375+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:18.753375+0000', 'last_peered': '2024-09-18T11:54:18.753375+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:18.753375+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12906353, 'num_objects': 1576, 'num_object_clones': 0, 'num_object_copies': 6304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11454, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -458}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11581", 'reported_seq': 11674, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:18.878506+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:18.878506+0000', 'last_peered': '2024-09-18T11:54:18.878506+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:18.878506+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11876987, 'num_objects': 1450, 'num_object_clones': 0, 'num_object_copies': 5800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11580, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -634}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11478", 'reported_seq': 11576, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:18.497082+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:18.497082+0000', 'last_peered': '2024-09-18T11:54:18.497082+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:18.497082+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12196475, 'num_objects': 1489, 'num_object_clones': 0, 'num_object_copies': 5956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11477, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11510", 'reported_seq': 11635, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:18.526005+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:18.526005+0000', 'last_peered': '2024-09-18T11:54:18.526005+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:18.526005+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12940534, 'num_objects': 1580, 'num_object_clones': 0, 'num_object_copies': 6320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11508, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -543}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:54:21.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11192", 'reported_seq': 11281, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:19.694987+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:19.694987+0000', 'last_peered': '2024-09-18T11:54:19.694987+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:19.694987+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12408054, 'num_objects': 1515, 'num_object_clones': 0, 'num_object_copies': 6060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11190, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -487}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:21.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:22.207 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:22.207 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25767/96688 objects degraded (26.650%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":660,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24172,"data_bytes":197969292,"bytes_used":2841755648,"bytes_avail":383705300992,"bytes_total":386547056640,"degraded_objects":25767,"degraded_total":96688,"degraded_ratio":0.26649635942412708,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":179},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:22.229 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:22.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:22.880 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:22.881 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:22.922 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-18T11:54:22.061434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":197969292,"num_objects":24172,"num_object_clones":0,"num_object_copies":96688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24172,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182915,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7689,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775152,"kb_used_data":389192,"kb_used_omap":14,"kb_used_meta":2385905,"kb_avail":374712208,"statfs":{"total":386547056640,"available":383705300992,"internally_reserved":0,"allocated":398532608,"data_stored":397797324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2443166747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,7,20,10,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17661952,"num_objects":-2156,"num_object_clones":0,"num_object_copies":-8624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003717"},"pg_stats":[{"pgid":"1.f","version":"37'11716","reported_seq":11816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.874658+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:18.874658+0000","last_peered":"2024-09-18T11:54:18.874658+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:18.874658+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12566528,"num_objects":1534,"num_object_clones":0,"num_object_copies":6136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11716,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11549","reported_seq":15841,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:20.795565+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:20.795565+0000","last_peered":"2024-09-18T11:54:20.795565+0000","last_clean":"2024-09-18T11:54:20.795565+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:20.795565+0000","last_undegraded":"2024-09-18T11:54:20.795565+0000","last_fullsized":"2024-09-18T11:54:20.795565+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12342518,"num_objects":1507,"num_object_clones":0,"num_object_copies":6028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11290","reported_seq":11407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.524112+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:18.524112+0000","last_peered":"2024-09-18T11:54:18.524112+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:18.524112+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12434067,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11289,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11397","reported_seq":11491,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.590285+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:19.590285+0000","last_peered":"2024-09-18T11:54:19.590285+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:19.590285+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12522742,"num_objects":1529,"num_object_clones":0,"num_object_copies":6116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11372","reported_seq":11493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.858177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:18.858177+0000","last_peered":"2024-09-18T11:54:18.858177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:18.858177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12473590,"num_objects":1523,"num_object_clones":0,"num_object_copies":6092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11370,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11359","reported_seq":11478,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.777524+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:18.777524+0000","last_peered":"2024-09-18T11:54:18.777524+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:18.777524+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11689984,"num_objects":1427,"num_object_clones":0,"num_object_copies":5708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11514","reported_seq":11621,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.649398+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:19.649398+0000","last_peered":"2024-09-18T11:54:19.649398+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:19.649398+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12301558,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11512,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-614}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11325","reported_seq":13535,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.747216+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:19.747216+0000","last_peered":"2024-09-18T11:54:19.747216+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:19.747216+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12694774,"num_objects":1550,"num_object_clones":0,"num_object_copies":6200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11323,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1072,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11658","reported_seq":16010,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.775066+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:18.775066+0000","last_peered":"2024-09-18T11:54:18.775066+0000","last_clean":"2024-09-18T11:54:18.775066+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:18.775066+0000","last_undegraded":"2024-09-18T11:54:18.775066+0000","last_fullsized":"2024-09-18T11:54:18.775066+0000","mapping_epoch":36,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12429804,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11655,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11109","reported_seq":11223,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.613828+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:18.613828+0000","last_peered":"2024-09-18T11:54:18.613828+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:18.613828+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12162294,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11107,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11457","reported_seq":11578,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.753375+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:18.753375+0000","last_peered":"2024-09-18T11:54:18.753375+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:18.753375+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12906353,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11581","reported_seq":11674,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.878506+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:18.878506+0000","last_peered":"2024-09-18T11:54:18.878506+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:18.878506+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11876987,"num_objects":1450,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-634}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11478","reported_seq":11576,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.497082+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:18.497082+0000","last_peered":"2024-09-18T11:54:18.497082+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:18.497082+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12196475,"num_objects":1489,"num_object_clones":0,"num_object_copies":5956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11510","reported_seq":11635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:18.526005+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:18.526005+0000","last_peered":"2024-09-18T11:54:18.526005+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:18.526005+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12940534,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11508,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-543}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11435","reported_seq":15724,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:20.786016+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:20.786016+0000","last_peered":"2024-09-18T11:54:20.786016+0000","last_clean":"2024-09-18T11:54:20.786016+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:20.786016+0000","last_undegraded":"2024-09-18T11:54:20.786016+0000","last_fullsized":"2024-09-18T11:54:20.786016+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12023030,"num_objects":1468,"num_object_clones":0,"num_object_copies":5872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11433,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11192","reported_seq":11281,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:19.694987+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:19.694987+0000","last_peered":"2024-09-18T11:54:19.694987+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:19.694987+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12408054,"num_objects":1515,"num_object_clones":0,"num_object_copies":6060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11190,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":197969292,"num_objects":24172,"num_object_clones":0,"num_object_copies":96688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24172,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182915,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7689,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":395251712,"data_stored":395251712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700296,"kb_used_data":97400,"kb_used_omap":4,"kb_used_meta":602875,"kb_avail":93671544,"statfs":{"total":96636764160,"available":95919661056,"internally_reserved":0,"allocated":99737600,"data_stored":99553779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617344577},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":8.7759999999999998,"5min":6.7160000000000002,"15min":4.4619999999999997},"min":{"1min":0.63,"5min":0.186,"15min":0.155},"max":{"1min":23.504000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":17.937999999999999},{"interface":"front","average":{"1min":7.5369999999999999,"5min":6.2229999999999999,"15min":4.3730000000000002},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.518999999999998,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.3680000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":18.143000000000001},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":7.7039999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":5.6900000000000004,"5min":7.069,"15min":3.7269999999999999},"min":{"1min":0.23799999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.742000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":17.045000000000002},{"interface":"front","average":{"1min":3.9540000000000002,"5min":4.5810000000000004,"15min":3.6230000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":14.603,"5min":21.431999999999999,"15min":21.431999999999999},"last":9.452}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692104,"kb_used_data":98048,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93679736,"statfs":{"total":96636764160,"available":95928049664,"internally_reserved":0,"allocated":100401152,"data_stored":100217331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.681},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.7039999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":5.6139999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":5.6440000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":17.547999999999998},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":5.7359999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695236,"kb_used_data":96572,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93676604,"statfs":{"total":96636764160,"available":95924842496,"internally_reserved":0,"allocated":98889728,"data_stored":98705907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,12,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":4.9329999999999998},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":12.592000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":4.4729999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.4829999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.37}]}]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687516,"kb_used_data":97172,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93684324,"statfs":{"total":96636764160,"available":95932747776,"internally_reserved":0,"allocated":99504128,"data_stored":99320307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.89500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.431},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.6599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":7.0789999999999997,"5min":5.5839999999999996,"15min":3.7130000000000001},"min":{"1min":0.249,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":26.114000000000001,"5min":26.149999999999999,"15min":26.149999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":9.141,"5min":7.5899999999999999,"15min":4.1609999999999996},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.17299999999999999},"max":{"1min":23.18,"5min":32.947000000000003,"15min":32.947000000000003},"last":2.069}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98541568,"data_stored":98541568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98033664,"data_stored":98033664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":99414016,"data_stored":99414016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99262464,"data_stored":99262464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:25.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:26.478 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:26.479 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:26.499 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-18T11:54:26.062595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189719948,"num_objects":23165,"num_object_clones":0,"num_object_copies":92660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183922,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7735,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2746704,"kb_used_data":373232,"kb_used_omap":14,"kb_used_meta":2373425,"kb_avail":374740656,"statfs":{"total":386547056640,"available":383734431744,"internally_reserved":0,"allocated":382189568,"data_stored":381454284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430387227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,10,14,17,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19668992,"num_objects":-2401,"num_object_clones":0,"num_object_copies":-9604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2401,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003864"},"pg_stats":[{"pgid":"1.f","version":"37'11781","reported_seq":11882,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.802135+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:23.802135+0000","last_peered":"2024-09-18T11:54:23.802135+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:23.802135+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12034048,"num_objects":1469,"num_object_clones":0,"num_object_copies":5876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11613","reported_seq":15905,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:25.840273+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:25.840273+0000","last_peered":"2024-09-18T11:54:25.840273+0000","last_clean":"2024-09-18T11:54:25.840273+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:25.840273+0000","last_undegraded":"2024-09-18T11:54:25.840273+0000","last_fullsized":"2024-09-18T11:54:25.840273+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11818230,"num_objects":1443,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11611,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11350","reported_seq":11467,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.771576+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:23.771576+0000","last_peered":"2024-09-18T11:54:23.771576+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:23.771576+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11942547,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11349,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-581}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11456","reported_seq":11550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.700764+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:24.700764+0000","last_peered":"2024-09-18T11:54:24.700764+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:24.700764+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12039414,"num_objects":1470,"num_object_clones":0,"num_object_copies":5880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11418","reported_seq":11539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.813326+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:23.813326+0000","last_peered":"2024-09-18T11:54:23.813326+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:23.813326+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12096758,"num_objects":1477,"num_object_clones":0,"num_object_copies":5908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11433","reported_seq":11552,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.871478+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:23.871478+0000","last_peered":"2024-09-18T11:54:23.871478+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:23.871478+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11083776,"num_objects":1353,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11433,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11579","reported_seq":11686,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.620440+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:24.620440+0000","last_peered":"2024-09-18T11:54:24.620440+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:24.620440+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11769078,"num_objects":1437,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11577,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-679}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11380","reported_seq":13682,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.793786+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:24.793786+0000","last_peered":"2024-09-18T11:54:24.793786+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:24.793786+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12244214,"num_objects":1495,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11378,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":575}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11722","reported_seq":16075,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:23.896921+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:23.896921+0000","last_peered":"2024-09-18T11:54:23.896921+0000","last_clean":"2024-09-18T11:54:23.896921+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:23.896921+0000","last_undegraded":"2024-09-18T11:54:23.896921+0000","last_fullsized":"2024-09-18T11:54:23.896921+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11905516,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11719,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11170","reported_seq":11284,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.772926+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:23.772926+0000","last_peered":"2024-09-18T11:54:23.772926+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:23.772926+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11662582,"num_objects":1424,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11168,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11526","reported_seq":11645,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.579136+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:23.579136+0000","last_peered":"2024-09-18T11:54:23.579136+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:23.579136+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12341105,"num_objects":1507,"num_object_clones":0,"num_object_copies":6028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11523,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-527}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11633","reported_seq":11726,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.854365+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:23.854365+0000","last_peered":"2024-09-18T11:54:23.854365+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:23.854365+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11451003,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11632,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-686}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11547","reported_seq":11645,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.534283+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:23.534283+0000","last_peered":"2024-09-18T11:54:23.534283+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:23.534283+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11631227,"num_objects":1420,"num_object_clones":0,"num_object_copies":5680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11546,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-626}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11574","reported_seq":11699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.802639+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:23.802639+0000","last_peered":"2024-09-18T11:54:23.802639+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:23.802639+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12416246,"num_objects":1516,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11572,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-607}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11511","reported_seq":15800,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:25.955026+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:25.955026+0000","last_peered":"2024-09-18T11:54:25.955026+0000","last_clean":"2024-09-18T11:54:25.955026+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:25.955026+0000","last_undegraded":"2024-09-18T11:54:25.955026+0000","last_fullsized":"2024-09-18T11:54:25.955026+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11400438,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11256","reported_seq":11346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.668555+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:24.668555+0000","last_peered":"2024-09-18T11:54:24.668555+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:24.668555+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11883766,"num_objects":1451,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-551}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189719948,"num_objects":23165,"num_object_clones":0,"num_object_copies":92660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183922,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7735,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":378716160,"data_stored":378716160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683764,"kb_used_data":93356,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93688076,"statfs":{"total":96636764160,"available":95936589824,"internally_reserved":0,"allocated":95596544,"data_stored":95412723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,5,3,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":4.923},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.22600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":4.899},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":4.9969999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":5.0129999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":5.0350000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692240,"kb_used_data":94024,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93679600,"statfs":{"total":96636764160,"available":95927910400,"internally_reserved":0,"allocated":96280576,"data_stored":96096755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,1,4,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":0.40699999999999997},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":0.435}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.6480000000000001},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.61599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":12.000999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":7.1369999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682900,"kb_used_data":92556,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93688940,"statfs":{"total":96636764160,"available":95937474560,"internally_reserved":0,"allocated":94777344,"data_stored":94593523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":6.4329999999999998},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":5.4359999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":6.1139999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.29199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":5.5960000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.8940000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687800,"kb_used_data":93296,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93684040,"statfs":{"total":96636764160,"available":95932456960,"internally_reserved":0,"allocated":95535104,"data_stored":95351283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.133},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.93300000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":2.7229999999999999},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.2199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":7.0960000000000001},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.2590000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94408704,"data_stored":94408704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":93822976,"data_stored":93822976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95293440,"data_stored":95293440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":95191040,"data_stored":95191040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:26.501 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:27.083 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:27.084 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:27.105 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-18T11:54:26.062595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189719948,"num_objects":23165,"num_object_clones":0,"num_object_copies":92660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183922,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7735,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2746704,"kb_used_data":373232,"kb_used_omap":14,"kb_used_meta":2373425,"kb_avail":374740656,"statfs":{"total":386547056640,"available":383734431744,"internally_reserved":0,"allocated":382189568,"data_stored":381454284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430387227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,10,14,17,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19668992,"num_objects":-2401,"num_object_clones":0,"num_object_copies":-9604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2401,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003864"},"pg_stats":[{"pgid":"1.f","version":"37'11781","reported_seq":11882,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.802135+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:23.802135+0000","last_peered":"2024-09-18T11:54:23.802135+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:23.802135+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12034048,"num_objects":1469,"num_object_clones":0,"num_object_copies":5876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11613","reported_seq":15905,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:25.840273+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:25.840273+0000","last_peered":"2024-09-18T11:54:25.840273+0000","last_clean":"2024-09-18T11:54:25.840273+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:25.840273+0000","last_undegraded":"2024-09-18T11:54:25.840273+0000","last_fullsized":"2024-09-18T11:54:25.840273+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11818230,"num_objects":1443,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11611,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11350","reported_seq":11467,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.771576+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:23.771576+0000","last_peered":"2024-09-18T11:54:23.771576+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:23.771576+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11942547,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11349,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-581}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11456","reported_seq":11550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.700764+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:24.700764+0000","last_peered":"2024-09-18T11:54:24.700764+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:24.700764+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12039414,"num_objects":1470,"num_object_clones":0,"num_object_copies":5880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11418","reported_seq":11539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.813326+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:23.813326+0000","last_peered":"2024-09-18T11:54:23.813326+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:23.813326+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12096758,"num_objects":1477,"num_object_clones":0,"num_object_copies":5908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11433","reported_seq":11552,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.871478+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:23.871478+0000","last_peered":"2024-09-18T11:54:23.871478+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:23.871478+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11083776,"num_objects":1353,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11433,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11579","reported_seq":11686,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.620440+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:24.620440+0000","last_peered":"2024-09-18T11:54:24.620440+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:24.620440+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11769078,"num_objects":1437,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11577,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-679}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11380","reported_seq":13682,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.793786+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:24.793786+0000","last_peered":"2024-09-18T11:54:24.793786+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:24.793786+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12244214,"num_objects":1495,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11378,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":575}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11722","reported_seq":16075,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:23.896921+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:23.896921+0000","last_peered":"2024-09-18T11:54:23.896921+0000","last_clean":"2024-09-18T11:54:23.896921+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:23.896921+0000","last_undegraded":"2024-09-18T11:54:23.896921+0000","last_fullsized":"2024-09-18T11:54:23.896921+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11905516,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11719,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11170","reported_seq":11284,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.772926+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:23.772926+0000","last_peered":"2024-09-18T11:54:23.772926+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:23.772926+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11662582,"num_objects":1424,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11168,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11526","reported_seq":11645,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.579136+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:23.579136+0000","last_peered":"2024-09-18T11:54:23.579136+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:23.579136+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12341105,"num_objects":1507,"num_object_clones":0,"num_object_copies":6028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11523,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-527}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11633","reported_seq":11726,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.854365+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:23.854365+0000","last_peered":"2024-09-18T11:54:23.854365+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:23.854365+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11451003,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11632,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-686}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11547","reported_seq":11645,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.534283+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:23.534283+0000","last_peered":"2024-09-18T11:54:23.534283+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:23.534283+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11631227,"num_objects":1420,"num_object_clones":0,"num_object_copies":5680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11546,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-626}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11574","reported_seq":11699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.802639+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:23.802639+0000","last_peered":"2024-09-18T11:54:23.802639+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:23.802639+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12416246,"num_objects":1516,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11572,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-607}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11511","reported_seq":15800,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:25.955026+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:25.955026+0000","last_peered":"2024-09-18T11:54:25.955026+0000","last_clean":"2024-09-18T11:54:25.955026+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:25.955026+0000","last_undegraded":"2024-09-18T11:54:25.955026+0000","last_fullsized":"2024-09-18T11:54:25.955026+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11400438,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11256","reported_seq":11346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.668555+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:24.668555+0000","last_peered":"2024-09-18T11:54:24.668555+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:24.668555+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11883766,"num_objects":1451,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-551}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189719948,"num_objects":23165,"num_object_clones":0,"num_object_copies":92660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183922,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7735,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":378716160,"data_stored":378716160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683764,"kb_used_data":93356,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93688076,"statfs":{"total":96636764160,"available":95936589824,"internally_reserved":0,"allocated":95596544,"data_stored":95412723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,5,3,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":4.923},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.22600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":4.899},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":4.9969999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":5.0129999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":5.0350000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692240,"kb_used_data":94024,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93679600,"statfs":{"total":96636764160,"available":95927910400,"internally_reserved":0,"allocated":96280576,"data_stored":96096755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,1,4,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":0.40699999999999997},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":0.435}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.6480000000000001},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.61599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":12.000999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":7.1369999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682900,"kb_used_data":92556,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93688940,"statfs":{"total":96636764160,"available":95937474560,"internally_reserved":0,"allocated":94777344,"data_stored":94593523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":6.4329999999999998},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":5.4359999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":6.1139999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.29199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":5.5960000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.8940000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687800,"kb_used_data":93296,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93684040,"statfs":{"total":96636764160,"available":95932456960,"internally_reserved":0,"allocated":95535104,"data_stored":95351283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.133},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.93300000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":2.7229999999999999},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.2199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":7.0960000000000001},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.2590000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94408704,"data_stored":94408704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":93822976,"data_stored":93822976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95293440,"data_stored":95293440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":95191040,"data_stored":95191040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:27.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:54:27.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11781", 'reported_seq': 11882, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:23.802135+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:23.802135+0000', 'last_peered': '2024-09-18T11:54:23.802135+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:23.802135+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12034048, 'num_objects': 1469, 'num_object_clones': 0, 'num_object_copies': 5876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11781, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -634}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:27.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:54:27.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11350", 'reported_seq': 11467, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:23.771576+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:23.771576+0000', 'last_peered': '2024-09-18T11:54:23.771576+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:23.771576+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11942547, 'num_objects': 1459, 'num_object_clones': 0, 'num_object_copies': 5836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1459, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11349, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -581}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:27.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:54:27.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11456", 'reported_seq': 11550, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:24.700764+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:54:24.700764+0000', 'last_peered': '2024-09-18T11:54:24.700764+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:54:24.700764+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12039414, 'num_objects': 1470, 'num_object_clones': 0, 'num_object_copies': 5880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11454, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:27.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:54:27.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11418", 'reported_seq': 11539, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:23.813326+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:23.813326+0000', 'last_peered': '2024-09-18T11:54:23.813326+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:23.813326+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12096758, 'num_objects': 1477, 'num_object_clones': 0, 'num_object_copies': 5908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11416, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -575}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:27.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:54:27.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11433", 'reported_seq': 11552, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:23.871478+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:23.871478+0000', 'last_peered': '2024-09-18T11:54:23.871478+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:23.871478+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11083776, 'num_objects': 1353, 'num_object_clones': 0, 'num_object_copies': 5412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11433, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -766}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:27.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:54:27.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11579", 'reported_seq': 11686, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:24.620440+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:24.620440+0000', 'last_peered': '2024-09-18T11:54:24.620440+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:24.620440+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11769078, 'num_objects': 1437, 'num_object_clones': 0, 'num_object_copies': 5748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11577, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -679}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:27.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:54:27.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11380", 'reported_seq': 13682, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:24.793786+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:54:24.793786+0000', 'last_peered': '2024-09-18T11:54:24.793786+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:54:24.793786+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12244214, 'num_objects': 1495, 'num_object_clones': 0, 'num_object_copies': 5980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 920, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11378, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1118, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 575}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:27.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:54:27.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11170", 'reported_seq': 11284, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:23.772926+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:23.772926+0000', 'last_peered': '2024-09-18T11:54:23.772926+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:23.772926+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11662582, 'num_objects': 1424, 'num_object_clones': 0, 'num_object_copies': 5696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11168, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -539}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:27.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:54:27.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11526", 'reported_seq': 11645, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:23.579136+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:23.579136+0000', 'last_peered': '2024-09-18T11:54:23.579136+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:23.579136+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12341105, 'num_objects': 1507, 'num_object_clones': 0, 'num_object_copies': 6028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11523, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -527}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:27.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:54:27.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11633", 'reported_seq': 11726, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:23.854365+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:23.854365+0000', 'last_peered': '2024-09-18T11:54:23.854365+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:23.854365+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11451003, 'num_objects': 1398, 'num_object_clones': 0, 'num_object_copies': 5592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11632, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:27.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:54:27.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11547", 'reported_seq': 11645, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:23.534283+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:23.534283+0000', 'last_peered': '2024-09-18T11:54:23.534283+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:23.534283+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11631227, 'num_objects': 1420, 'num_object_clones': 0, 'num_object_copies': 5680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11546, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -626}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:27.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:54:27.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11574", 'reported_seq': 11699, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:23.802639+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:23.802639+0000', 'last_peered': '2024-09-18T11:54:23.802639+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:23.802639+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12416246, 'num_objects': 1516, 'num_object_clones': 0, 'num_object_copies': 6064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11572, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -607}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:27.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:54:27.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11256", 'reported_seq': 11346, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:24.668555+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:24.668555+0000', 'last_peered': '2024-09-18T11:54:24.668555+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:24.668555+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11883766, 'num_objects': 1451, 'num_object_clones': 0, 'num_object_copies': 5804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11254, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -551}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:27.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:27.874 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:27.874 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25721/92660 objects degraded (27.758%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":665,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":23165,"data_bytes":189719948,"bytes_used":2812624896,"bytes_avail":383734431744,"bytes_total":386547056640,"degraded_objects":25721,"degraded_total":92660,"degraded_ratio":0.27758471832505938,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":200},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:27.900 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:27.900 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:28.477 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:28.478 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:28.505 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-09-18T11:54:28.063363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189719948,"num_objects":23165,"num_object_clones":0,"num_object_copies":92660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183922,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7735,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2746704,"kb_used_data":373232,"kb_used_omap":14,"kb_used_meta":2373425,"kb_avail":374740656,"statfs":{"total":386547056640,"available":383734431744,"internally_reserved":0,"allocated":382189568,"data_stored":381454284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430387227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,10,14,17,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16449536,"num_objects":-2008,"num_object_clones":0,"num_object_copies":-8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004005"},"pg_stats":[{"pgid":"1.f","version":"37'11781","reported_seq":11882,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.802135+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:23.802135+0000","last_peered":"2024-09-18T11:54:23.802135+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:23.802135+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12034048,"num_objects":1469,"num_object_clones":0,"num_object_copies":5876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11613","reported_seq":15905,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:25.840273+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:25.840273+0000","last_peered":"2024-09-18T11:54:25.840273+0000","last_clean":"2024-09-18T11:54:25.840273+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:25.840273+0000","last_undegraded":"2024-09-18T11:54:25.840273+0000","last_fullsized":"2024-09-18T11:54:25.840273+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11818230,"num_objects":1443,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11611,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11350","reported_seq":11467,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.771576+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:23.771576+0000","last_peered":"2024-09-18T11:54:23.771576+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:23.771576+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11942547,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11349,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-581}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11456","reported_seq":11550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.700764+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:24.700764+0000","last_peered":"2024-09-18T11:54:24.700764+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:24.700764+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12039414,"num_objects":1470,"num_object_clones":0,"num_object_copies":5880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11418","reported_seq":11539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.813326+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:23.813326+0000","last_peered":"2024-09-18T11:54:23.813326+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:23.813326+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12096758,"num_objects":1477,"num_object_clones":0,"num_object_copies":5908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11433","reported_seq":11552,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.871478+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:23.871478+0000","last_peered":"2024-09-18T11:54:23.871478+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:23.871478+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11083776,"num_objects":1353,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11433,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11579","reported_seq":11686,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.620440+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:24.620440+0000","last_peered":"2024-09-18T11:54:24.620440+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:24.620440+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11769078,"num_objects":1437,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11577,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-679}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11380","reported_seq":13682,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.793786+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:24.793786+0000","last_peered":"2024-09-18T11:54:24.793786+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:24.793786+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12244214,"num_objects":1495,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11378,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":575}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11722","reported_seq":16075,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:23.896921+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:23.896921+0000","last_peered":"2024-09-18T11:54:23.896921+0000","last_clean":"2024-09-18T11:54:23.896921+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:23.896921+0000","last_undegraded":"2024-09-18T11:54:23.896921+0000","last_fullsized":"2024-09-18T11:54:23.896921+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11905516,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11719,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11170","reported_seq":11284,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.772926+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:23.772926+0000","last_peered":"2024-09-18T11:54:23.772926+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:23.772926+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11662582,"num_objects":1424,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11168,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11526","reported_seq":11645,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.579136+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:23.579136+0000","last_peered":"2024-09-18T11:54:23.579136+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:23.579136+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12341105,"num_objects":1507,"num_object_clones":0,"num_object_copies":6028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11523,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-527}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11633","reported_seq":11726,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.854365+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:23.854365+0000","last_peered":"2024-09-18T11:54:23.854365+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:23.854365+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11451003,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11632,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-686}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11547","reported_seq":11645,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.534283+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:23.534283+0000","last_peered":"2024-09-18T11:54:23.534283+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:23.534283+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11631227,"num_objects":1420,"num_object_clones":0,"num_object_copies":5680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11546,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-626}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11574","reported_seq":11699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:23.802639+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:23.802639+0000","last_peered":"2024-09-18T11:54:23.802639+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:23.802639+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12416246,"num_objects":1516,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11572,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-607}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11511","reported_seq":15800,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:25.955026+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:25.955026+0000","last_peered":"2024-09-18T11:54:25.955026+0000","last_clean":"2024-09-18T11:54:25.955026+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:25.955026+0000","last_undegraded":"2024-09-18T11:54:25.955026+0000","last_fullsized":"2024-09-18T11:54:25.955026+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11400438,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11256","reported_seq":11346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:24.668555+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:24.668555+0000","last_peered":"2024-09-18T11:54:24.668555+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:24.668555+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11883766,"num_objects":1451,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-551}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189719948,"num_objects":23165,"num_object_clones":0,"num_object_copies":92660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183922,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7735,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":378716160,"data_stored":378716160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683764,"kb_used_data":93356,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93688076,"statfs":{"total":96636764160,"available":95936589824,"internally_reserved":0,"allocated":95596544,"data_stored":95412723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,5,3,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":4.923},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.22600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":4.899},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":4.9969999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":5.0129999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":5.0350000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692240,"kb_used_data":94024,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93679600,"statfs":{"total":96636764160,"available":95927910400,"internally_reserved":0,"allocated":96280576,"data_stored":96096755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,1,4,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":0.40699999999999997},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":0.435}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.6480000000000001},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.61599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:53:21 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":4.8010000000000002,"15min":3.492},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.816000000000001,"5min":17.001000000000001,"15min":17.001000000000001},"last":12.000999999999999},{"interface":"front","average":{"1min":8.0709999999999997,"5min":4.774,"15min":3.0390000000000001},"min":{"1min":0.371,"5min":0.217,"15min":0.217},"max":{"1min":20.728999999999999,"5min":20.728999999999999,"15min":20.728999999999999},"last":7.1369999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682900,"kb_used_data":92556,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93688940,"statfs":{"total":96636764160,"available":95937474560,"internally_reserved":0,"allocated":94777344,"data_stored":94593523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":6.4329999999999998},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":5.4359999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":6.1139999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.29199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":5.5960000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.8940000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687800,"kb_used_data":93296,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93684040,"statfs":{"total":96636764160,"available":95932456960,"internally_reserved":0,"allocated":95535104,"data_stored":95351283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.133},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.93300000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":2.7229999999999999},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.2199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":7.0960000000000001},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.2590000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94408704,"data_stored":94408704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":93822976,"data_stored":93822976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95293440,"data_stored":95293440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":95191040,"data_stored":95191040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:31.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:32.142 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:32.143 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:32.160 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-18T11:54:32.064915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":181314956,"num_objects":22139,"num_object_clones":0,"num_object_copies":88556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22139,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184948,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7792,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726444,"kb_used_data":357096,"kb_used_omap":14,"kb_used_meta":2369265,"kb_avail":374760916,"statfs":{"total":386547056640,"available":383755177984,"internally_reserved":0,"allocated":365666304,"data_stored":364931020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2426127387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,9,14,17,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17776640,"num_objects":-2170,"num_object_clones":0,"num_object_copies":-8680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2170,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004181"},"pg_stats":[{"pgid":"1.f","version":"37'11840","reported_seq":11941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.921643+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:28.921643+0000","last_peered":"2024-09-18T11:54:28.921643+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:28.921643+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11550720,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11840,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11673","reported_seq":15965,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:30.966543+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:30.966543+0000","last_peered":"2024-09-18T11:54:30.966543+0000","last_clean":"2024-09-18T11:54:30.966543+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:30.966543+0000","last_undegraded":"2024-09-18T11:54:30.966543+0000","last_fullsized":"2024-09-18T11:54:30.966543+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11326710,"num_objects":1383,"num_object_clones":0,"num_object_copies":5532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11671,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11421","reported_seq":11538,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.701174+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:28.701174+0000","last_peered":"2024-09-18T11:54:28.701174+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:28.701174+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11360915,"num_objects":1388,"num_object_clones":0,"num_object_copies":5552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11420,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-652}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11520","reported_seq":11614,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.647000+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:29.647000+0000","last_peered":"2024-09-18T11:54:29.647000+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:29.647000+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11515126,"num_objects":1406,"num_object_clones":0,"num_object_copies":5624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-695}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11507","reported_seq":11628,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.877414+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:28.877414+0000","last_peered":"2024-09-18T11:54:28.877414+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:28.877414+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11367670,"num_objects":1388,"num_object_clones":0,"num_object_copies":5552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11505,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11495","reported_seq":11614,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.809944+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:28.809944+0000","last_peered":"2024-09-18T11:54:28.809944+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:28.809944+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10575872,"num_objects":1291,"num_object_clones":0,"num_object_copies":5164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11495,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11648","reported_seq":11754,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.792055+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:29.792055+0000","last_peered":"2024-09-18T11:54:29.792055+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:29.792055+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11203830,"num_objects":1368,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11449","reported_seq":13865,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.846796+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:29.846796+0000","last_peered":"2024-09-18T11:54:29.846796+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:29.846796+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678966,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11783","reported_seq":16136,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:28.895335+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:28.895335+0000","last_peered":"2024-09-18T11:54:28.895335+0000","last_clean":"2024-09-18T11:54:28.895335+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:28.895335+0000","last_undegraded":"2024-09-18T11:54:28.895335+0000","last_fullsized":"2024-09-18T11:54:28.895335+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11405804,"num_objects":1393,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11780,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11231","reported_seq":11345,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.662015+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:28.662015+0000","last_peered":"2024-09-18T11:54:28.662015+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:28.662015+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11162870,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11229,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-600}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11577","reported_seq":11698,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.833212+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:28.833212+0000","last_peered":"2024-09-18T11:54:28.833212+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:28.833212+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11923313,"num_objects":1456,"num_object_clones":0,"num_object_copies":5824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11574,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11696","reported_seq":11789,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.857466+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:28.857466+0000","last_peered":"2024-09-18T11:54:28.857466+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:28.857466+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10934907,"num_objects":1335,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11695,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11611","reported_seq":11709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.851397+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:28.851397+0000","last_peered":"2024-09-18T11:54:28.851397+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:28.851397+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11106939,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11610,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11638","reported_seq":11763,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.016383+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:29.016383+0000","last_peered":"2024-09-18T11:54:29.016383+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:29.016383+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891958,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11636,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-671}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11577","reported_seq":15866,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:31.026979+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:31.026979+0000","last_peered":"2024-09-18T11:54:31.026979+0000","last_clean":"2024-09-18T11:54:31.026979+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:31.026979+0000","last_undegraded":"2024-09-18T11:54:31.026979+0000","last_fullsized":"2024-09-18T11:54:31.026979+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10859766,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11309","reported_seq":11399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.685436+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:29.685436+0000","last_peered":"2024-09-18T11:54:29.685436+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:29.685436+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449590,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":181314956,"num_objects":22139,"num_object_clones":0,"num_object_copies":88556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22139,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184948,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7792,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362045440,"data_stored":362045440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679788,"kb_used_data":89364,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93692052,"statfs":{"total":96636764160,"available":95940661248,"internally_reserved":0,"allocated":91508736,"data_stored":91324915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,6,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.569},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.968}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":18.102},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.9770000000000003},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.32000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675732,"kb_used_data":89996,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93696108,"statfs":{"total":96636764160,"available":95944814592,"internally_reserved":0,"allocated":92155904,"data_stored":91972083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":1.129},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.159}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.021999999999998},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.7029999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":0.40799999999999997},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":0.30199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682972,"kb_used_data":88460,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93688868,"statfs":{"total":96636764160,"available":95937400832,"internally_reserved":0,"allocated":90583040,"data_stored":90399219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":11.526},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.01}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":11.747999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":19.265999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":19.225000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":10.855}]}]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687952,"kb_used_data":89276,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93683888,"statfs":{"total":96636764160,"available":95932301312,"internally_reserved":0,"allocated":91418624,"data_stored":91234803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.133},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.93300000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":2.7229999999999999},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.2199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":7.0960000000000001},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.2590000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90304512,"data_stored":90304512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":89579520,"data_stored":89579520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":91144192,"data_stored":91144192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":91017216,"data_stored":91017216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:32.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:32.745 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:32.746 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:32.776 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-18T11:54:32.064915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":181314956,"num_objects":22139,"num_object_clones":0,"num_object_copies":88556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22139,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184948,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7792,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726444,"kb_used_data":357096,"kb_used_omap":14,"kb_used_meta":2369265,"kb_avail":374760916,"statfs":{"total":386547056640,"available":383755177984,"internally_reserved":0,"allocated":365666304,"data_stored":364931020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2426127387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,9,14,17,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17776640,"num_objects":-2170,"num_object_clones":0,"num_object_copies":-8680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2170,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004181"},"pg_stats":[{"pgid":"1.f","version":"37'11840","reported_seq":11941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.921643+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:28.921643+0000","last_peered":"2024-09-18T11:54:28.921643+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:28.921643+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11550720,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11840,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11673","reported_seq":15965,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:30.966543+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:30.966543+0000","last_peered":"2024-09-18T11:54:30.966543+0000","last_clean":"2024-09-18T11:54:30.966543+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:30.966543+0000","last_undegraded":"2024-09-18T11:54:30.966543+0000","last_fullsized":"2024-09-18T11:54:30.966543+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11326710,"num_objects":1383,"num_object_clones":0,"num_object_copies":5532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11671,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11421","reported_seq":11538,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.701174+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:28.701174+0000","last_peered":"2024-09-18T11:54:28.701174+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:28.701174+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11360915,"num_objects":1388,"num_object_clones":0,"num_object_copies":5552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11420,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-652}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11520","reported_seq":11614,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.647000+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:29.647000+0000","last_peered":"2024-09-18T11:54:29.647000+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:29.647000+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11515126,"num_objects":1406,"num_object_clones":0,"num_object_copies":5624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-695}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11507","reported_seq":11628,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.877414+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:28.877414+0000","last_peered":"2024-09-18T11:54:28.877414+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:28.877414+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11367670,"num_objects":1388,"num_object_clones":0,"num_object_copies":5552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11505,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11495","reported_seq":11614,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.809944+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:28.809944+0000","last_peered":"2024-09-18T11:54:28.809944+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:28.809944+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10575872,"num_objects":1291,"num_object_clones":0,"num_object_copies":5164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11495,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11648","reported_seq":11754,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.792055+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:29.792055+0000","last_peered":"2024-09-18T11:54:29.792055+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:29.792055+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11203830,"num_objects":1368,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11449","reported_seq":13865,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.846796+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:29.846796+0000","last_peered":"2024-09-18T11:54:29.846796+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:29.846796+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678966,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11783","reported_seq":16136,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:28.895335+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:28.895335+0000","last_peered":"2024-09-18T11:54:28.895335+0000","last_clean":"2024-09-18T11:54:28.895335+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:28.895335+0000","last_undegraded":"2024-09-18T11:54:28.895335+0000","last_fullsized":"2024-09-18T11:54:28.895335+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11405804,"num_objects":1393,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11780,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11231","reported_seq":11345,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.662015+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:28.662015+0000","last_peered":"2024-09-18T11:54:28.662015+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:28.662015+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11162870,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11229,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-600}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11577","reported_seq":11698,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.833212+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:28.833212+0000","last_peered":"2024-09-18T11:54:28.833212+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:28.833212+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11923313,"num_objects":1456,"num_object_clones":0,"num_object_copies":5824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11574,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11696","reported_seq":11789,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.857466+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:28.857466+0000","last_peered":"2024-09-18T11:54:28.857466+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:28.857466+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10934907,"num_objects":1335,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11695,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11611","reported_seq":11709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.851397+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:28.851397+0000","last_peered":"2024-09-18T11:54:28.851397+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:28.851397+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11106939,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11610,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11638","reported_seq":11763,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.016383+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:29.016383+0000","last_peered":"2024-09-18T11:54:29.016383+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:29.016383+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891958,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11636,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-671}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11577","reported_seq":15866,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:31.026979+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:31.026979+0000","last_peered":"2024-09-18T11:54:31.026979+0000","last_clean":"2024-09-18T11:54:31.026979+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:31.026979+0000","last_undegraded":"2024-09-18T11:54:31.026979+0000","last_fullsized":"2024-09-18T11:54:31.026979+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10859766,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11309","reported_seq":11399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.685436+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:29.685436+0000","last_peered":"2024-09-18T11:54:29.685436+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:29.685436+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449590,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":181314956,"num_objects":22139,"num_object_clones":0,"num_object_copies":88556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22139,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184948,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7792,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362045440,"data_stored":362045440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679788,"kb_used_data":89364,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93692052,"statfs":{"total":96636764160,"available":95940661248,"internally_reserved":0,"allocated":91508736,"data_stored":91324915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,6,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.569},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.968}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":18.102},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.9770000000000003},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.32000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675732,"kb_used_data":89996,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93696108,"statfs":{"total":96636764160,"available":95944814592,"internally_reserved":0,"allocated":92155904,"data_stored":91972083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":1.129},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.159}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.021999999999998},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.7029999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":0.40799999999999997},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":0.30199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682972,"kb_used_data":88460,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93688868,"statfs":{"total":96636764160,"available":95937400832,"internally_reserved":0,"allocated":90583040,"data_stored":90399219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":11.526},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.01}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":11.747999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":19.265999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":19.225000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":10.855}]}]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687952,"kb_used_data":89276,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93683888,"statfs":{"total":96636764160,"available":95932301312,"internally_reserved":0,"allocated":91418624,"data_stored":91234803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.133},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.93300000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":2.7229999999999999},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.2199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":7.0960000000000001},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.2590000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90304512,"data_stored":90304512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":89579520,"data_stored":89579520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":91144192,"data_stored":91144192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":91017216,"data_stored":91017216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:32.777 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:54:32.777 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11840", 'reported_seq': 11941, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:28.921643+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:28.921643+0000', 'last_peered': '2024-09-18T11:54:28.921643+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:28.921643+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11550720, 'num_objects': 1410, 'num_object_clones': 0, 'num_object_copies': 5640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11840, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -693}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:32.777 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:54:32.777 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11421", 'reported_seq': 11538, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:28.701174+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:28.701174+0000', 'last_peered': '2024-09-18T11:54:28.701174+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:28.701174+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11360915, 'num_objects': 1388, 'num_object_clones': 0, 'num_object_copies': 5552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1388, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11420, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -652}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:32.778 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:54:32.778 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11520", 'reported_seq': 11614, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:29.647000+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:54:29.647000+0000', 'last_peered': '2024-09-18T11:54:29.647000+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:54:29.647000+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11515126, 'num_objects': 1406, 'num_object_clones': 0, 'num_object_copies': 5624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11518, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -695}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:32.778 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:54:32.778 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11507", 'reported_seq': 11628, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:28.877414+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:28.877414+0000', 'last_peered': '2024-09-18T11:54:28.877414+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:28.877414+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11367670, 'num_objects': 1388, 'num_object_clones': 0, 'num_object_copies': 5552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11505, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -664}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:32.778 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:54:32.778 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11495", 'reported_seq': 11614, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:28.809944+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:28.809944+0000', 'last_peered': '2024-09-18T11:54:28.809944+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:28.809944+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10575872, 'num_objects': 1291, 'num_object_clones': 0, 'num_object_copies': 5164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11495, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -828}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:32.778 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:54:32.778 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11648", 'reported_seq': 11754, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:29.792055+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:29.792055+0000', 'last_peered': '2024-09-18T11:54:29.792055+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:29.792055+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11203830, 'num_objects': 1368, 'num_object_clones': 0, 'num_object_copies': 5472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11646, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -748}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:32.778 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11449", 'reported_seq': 13865, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:29.846796+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:54:29.846796+0000', 'last_peered': '2024-09-18T11:54:29.846796+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:54:29.846796+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11678966, 'num_objects': 1426, 'num_object_clones': 0, 'num_object_copies': 5704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 863, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1426, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11447, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1175, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 563}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11231", 'reported_seq': 11345, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:28.662015+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:28.662015+0000', 'last_peered': '2024-09-18T11:54:28.662015+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:28.662015+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11162870, 'num_objects': 1363, 'num_object_clones': 0, 'num_object_copies': 5452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11229, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -600}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11577", 'reported_seq': 11698, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:28.833212+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:28.833212+0000', 'last_peered': '2024-09-18T11:54:28.833212+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:28.833212+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11923313, 'num_objects': 1456, 'num_object_clones': 0, 'num_object_copies': 5824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11574, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -578}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11696", 'reported_seq': 11789, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:28.857466+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:28.857466+0000', 'last_peered': '2024-09-18T11:54:28.857466+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:28.857466+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10934907, 'num_objects': 1335, 'num_object_clones': 0, 'num_object_copies': 5340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11695, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11611", 'reported_seq': 11709, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:28.851397+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:28.851397+0000', 'last_peered': '2024-09-18T11:54:28.851397+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:28.851397+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11106939, 'num_objects': 1356, 'num_object_clones': 0, 'num_object_copies': 5424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11610, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -690}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:54:32.779 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11638", 'reported_seq': 11763, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:29.016383+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:29.016383+0000', 'last_peered': '2024-09-18T11:54:29.016383+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:29.016383+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11891958, 'num_objects': 1452, 'num_object_clones': 0, 'num_object_copies': 5808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11636, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:32.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:54:32.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11309", 'reported_seq': 11399, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:29.685436+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:29.685436+0000', 'last_peered': '2024-09-18T11:54:29.685436+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:29.685436+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11449590, 'num_objects': 1398, 'num_object_clones': 0, 'num_object_copies': 5592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11307, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:32.780 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:33.471 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25664/88556 objects degraded (28.981%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":671,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22139,"data_bytes":181314956,"bytes_used":2791878656,"bytes_avail":383755177984,"bytes_total":386547056640,"degraded_objects":25664,"degraded_total":88556,"degraded_ratio":0.28980532092687111,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:33.502 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:33.502 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:34.052 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:34.053 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:34.071 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-18T11:54:32.064915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":181314956,"num_objects":22139,"num_object_clones":0,"num_object_copies":88556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22139,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184948,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7792,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726444,"kb_used_data":357096,"kb_used_omap":14,"kb_used_meta":2369265,"kb_avail":374760916,"statfs":{"total":386547056640,"available":383755177984,"internally_reserved":0,"allocated":365666304,"data_stored":364931020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2426127387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,9,14,17,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17776640,"num_objects":-2170,"num_object_clones":0,"num_object_copies":-8680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2170,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004181"},"pg_stats":[{"pgid":"1.f","version":"37'11840","reported_seq":11941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.921643+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:28.921643+0000","last_peered":"2024-09-18T11:54:28.921643+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:28.921643+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11550720,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11840,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11673","reported_seq":15965,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:30.966543+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:30.966543+0000","last_peered":"2024-09-18T11:54:30.966543+0000","last_clean":"2024-09-18T11:54:30.966543+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:30.966543+0000","last_undegraded":"2024-09-18T11:54:30.966543+0000","last_fullsized":"2024-09-18T11:54:30.966543+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11326710,"num_objects":1383,"num_object_clones":0,"num_object_copies":5532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11671,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11421","reported_seq":11538,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.701174+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:28.701174+0000","last_peered":"2024-09-18T11:54:28.701174+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:28.701174+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11360915,"num_objects":1388,"num_object_clones":0,"num_object_copies":5552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11420,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-652}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11520","reported_seq":11614,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.647000+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:29.647000+0000","last_peered":"2024-09-18T11:54:29.647000+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:29.647000+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11515126,"num_objects":1406,"num_object_clones":0,"num_object_copies":5624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-695}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11507","reported_seq":11628,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.877414+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:28.877414+0000","last_peered":"2024-09-18T11:54:28.877414+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:28.877414+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11367670,"num_objects":1388,"num_object_clones":0,"num_object_copies":5552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11505,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11495","reported_seq":11614,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.809944+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:28.809944+0000","last_peered":"2024-09-18T11:54:28.809944+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:28.809944+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10575872,"num_objects":1291,"num_object_clones":0,"num_object_copies":5164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11495,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11648","reported_seq":11754,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.792055+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:29.792055+0000","last_peered":"2024-09-18T11:54:29.792055+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:29.792055+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11203830,"num_objects":1368,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11449","reported_seq":13865,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.846796+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:29.846796+0000","last_peered":"2024-09-18T11:54:29.846796+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:29.846796+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678966,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11783","reported_seq":16136,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:28.895335+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:28.895335+0000","last_peered":"2024-09-18T11:54:28.895335+0000","last_clean":"2024-09-18T11:54:28.895335+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:28.895335+0000","last_undegraded":"2024-09-18T11:54:28.895335+0000","last_fullsized":"2024-09-18T11:54:28.895335+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11405804,"num_objects":1393,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11780,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11231","reported_seq":11345,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.662015+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:28.662015+0000","last_peered":"2024-09-18T11:54:28.662015+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:28.662015+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11162870,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11229,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-600}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11577","reported_seq":11698,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.833212+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:28.833212+0000","last_peered":"2024-09-18T11:54:28.833212+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:28.833212+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11923313,"num_objects":1456,"num_object_clones":0,"num_object_copies":5824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11574,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11696","reported_seq":11789,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.857466+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:28.857466+0000","last_peered":"2024-09-18T11:54:28.857466+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:28.857466+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10934907,"num_objects":1335,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11695,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11611","reported_seq":11709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:28.851397+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:28.851397+0000","last_peered":"2024-09-18T11:54:28.851397+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:28.851397+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11106939,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11610,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11638","reported_seq":11763,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.016383+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:29.016383+0000","last_peered":"2024-09-18T11:54:29.016383+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:29.016383+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891958,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11636,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-671}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11577","reported_seq":15866,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:31.026979+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:31.026979+0000","last_peered":"2024-09-18T11:54:31.026979+0000","last_clean":"2024-09-18T11:54:31.026979+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:31.026979+0000","last_undegraded":"2024-09-18T11:54:31.026979+0000","last_fullsized":"2024-09-18T11:54:31.026979+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10859766,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11309","reported_seq":11399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:29.685436+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:29.685436+0000","last_peered":"2024-09-18T11:54:29.685436+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:29.685436+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449590,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":181314956,"num_objects":22139,"num_object_clones":0,"num_object_copies":88556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22139,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184948,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7792,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362045440,"data_stored":362045440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679788,"kb_used_data":89364,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93692052,"statfs":{"total":96636764160,"available":95940661248,"internally_reserved":0,"allocated":91508736,"data_stored":91324915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,6,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.569},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.968}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":18.102},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.9770000000000003},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.32000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675732,"kb_used_data":89996,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93696108,"statfs":{"total":96636764160,"available":95944814592,"internally_reserved":0,"allocated":92155904,"data_stored":91972083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":1.129},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.159}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.021999999999998},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.7029999999999994}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":0.40799999999999997},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":0.30199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682972,"kb_used_data":88460,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93688868,"statfs":{"total":96636764160,"available":95937400832,"internally_reserved":0,"allocated":90583040,"data_stored":90399219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":11.526},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.01}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":11.747999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":19.265999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":19.225000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":10.855}]}]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687952,"kb_used_data":89276,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93683888,"statfs":{"total":96636764160,"available":95932301312,"internally_reserved":0,"allocated":91418624,"data_stored":91234803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.133},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.93300000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":2.7229999999999999},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.2199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":7.0960000000000001},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.2590000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90304512,"data_stored":90304512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":89579520,"data_stored":89579520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":91144192,"data_stored":91144192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":91017216,"data_stored":91017216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:37.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:37.601 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:37.601 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:37.631 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-18T11:54:36.069147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":174089612,"num_objects":21257,"num_object_clones":0,"num_object_copies":85028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21257,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185830,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7844,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160957,"ondisk_log_size":160957,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2710244,"kb_used_data":345076,"kb_used_omap":14,"kb_used_meta":2365105,"kb_avail":374777116,"statfs":{"total":386547056640,"available":383771766784,"internally_reserved":0,"allocated":353357824,"data_stored":352622540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2421867547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,0,9,9,14,11,16,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18767872,"num_objects":-2291,"num_object_clones":0,"num_object_copies":-9164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007059"},"pg_stats":[{"pgid":"1.f","version":"37'11920","reported_seq":12021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.061256+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:34.061256+0000","last_peered":"2024-09-18T11:54:34.061256+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:34.061256+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10895360,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11920,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11673","reported_seq":15965,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:30.966543+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:30.966543+0000","last_peered":"2024-09-18T11:54:30.966543+0000","last_clean":"2024-09-18T11:54:30.966543+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:30.966543+0000","last_undegraded":"2024-09-18T11:54:30.966543+0000","last_fullsized":"2024-09-18T11:54:30.966543+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11326710,"num_objects":1383,"num_object_clones":0,"num_object_copies":5532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11671,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11483","reported_seq":11600,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.718796+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:33.718796+0000","last_peered":"2024-09-18T11:54:33.718796+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:33.718796+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10853011,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11482,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11589","reported_seq":11683,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.821012+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:34.821012+0000","last_peered":"2024-09-18T11:54:34.821012+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:34.821012+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10949878,"num_objects":1337,"num_object_clones":0,"num_object_copies":5348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11587,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11576","reported_seq":11693,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.100594+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:34.100594+0000","last_peered":"2024-09-18T11:54:34.100594+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:34.100594+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10802422,"num_objects":1319,"num_object_clones":0,"num_object_copies":5276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11574,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11556","reported_seq":11675,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.087968+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:34.087968+0000","last_peered":"2024-09-18T11:54:34.087968+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:34.087968+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10076160,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11708","reported_seq":11815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.711201+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:34.711201+0000","last_peered":"2024-09-18T11:54:34.711201+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:34.711201+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10712310,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-808}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11520","reported_seq":14040,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.805806+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:34.805806+0000","last_peered":"2024-09-18T11:54:34.805806+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:34.805806+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11097334,"num_objects":1355,"num_object_clones":0,"num_object_copies":5420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11834","reported_seq":16187,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:33.908708+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:33.908708+0000","last_peered":"2024-09-18T11:54:33.908708+0000","last_clean":"2024-09-18T11:54:33.908708+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:33.908708+0000","last_undegraded":"2024-09-18T11:54:33.908708+0000","last_fullsized":"2024-09-18T11:54:33.908708+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10988012,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11831,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11290","reported_seq":11404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.695267+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:33.695267+0000","last_peered":"2024-09-18T11:54:33.695267+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:33.695267+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679542,"num_objects":1304,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11646","reported_seq":11767,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.037905+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:34.037905+0000","last_peered":"2024-09-18T11:54:34.037905+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:34.037905+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11358065,"num_objects":1387,"num_object_clones":0,"num_object_copies":5548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11643,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11757","reported_seq":11850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.840311+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:33.840311+0000","last_peered":"2024-09-18T11:54:33.840311+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:33.840311+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10435195,"num_objects":1274,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-810}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11667","reported_seq":11765,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.774374+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:33.774374+0000","last_peered":"2024-09-18T11:54:33.774374+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:33.774374+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10648187,"num_objects":1300,"num_object_clones":0,"num_object_copies":5200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11666,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-746}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11692","reported_seq":11816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.024812+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:34.024812+0000","last_peered":"2024-09-18T11:54:34.024812+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:34.024812+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449590,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11690,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-725}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11577","reported_seq":15866,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:31.026979+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:31.026979+0000","last_peered":"2024-09-18T11:54:31.026979+0000","last_clean":"2024-09-18T11:54:31.026979+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:31.026979+0000","last_undegraded":"2024-09-18T11:54:31.026979+0000","last_fullsized":"2024-09-18T11:54:31.026979+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10859766,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11369","reported_seq":11458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.856774+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:34.856774+0000","last_peered":"2024-09-18T11:54:34.856774+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:34.856774+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10958070,"num_objects":1338,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11367,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":174089612,"num_objects":21257,"num_object_clones":0,"num_object_copies":85028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21257,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185830,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7844,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":349765632,"data_stored":349765632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160957,"ondisk_log_size":160957,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679860,"kb_used_data":85288,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93691980,"statfs":{"total":96636764160,"available":95940587520,"internally_reserved":0,"allocated":87334912,"data_stored":87151091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,6,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.569},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.968}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":18.102},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.9770000000000003},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.32000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675888,"kb_used_data":85988,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93695952,"statfs":{"total":96636764160,"available":95944654848,"internally_reserved":0,"allocated":88051712,"data_stored":87867891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,1,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":3.5830000000000002},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":3.6120000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":5.4349999999999996},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":1.1930000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":0.68200000000000005},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":4.5800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682972,"kb_used_data":88460,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93688868,"statfs":{"total":96636764160,"available":95937400832,"internally_reserved":0,"allocated":90583040,"data_stored":90399219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":11.526},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.01}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":11.747999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":19.265999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":19.225000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":10.855}]}]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671524,"kb_used_data":85340,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93700316,"statfs":{"total":96636764160,"available":95949123584,"internally_reserved":0,"allocated":87388160,"data_stored":87204339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,0,1,0,0,1,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":17.617999999999999},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":3.875}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.639},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":3.0899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":9.8230000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86245376,"data_stored":86245376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":89579520,"data_stored":89579520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":87056384,"data_stored":87056384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86884352,"data_stored":86884352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:37.633 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:38.167 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:38.167 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:38.194 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-09-18T11:54:38.069638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":173155724,"num_objects":21143,"num_object_clones":0,"num_object_copies":84572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21143,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185944,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7844,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2706532,"kb_used_data":341348,"kb_used_omap":14,"kb_used_meta":2365105,"kb_avail":374780828,"statfs":{"total":386547056640,"available":383775567872,"internally_reserved":0,"allocated":349540352,"data_stored":348805068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2421867547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,0,9,16,8,11,20,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16564224,"num_objects":-2022,"num_object_clones":0,"num_object_copies":-8088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2022,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007043"},"pg_stats":[{"pgid":"1.f","version":"37'11920","reported_seq":12021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.061256+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:34.061256+0000","last_peered":"2024-09-18T11:54:34.061256+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:34.061256+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10895360,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11920,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11727","reported_seq":16019,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:35.773863+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:35.773863+0000","last_peered":"2024-09-18T11:54:35.773863+0000","last_clean":"2024-09-18T11:54:35.773863+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:35.773863+0000","last_undegraded":"2024-09-18T11:54:35.773863+0000","last_fullsized":"2024-09-18T11:54:35.773863+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10884342,"num_objects":1329,"num_object_clones":0,"num_object_copies":5316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11483","reported_seq":11600,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.718796+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:33.718796+0000","last_peered":"2024-09-18T11:54:33.718796+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:33.718796+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10853011,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11482,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11589","reported_seq":11683,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.821012+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:34.821012+0000","last_peered":"2024-09-18T11:54:34.821012+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:34.821012+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10949878,"num_objects":1337,"num_object_clones":0,"num_object_copies":5348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11587,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11576","reported_seq":11693,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.100594+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:34.100594+0000","last_peered":"2024-09-18T11:54:34.100594+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:34.100594+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10802422,"num_objects":1319,"num_object_clones":0,"num_object_copies":5276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11574,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11556","reported_seq":11675,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.087968+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:34.087968+0000","last_peered":"2024-09-18T11:54:34.087968+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:34.087968+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10076160,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11708","reported_seq":11815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.711201+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:34.711201+0000","last_peered":"2024-09-18T11:54:34.711201+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:34.711201+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10712310,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-808}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11520","reported_seq":14040,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.805806+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:34.805806+0000","last_peered":"2024-09-18T11:54:34.805806+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:34.805806+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11097334,"num_objects":1355,"num_object_clones":0,"num_object_copies":5420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11834","reported_seq":16187,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:33.908708+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:33.908708+0000","last_peered":"2024-09-18T11:54:33.908708+0000","last_clean":"2024-09-18T11:54:33.908708+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:33.908708+0000","last_undegraded":"2024-09-18T11:54:33.908708+0000","last_fullsized":"2024-09-18T11:54:33.908708+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10988012,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11831,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11290","reported_seq":11404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.695267+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:33.695267+0000","last_peered":"2024-09-18T11:54:33.695267+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:33.695267+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679542,"num_objects":1304,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11646","reported_seq":11767,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.037905+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:34.037905+0000","last_peered":"2024-09-18T11:54:34.037905+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:34.037905+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11358065,"num_objects":1387,"num_object_clones":0,"num_object_copies":5548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11643,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11757","reported_seq":11850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.840311+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:33.840311+0000","last_peered":"2024-09-18T11:54:33.840311+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:33.840311+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10435195,"num_objects":1274,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-810}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11667","reported_seq":11765,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.774374+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:33.774374+0000","last_peered":"2024-09-18T11:54:33.774374+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:33.774374+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10648187,"num_objects":1300,"num_object_clones":0,"num_object_copies":5200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11666,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-746}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11692","reported_seq":11816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.024812+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:34.024812+0000","last_peered":"2024-09-18T11:54:34.024812+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:34.024812+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449590,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11690,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-725}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11637","reported_seq":15926,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:36.128042+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:36.128042+0000","last_peered":"2024-09-18T11:54:36.128042+0000","last_clean":"2024-09-18T11:54:36.128042+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:36.128042+0000","last_undegraded":"2024-09-18T11:54:36.128042+0000","last_fullsized":"2024-09-18T11:54:36.128042+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10368246,"num_objects":1266,"num_object_clones":0,"num_object_copies":5064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11369","reported_seq":11458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.856774+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:34.856774+0000","last_peered":"2024-09-18T11:54:34.856774+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:34.856774+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10958070,"num_objects":1338,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11367,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":173155724,"num_objects":21143,"num_object_clones":0,"num_object_copies":84572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21143,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185944,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7844,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":345878528,"data_stored":345878528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679860,"kb_used_data":85288,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93691980,"statfs":{"total":96636764160,"available":95940587520,"internally_reserved":0,"allocated":87334912,"data_stored":87151091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,6,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.569},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.968}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":18.102},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.9770000000000003},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.32000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675888,"kb_used_data":85988,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93695952,"statfs":{"total":96636764160,"available":95944654848,"internally_reserved":0,"allocated":88051712,"data_stored":87867891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,1,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":3.5830000000000002},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":3.6120000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":5.4349999999999996},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":1.1930000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":0.68200000000000005},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":4.5800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679260,"kb_used_data":84732,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93692580,"statfs":{"total":96636764160,"available":95941201920,"internally_reserved":0,"allocated":86765568,"data_stored":86581747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":3.048},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.8330000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":8.3469999999999995},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.7800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":2.8029999999999999},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":2.8650000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671524,"kb_used_data":85340,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93700316,"statfs":{"total":96636764160,"available":95949123584,"internally_reserved":0,"allocated":87388160,"data_stored":87204339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,0,1,0,0,1,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":17.617999999999999},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":3.875}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.639},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":3.0899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":9.8230000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86245376,"data_stored":86245376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":85692416,"data_stored":85692416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":87056384,"data_stored":87056384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86884352,"data_stored":86884352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:38.195 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:54:38.195 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11920", 'reported_seq': 12021, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:34.061256+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:34.061256+0000', 'last_peered': '2024-09-18T11:54:34.061256+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:34.061256+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10895360, 'num_objects': 1330, 'num_object_clones': 0, 'num_object_copies': 5320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11920, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -773}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:38.195 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:54:38.195 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11483", 'reported_seq': 11600, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:33.718796+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:33.718796+0000', 'last_peered': '2024-09-18T11:54:33.718796+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:33.718796+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10853011, 'num_objects': 1326, 'num_object_clones': 0, 'num_object_copies': 5304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1326, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11482, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -714}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:38.195 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:54:38.195 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11589", 'reported_seq': 11683, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:34.821012+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:54:34.821012+0000', 'last_peered': '2024-09-18T11:54:34.821012+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:54:34.821012+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10949878, 'num_objects': 1337, 'num_object_clones': 0, 'num_object_copies': 5348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11587, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -764}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:38.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:54:38.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11576", 'reported_seq': 11693, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:34.100594+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:34.100594+0000', 'last_peered': '2024-09-18T11:54:34.100594+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:34.100594+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10802422, 'num_objects': 1319, 'num_object_clones': 0, 'num_object_copies': 5276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11574, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -733}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:38.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:54:38.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11556", 'reported_seq': 11675, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:34.087968+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:34.087968+0000', 'last_peered': '2024-09-18T11:54:34.087968+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:34.087968+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10076160, 'num_objects': 1230, 'num_object_clones': 0, 'num_object_copies': 4920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11556, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -889}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:38.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:54:38.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11708", 'reported_seq': 11815, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:34.711201+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:34.711201+0000', 'last_peered': '2024-09-18T11:54:34.711201+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:34.711201+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10712310, 'num_objects': 1308, 'num_object_clones': 0, 'num_object_copies': 5232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11706, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -808}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:38.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:54:38.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11520", 'reported_seq': 14040, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:34.805806+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:54:34.805806+0000', 'last_peered': '2024-09-18T11:54:34.805806+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:54:34.805806+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11097334, 'num_objects': 1355, 'num_object_clones': 0, 'num_object_copies': 5420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 811, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11518, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1227, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 544}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:38.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:54:38.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11290", 'reported_seq': 11404, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:33.695267+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:33.695267+0000', 'last_peered': '2024-09-18T11:54:33.695267+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:33.695267+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10679542, 'num_objects': 1304, 'num_object_clones': 0, 'num_object_copies': 5216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11288, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -659}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:38.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:54:38.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11646", 'reported_seq': 11767, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:34.037905+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:34.037905+0000', 'last_peered': '2024-09-18T11:54:34.037905+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:34.037905+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11358065, 'num_objects': 1387, 'num_object_clones': 0, 'num_object_copies': 5548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11643, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -647}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:38.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:54:38.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11757", 'reported_seq': 11850, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:33.840311+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:33.840311+0000', 'last_peered': '2024-09-18T11:54:33.840311+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:33.840311+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10435195, 'num_objects': 1274, 'num_object_clones': 0, 'num_object_copies': 5096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11756, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -810}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:38.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:54:38.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11667", 'reported_seq': 11765, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:33.774374+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:33.774374+0000', 'last_peered': '2024-09-18T11:54:33.774374+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:33.774374+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10648187, 'num_objects': 1300, 'num_object_clones': 0, 'num_object_copies': 5200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11666, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -746}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:38.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:54:38.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11692", 'reported_seq': 11816, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:34.024812+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:34.024812+0000', 'last_peered': '2024-09-18T11:54:34.024812+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:34.024812+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11449590, 'num_objects': 1398, 'num_object_clones': 0, 'num_object_copies': 5592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11690, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -725}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:38.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:54:38.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11369", 'reported_seq': 11458, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:34.856774+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:34.856774+0000', 'last_peered': '2024-09-18T11:54:34.856774+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:34.856774+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10958070, 'num_objects': 1338, 'num_object_clones': 0, 'num_object_copies': 5352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11367, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -664}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:38.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:38.858 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:38.858 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25612/84572 objects degraded (30.284%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":676,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":21143,"data_bytes":173155724,"bytes_used":2771488768,"bytes_avail":383775567872,"bytes_total":386547056640,"degraded_objects":25612,"degraded_total":84572,"degraded_ratio":0.3028425483611597,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":168},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:38.881 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:38.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:39.442 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:39.443 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:39.468 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-09-18T11:54:38.069638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":173155724,"num_objects":21143,"num_object_clones":0,"num_object_copies":84572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21143,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185944,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7844,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2706532,"kb_used_data":341348,"kb_used_omap":14,"kb_used_meta":2365105,"kb_avail":374780828,"statfs":{"total":386547056640,"available":383775567872,"internally_reserved":0,"allocated":349540352,"data_stored":348805068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2421867547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,0,9,16,8,11,20,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16564224,"num_objects":-2022,"num_object_clones":0,"num_object_copies":-8088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2022,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007043"},"pg_stats":[{"pgid":"1.f","version":"37'11920","reported_seq":12021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.061256+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:34.061256+0000","last_peered":"2024-09-18T11:54:34.061256+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:34.061256+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10895360,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11920,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11727","reported_seq":16019,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:35.773863+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:35.773863+0000","last_peered":"2024-09-18T11:54:35.773863+0000","last_clean":"2024-09-18T11:54:35.773863+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:35.773863+0000","last_undegraded":"2024-09-18T11:54:35.773863+0000","last_fullsized":"2024-09-18T11:54:35.773863+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10884342,"num_objects":1329,"num_object_clones":0,"num_object_copies":5316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11483","reported_seq":11600,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.718796+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:33.718796+0000","last_peered":"2024-09-18T11:54:33.718796+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:33.718796+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10853011,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11482,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11589","reported_seq":11683,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.821012+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:34.821012+0000","last_peered":"2024-09-18T11:54:34.821012+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:34.821012+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10949878,"num_objects":1337,"num_object_clones":0,"num_object_copies":5348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11587,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11576","reported_seq":11693,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.100594+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:34.100594+0000","last_peered":"2024-09-18T11:54:34.100594+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:34.100594+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10802422,"num_objects":1319,"num_object_clones":0,"num_object_copies":5276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11574,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11556","reported_seq":11675,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.087968+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:34.087968+0000","last_peered":"2024-09-18T11:54:34.087968+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:34.087968+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10076160,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11708","reported_seq":11815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.711201+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:34.711201+0000","last_peered":"2024-09-18T11:54:34.711201+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:34.711201+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10712310,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-808}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11520","reported_seq":14040,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.805806+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:34.805806+0000","last_peered":"2024-09-18T11:54:34.805806+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:34.805806+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11097334,"num_objects":1355,"num_object_clones":0,"num_object_copies":5420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11834","reported_seq":16187,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:33.908708+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:33.908708+0000","last_peered":"2024-09-18T11:54:33.908708+0000","last_clean":"2024-09-18T11:54:33.908708+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:33.908708+0000","last_undegraded":"2024-09-18T11:54:33.908708+0000","last_fullsized":"2024-09-18T11:54:33.908708+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10988012,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11831,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11290","reported_seq":11404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.695267+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:33.695267+0000","last_peered":"2024-09-18T11:54:33.695267+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:33.695267+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679542,"num_objects":1304,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11646","reported_seq":11767,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.037905+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:34.037905+0000","last_peered":"2024-09-18T11:54:34.037905+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:34.037905+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11358065,"num_objects":1387,"num_object_clones":0,"num_object_copies":5548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11643,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11757","reported_seq":11850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.840311+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:33.840311+0000","last_peered":"2024-09-18T11:54:33.840311+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:33.840311+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10435195,"num_objects":1274,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-810}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11667","reported_seq":11765,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:33.774374+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:33.774374+0000","last_peered":"2024-09-18T11:54:33.774374+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:33.774374+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10648187,"num_objects":1300,"num_object_clones":0,"num_object_copies":5200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11666,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-746}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11692","reported_seq":11816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.024812+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:34.024812+0000","last_peered":"2024-09-18T11:54:34.024812+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:34.024812+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449590,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11690,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-725}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11637","reported_seq":15926,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:36.128042+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:36.128042+0000","last_peered":"2024-09-18T11:54:36.128042+0000","last_clean":"2024-09-18T11:54:36.128042+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:36.128042+0000","last_undegraded":"2024-09-18T11:54:36.128042+0000","last_fullsized":"2024-09-18T11:54:36.128042+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10368246,"num_objects":1266,"num_object_clones":0,"num_object_copies":5064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11369","reported_seq":11458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:34.856774+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:34.856774+0000","last_peered":"2024-09-18T11:54:34.856774+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:34.856774+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10958070,"num_objects":1338,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11367,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":173155724,"num_objects":21143,"num_object_clones":0,"num_object_copies":84572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21143,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185944,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7844,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":345878528,"data_stored":345878528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679860,"kb_used_data":85288,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93691980,"statfs":{"total":96636764160,"available":95940587520,"internally_reserved":0,"allocated":87334912,"data_stored":87151091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,6,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.569},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.968}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":18.102},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.9770000000000003},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.32000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675888,"kb_used_data":85988,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93695952,"statfs":{"total":96636764160,"available":95944654848,"internally_reserved":0,"allocated":88051712,"data_stored":87867891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,1,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":3.5830000000000002},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":3.6120000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":5.4349999999999996},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":1.1930000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":0.68200000000000005},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":4.5800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679260,"kb_used_data":84732,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93692580,"statfs":{"total":96636764160,"available":95941201920,"internally_reserved":0,"allocated":86765568,"data_stored":86581747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":3.048},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.8330000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":8.3469999999999995},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.7800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":2.8029999999999999},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":2.8650000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671524,"kb_used_data":85340,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93700316,"statfs":{"total":96636764160,"available":95949123584,"internally_reserved":0,"allocated":87388160,"data_stored":87204339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,0,1,0,0,1,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":17.617999999999999},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":3.875}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.639},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":3.0899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":9.8230000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86245376,"data_stored":86245376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":85692416,"data_stored":85692416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":87056384,"data_stored":87056384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86884352,"data_stored":86884352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:42.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:43.069 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:43.069 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:43.099 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-18T11:54:42.070842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165373324,"num_objects":20193,"num_object_clones":0,"num_object_copies":80772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186894,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7909,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160821,"ondisk_log_size":160821,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3694476,"kb_used_data":325784,"kb_used_omap":14,"kb_used_meta":3368625,"kb_avail":373792884,"statfs":{"total":386547056640,"available":382763913216,"internally_reserved":0,"allocated":333602816,"data_stored":332867532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3449472027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,5,4,9,14,15,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16973824,"num_objects":-2072,"num_object_clones":0,"num_object_copies":-8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2072,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006778"},"pg_stats":[{"pgid":"1.f","version":"37'11976","reported_seq":12077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.933198+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:38.933198+0000","last_peered":"2024-09-18T11:54:38.933198+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:38.933198+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10436608,"num_objects":1274,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11976,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-829}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11777","reported_seq":16069,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:41.051456+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:41.051456+0000","last_peered":"2024-09-18T11:54:41.051456+0000","last_clean":"2024-09-18T11:54:41.051456+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:41.051456+0000","last_undegraded":"2024-09-18T11:54:41.051456+0000","last_fullsized":"2024-09-18T11:54:41.051456+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10474742,"num_objects":1279,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11541","reported_seq":11658,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.640577+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:38.640577+0000","last_peered":"2024-09-18T11:54:38.640577+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:38.640577+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10377875,"num_objects":1268,"num_object_clones":0,"num_object_copies":5072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11540,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-772}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11647","reported_seq":11741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.530194+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:39.530194+0000","last_peered":"2024-09-18T11:54:39.530194+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:39.530194+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10474742,"num_objects":1279,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11639","reported_seq":11760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.032488+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:39.032488+0000","last_peered":"2024-09-18T11:54:39.032488+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:39.032488+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10286326,"num_objects":1256,"num_object_clones":0,"num_object_copies":5024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11637,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-796}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11616","reported_seq":11735,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.992693+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:38.992693+0000","last_peered":"2024-09-18T11:54:38.992693+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:38.992693+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9584640,"num_objects":1170,"num_object_clones":0,"num_object_copies":4680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11771","reported_seq":11878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.518844+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:39.518844+0000","last_peered":"2024-09-18T11:54:39.518844+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:39.518844+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10196214,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-871}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11581","reported_seq":14231,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.862171+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:39.862171+0000","last_peered":"2024-09-18T11:54:39.862171+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:39.862171+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10597622,"num_objects":1294,"num_object_clones":0,"num_object_copies":5176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11579,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1292,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11880","reported_seq":16233,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:38.875135+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:38.875135+0000","last_peered":"2024-09-18T11:54:38.875135+0000","last_clean":"2024-09-18T11:54:38.875135+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:38.875135+0000","last_undegraded":"2024-09-18T11:54:38.875135+0000","last_fullsized":"2024-09-18T11:54:38.875135+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10611180,"num_objects":1296,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11345","reported_seq":11459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.874560+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:38.874560+0000","last_peered":"2024-09-18T11:54:38.874560+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:38.874560+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10228982,"num_objects":1249,"num_object_clones":0,"num_object_copies":4996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11343,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11714","reported_seq":11835,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.154644+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:39.154644+0000","last_peered":"2024-09-18T11:54:39.154644+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:39.154644+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10801009,"num_objects":1319,"num_object_clones":0,"num_object_copies":5276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11815","reported_seq":11908,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.814576+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:38.814576+0000","last_peered":"2024-09-18T11:54:38.814576+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:38.814576+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9960059,"num_objects":1216,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11814,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-868}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11726","reported_seq":11824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.769437+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:38.769437+0000","last_peered":"2024-09-18T11:54:38.769437+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:38.769437+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10164859,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-805}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11766","reported_seq":11891,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.217617+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:39.217617+0000","last_peered":"2024-09-18T11:54:39.217617+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:39.217617+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10843382,"num_objects":1324,"num_object_clones":0,"num_object_copies":5296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-799}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11695","reported_seq":15984,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:41.066061+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:41.066061+0000","last_peered":"2024-09-18T11:54:41.066061+0000","last_clean":"2024-09-18T11:54:41.066061+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:41.066061+0000","last_undegraded":"2024-09-18T11:54:41.066061+0000","last_fullsized":"2024-09-18T11:54:41.066061+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9893110,"num_objects":1208,"num_object_clones":0,"num_object_copies":4832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11693,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11432","reported_seq":11522,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.780948+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:39.780948+0000","last_peered":"2024-09-18T11:54:39.780948+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:39.780948+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10441974,"num_objects":1275,"num_object_clones":0,"num_object_copies":5100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11430,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":165373324,"num_objects":20193,"num_object_clones":0,"num_object_copies":80772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186894,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7909,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":330043392,"data_stored":330043392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160821,"ondisk_log_size":160821,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1009100,"kb_used_data":81404,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93362740,"statfs":{"total":96636764160,"available":95603445760,"internally_reserved":0,"allocated":83357696,"data_stored":83173875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,2,0,1,2,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":4.9640000000000004},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":4.9960000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.305},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":2.7610000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":1.044},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.726}]}]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1005212,"kb_used_data":82192,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93366628,"statfs":{"total":96636764160,"available":95607427072,"internally_reserved":0,"allocated":84164608,"data_stored":83980787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,3,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":15.198},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":15.228}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":14.207000000000001},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.44}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":3.4910000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":6.3529999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679396,"kb_used_data":80732,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93692444,"statfs":{"total":96636764160,"available":95941062656,"internally_reserved":0,"allocated":82669568,"data_stored":82485747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.82799999999999996},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.79700000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.32500000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.51400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000768,"kb_used_data":81456,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93371072,"statfs":{"total":96636764160,"available":95611977728,"internally_reserved":0,"allocated":83410944,"data_stored":83227123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,8,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":14.032},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.9630000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.9870000000000001},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":6.0179999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":21.512},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.9960000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":82362368,"data_stored":82362368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81608704,"data_stored":81608704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83144704,"data_stored":83144704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82927616,"data_stored":82927616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:43.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:43.640 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:43.641 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:43.679 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-18T11:54:42.070842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165373324,"num_objects":20193,"num_object_clones":0,"num_object_copies":80772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186894,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7909,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160821,"ondisk_log_size":160821,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3694476,"kb_used_data":325784,"kb_used_omap":14,"kb_used_meta":3368625,"kb_avail":373792884,"statfs":{"total":386547056640,"available":382763913216,"internally_reserved":0,"allocated":333602816,"data_stored":332867532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3449472027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,5,4,9,14,15,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16973824,"num_objects":-2072,"num_object_clones":0,"num_object_copies":-8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2072,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006778"},"pg_stats":[{"pgid":"1.f","version":"37'11976","reported_seq":12077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.933198+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:38.933198+0000","last_peered":"2024-09-18T11:54:38.933198+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:38.933198+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10436608,"num_objects":1274,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11976,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-829}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11777","reported_seq":16069,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:41.051456+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:41.051456+0000","last_peered":"2024-09-18T11:54:41.051456+0000","last_clean":"2024-09-18T11:54:41.051456+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:41.051456+0000","last_undegraded":"2024-09-18T11:54:41.051456+0000","last_fullsized":"2024-09-18T11:54:41.051456+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10474742,"num_objects":1279,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11541","reported_seq":11658,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.640577+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:38.640577+0000","last_peered":"2024-09-18T11:54:38.640577+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:38.640577+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10377875,"num_objects":1268,"num_object_clones":0,"num_object_copies":5072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11540,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-772}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11647","reported_seq":11741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.530194+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:39.530194+0000","last_peered":"2024-09-18T11:54:39.530194+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:39.530194+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10474742,"num_objects":1279,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11639","reported_seq":11760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.032488+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:39.032488+0000","last_peered":"2024-09-18T11:54:39.032488+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:39.032488+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10286326,"num_objects":1256,"num_object_clones":0,"num_object_copies":5024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11637,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-796}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11616","reported_seq":11735,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.992693+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:38.992693+0000","last_peered":"2024-09-18T11:54:38.992693+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:38.992693+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9584640,"num_objects":1170,"num_object_clones":0,"num_object_copies":4680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11771","reported_seq":11878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.518844+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:39.518844+0000","last_peered":"2024-09-18T11:54:39.518844+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:39.518844+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10196214,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-871}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11581","reported_seq":14231,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.862171+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:39.862171+0000","last_peered":"2024-09-18T11:54:39.862171+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:39.862171+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10597622,"num_objects":1294,"num_object_clones":0,"num_object_copies":5176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11579,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1292,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11880","reported_seq":16233,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:38.875135+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:38.875135+0000","last_peered":"2024-09-18T11:54:38.875135+0000","last_clean":"2024-09-18T11:54:38.875135+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:38.875135+0000","last_undegraded":"2024-09-18T11:54:38.875135+0000","last_fullsized":"2024-09-18T11:54:38.875135+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10611180,"num_objects":1296,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11345","reported_seq":11459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.874560+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:38.874560+0000","last_peered":"2024-09-18T11:54:38.874560+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:38.874560+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10228982,"num_objects":1249,"num_object_clones":0,"num_object_copies":4996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11343,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11714","reported_seq":11835,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.154644+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:39.154644+0000","last_peered":"2024-09-18T11:54:39.154644+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:39.154644+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10801009,"num_objects":1319,"num_object_clones":0,"num_object_copies":5276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11815","reported_seq":11908,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.814576+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:38.814576+0000","last_peered":"2024-09-18T11:54:38.814576+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:38.814576+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9960059,"num_objects":1216,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11814,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-868}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11726","reported_seq":11824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.769437+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:38.769437+0000","last_peered":"2024-09-18T11:54:38.769437+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:38.769437+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10164859,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-805}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11766","reported_seq":11891,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.217617+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:39.217617+0000","last_peered":"2024-09-18T11:54:39.217617+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:39.217617+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10843382,"num_objects":1324,"num_object_clones":0,"num_object_copies":5296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-799}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11695","reported_seq":15984,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:41.066061+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:41.066061+0000","last_peered":"2024-09-18T11:54:41.066061+0000","last_clean":"2024-09-18T11:54:41.066061+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:41.066061+0000","last_undegraded":"2024-09-18T11:54:41.066061+0000","last_fullsized":"2024-09-18T11:54:41.066061+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9893110,"num_objects":1208,"num_object_clones":0,"num_object_copies":4832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11693,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11432","reported_seq":11522,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.780948+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:39.780948+0000","last_peered":"2024-09-18T11:54:39.780948+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:39.780948+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10441974,"num_objects":1275,"num_object_clones":0,"num_object_copies":5100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11430,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":165373324,"num_objects":20193,"num_object_clones":0,"num_object_copies":80772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186894,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7909,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":330043392,"data_stored":330043392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160821,"ondisk_log_size":160821,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1009100,"kb_used_data":81404,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93362740,"statfs":{"total":96636764160,"available":95603445760,"internally_reserved":0,"allocated":83357696,"data_stored":83173875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,2,0,1,2,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":4.9640000000000004},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":4.9960000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.305},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":2.7610000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":1.044},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.726}]}]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1005212,"kb_used_data":82192,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93366628,"statfs":{"total":96636764160,"available":95607427072,"internally_reserved":0,"allocated":84164608,"data_stored":83980787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,3,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":15.198},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":15.228}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":14.207000000000001},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.44}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":3.4910000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":6.3529999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679396,"kb_used_data":80732,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93692444,"statfs":{"total":96636764160,"available":95941062656,"internally_reserved":0,"allocated":82669568,"data_stored":82485747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.82799999999999996},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.79700000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.32500000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.51400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000768,"kb_used_data":81456,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93371072,"statfs":{"total":96636764160,"available":95611977728,"internally_reserved":0,"allocated":83410944,"data_stored":83227123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,8,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":14.032},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.9630000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.9870000000000001},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":6.0179999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":21.512},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.9960000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":82362368,"data_stored":82362368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81608704,"data_stored":81608704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83144704,"data_stored":83144704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82927616,"data_stored":82927616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:43.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:54:43.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11976", 'reported_seq': 12077, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:38.933198+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:38.933198+0000', 'last_peered': '2024-09-18T11:54:38.933198+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:38.933198+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10436608, 'num_objects': 1274, 'num_object_clones': 0, 'num_object_copies': 5096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11976, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -829}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:43.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:54:43.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11541", 'reported_seq': 11658, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:38.640577+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:38.640577+0000', 'last_peered': '2024-09-18T11:54:38.640577+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:38.640577+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10377875, 'num_objects': 1268, 'num_object_clones': 0, 'num_object_copies': 5072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1268, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11540, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -772}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:43.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:54:43.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11647", 'reported_seq': 11741, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:39.530194+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:54:39.530194+0000', 'last_peered': '2024-09-18T11:54:39.530194+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:54:39.530194+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10474742, 'num_objects': 1279, 'num_object_clones': 0, 'num_object_copies': 5116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11645, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -822}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:43.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:54:43.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11639", 'reported_seq': 11760, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:39.032488+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:39.032488+0000', 'last_peered': '2024-09-18T11:54:39.032488+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:39.032488+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10286326, 'num_objects': 1256, 'num_object_clones': 0, 'num_object_copies': 5024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11637, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -796}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:43.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:54:43.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11616", 'reported_seq': 11735, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:38.992693+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:38.992693+0000', 'last_peered': '2024-09-18T11:54:38.992693+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:38.992693+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9584640, 'num_objects': 1170, 'num_object_clones': 0, 'num_object_copies': 4680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11616, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -949}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:43.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:54:43.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11771", 'reported_seq': 11878, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:39.518844+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:39.518844+0000', 'last_peered': '2024-09-18T11:54:39.518844+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:39.518844+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10196214, 'num_objects': 1245, 'num_object_clones': 0, 'num_object_copies': 4980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11769, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -871}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:43.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:54:43.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11581", 'reported_seq': 14231, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:39.862171+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:54:39.862171+0000', 'last_peered': '2024-09-18T11:54:39.862171+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:54:39.862171+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10597622, 'num_objects': 1294, 'num_object_clones': 0, 'num_object_copies': 5176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 746, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11579, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1292, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 548}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:43.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:54:43.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11345", 'reported_seq': 11459, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:38.874560+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:38.874560+0000', 'last_peered': '2024-09-18T11:54:38.874560+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:38.874560+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10228982, 'num_objects': 1249, 'num_object_clones': 0, 'num_object_copies': 4996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11343, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -714}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:43.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:54:43.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11714", 'reported_seq': 11835, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:39.154644+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:39.154644+0000', 'last_peered': '2024-09-18T11:54:39.154644+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:39.154644+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10801009, 'num_objects': 1319, 'num_object_clones': 0, 'num_object_copies': 5276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11711, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -715}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:43.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:54:43.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11815", 'reported_seq': 11908, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:38.814576+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:38.814576+0000', 'last_peered': '2024-09-18T11:54:38.814576+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:38.814576+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9960059, 'num_objects': 1216, 'num_object_clones': 0, 'num_object_copies': 4864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11814, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -868}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:43.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:54:43.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11726", 'reported_seq': 11824, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:38.769437+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:38.769437+0000', 'last_peered': '2024-09-18T11:54:38.769437+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:38.769437+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10164859, 'num_objects': 1241, 'num_object_clones': 0, 'num_object_copies': 4964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11725, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -805}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:43.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:54:43.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11766", 'reported_seq': 11891, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:39.217617+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:39.217617+0000', 'last_peered': '2024-09-18T11:54:39.217617+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:39.217617+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10843382, 'num_objects': 1324, 'num_object_clones': 0, 'num_object_copies': 5296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11764, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -799}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:43.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:54:43.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11432", 'reported_seq': 11522, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:39.780948+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:39.780948+0000', 'last_peered': '2024-09-18T11:54:39.780948+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:39.780948+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10441974, 'num_objects': 1275, 'num_object_clones': 0, 'num_object_copies': 5100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11430, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -727}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:43.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:44.342 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:44.342 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25547/80772 objects degraded (31.629%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":682,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20193,"data_bytes":165373324,"bytes_used":3783143424,"bytes_avail":382763913216,"bytes_total":386547056640,"degraded_objects":25547,"degraded_total":80772,"degraded_ratio":0.31628534640717082,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":172},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:44.364 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:44.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:44.935 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:44.936 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:44.965 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-18T11:54:44.071705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":163743116,"num_objects":19994,"num_object_clones":0,"num_object_copies":79976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187093,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7909,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3694556,"kb_used_data":321708,"kb_used_omap":14,"kb_used_meta":3372785,"kb_avail":373792804,"statfs":{"total":386547056640,"available":382763831296,"internally_reserved":0,"allocated":329428992,"data_stored":328693708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3453731867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,4,4,12,13,18,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17571840,"num_objects":-2145,"num_object_clones":0,"num_object_copies":-8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006789"},"pg_stats":[{"pgid":"1.f","version":"37'11976","reported_seq":12077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.933198+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:38.933198+0000","last_peered":"2024-09-18T11:54:38.933198+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:38.933198+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10436608,"num_objects":1274,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11976,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-829}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11777","reported_seq":16069,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:41.051456+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:41.051456+0000","last_peered":"2024-09-18T11:54:41.051456+0000","last_clean":"2024-09-18T11:54:41.051456+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:41.051456+0000","last_undegraded":"2024-09-18T11:54:41.051456+0000","last_fullsized":"2024-09-18T11:54:41.051456+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10474742,"num_objects":1279,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11608","reported_seq":11725,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.850778+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:43.850778+0000","last_peered":"2024-09-18T11:54:43.850778+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:43.850778+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9829011,"num_objects":1201,"num_object_clones":0,"num_object_copies":4804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1201,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11607,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11647","reported_seq":11741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.530194+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:39.530194+0000","last_peered":"2024-09-18T11:54:39.530194+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:39.530194+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10474742,"num_objects":1279,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11639","reported_seq":11760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.032488+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:39.032488+0000","last_peered":"2024-09-18T11:54:39.032488+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:39.032488+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10286326,"num_objects":1256,"num_object_clones":0,"num_object_copies":5024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11637,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-796}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11616","reported_seq":11735,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.992693+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:38.992693+0000","last_peered":"2024-09-18T11:54:38.992693+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:38.992693+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9584640,"num_objects":1170,"num_object_clones":0,"num_object_copies":4680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11771","reported_seq":11878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.518844+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:39.518844+0000","last_peered":"2024-09-18T11:54:39.518844+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:39.518844+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10196214,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-871}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11581","reported_seq":14231,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.862171+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:39.862171+0000","last_peered":"2024-09-18T11:54:39.862171+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:39.862171+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10597622,"num_objects":1294,"num_object_clones":0,"num_object_copies":5176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11579,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1292,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11880","reported_seq":16233,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:38.875135+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:38.875135+0000","last_peered":"2024-09-18T11:54:38.875135+0000","last_clean":"2024-09-18T11:54:38.875135+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:38.875135+0000","last_undegraded":"2024-09-18T11:54:38.875135+0000","last_fullsized":"2024-09-18T11:54:38.875135+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10611180,"num_objects":1296,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11409","reported_seq":11523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.472586+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:43.472586+0000","last_peered":"2024-09-18T11:54:43.472586+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:43.472586+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9704694,"num_objects":1185,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11407,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-778}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11714","reported_seq":11835,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.154644+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:39.154644+0000","last_peered":"2024-09-18T11:54:39.154644+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:39.154644+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10801009,"num_objects":1319,"num_object_clones":0,"num_object_copies":5276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11815","reported_seq":11908,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:38.814576+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:38.814576+0000","last_peered":"2024-09-18T11:54:38.814576+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:38.814576+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9960059,"num_objects":1216,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11814,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-868}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11794","reported_seq":11892,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.694353+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:43.694353+0000","last_peered":"2024-09-18T11:54:43.694353+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:43.694353+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9607803,"num_objects":1173,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11766","reported_seq":11891,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.217617+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:39.217617+0000","last_peered":"2024-09-18T11:54:39.217617+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:39.217617+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10843382,"num_objects":1324,"num_object_clones":0,"num_object_copies":5296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-799}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11695","reported_seq":15984,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:41.066061+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:41.066061+0000","last_peered":"2024-09-18T11:54:41.066061+0000","last_clean":"2024-09-18T11:54:41.066061+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:41.066061+0000","last_undegraded":"2024-09-18T11:54:41.066061+0000","last_fullsized":"2024-09-18T11:54:41.066061+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9893110,"num_objects":1208,"num_object_clones":0,"num_object_copies":4832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11693,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11432","reported_seq":11522,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:39.780948+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:39.780948+0000","last_peered":"2024-09-18T11:54:39.780948+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:39.780948+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10441974,"num_objects":1275,"num_object_clones":0,"num_object_copies":5100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11430,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":163743116,"num_objects":19994,"num_object_clones":0,"num_object_copies":79976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187093,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7909,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":325935104,"data_stored":325935104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1009100,"kb_used_data":81404,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93362740,"statfs":{"total":96636764160,"available":95603445760,"internally_reserved":0,"allocated":83357696,"data_stored":83173875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,2,0,1,2,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":4.9640000000000004},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":4.9960000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.305},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":2.7610000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":1.044},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.726}]}]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1005292,"kb_used_data":78116,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93366548,"statfs":{"total":96636764160,"available":95607345152,"internally_reserved":0,"allocated":79990784,"data_stored":79806963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":15.481},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":15.519}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.2080000000000002},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":15.802}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":15.827999999999999},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":9.9320000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679396,"kb_used_data":80732,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93692444,"statfs":{"total":96636764160,"available":95941062656,"internally_reserved":0,"allocated":82669568,"data_stored":82485747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":0.82799999999999996},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.79700000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.32500000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.51400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000768,"kb_used_data":81456,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93371072,"statfs":{"total":96636764160,"available":95611977728,"internally_reserved":0,"allocated":83410944,"data_stored":83227123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,8,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":14.032},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.9630000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.9870000000000001},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":6.0179999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":21.512},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.9960000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":82362368,"data_stored":82362368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81608704,"data_stored":81608704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79036416,"data_stored":79036416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82927616,"data_stored":82927616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:47.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:48.564 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:48.565 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:48.585 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-18T11:54:48.073142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":157001100,"num_objects":19171,"num_object_clones":0,"num_object_copies":76684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187916,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7964,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3661768,"kb_used_data":309732,"kb_used_omap":14,"kb_used_meta":3351985,"kb_avail":373825592,"statfs":{"total":386547056640,"available":382797406208,"internally_reserved":0,"allocated":317165568,"data_stored":316430284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3432432667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,0,1,13,10,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17088512,"num_objects":-2086,"num_object_clones":0,"num_object_copies":-8344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003995"},"pg_stats":[{"pgid":"1.f","version":"37'12038","reported_seq":12139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.259493+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:44.259493+0000","last_peered":"2024-09-18T11:54:44.259493+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:44.259493+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9928704,"num_objects":1212,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11846","reported_seq":16138,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:46.172769+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:46.172769+0000","last_peered":"2024-09-18T11:54:46.172769+0000","last_clean":"2024-09-18T11:54:46.172769+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:46.172769+0000","last_undegraded":"2024-09-18T11:54:46.172769+0000","last_fullsized":"2024-09-18T11:54:46.172769+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9909494,"num_objects":1210,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11608","reported_seq":11725,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.850778+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:43.850778+0000","last_peered":"2024-09-18T11:54:43.850778+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:43.850778+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9829011,"num_objects":1201,"num_object_clones":0,"num_object_copies":4804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1201,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11607,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11717","reported_seq":11811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.777446+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:44.777446+0000","last_peered":"2024-09-18T11:54:44.777446+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:44.777446+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9901302,"num_objects":1209,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11711","reported_seq":11832,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.242765+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:44.242765+0000","last_peered":"2024-09-18T11:54:44.242765+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:44.242765+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9696502,"num_objects":1184,"num_object_clones":0,"num_object_copies":4736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11709,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11678","reported_seq":11797,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.936602+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:43.936602+0000","last_peered":"2024-09-18T11:54:43.936602+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:43.936602+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9076736,"num_objects":1108,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1011}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11812","reported_seq":11919,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.728420+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:44.728420+0000","last_peered":"2024-09-18T11:54:44.728420+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:44.728420+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9860342,"num_objects":1204,"num_object_clones":0,"num_object_copies":4816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11810,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-912}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11640","reported_seq":14400,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.908312+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:44.908312+0000","last_peered":"2024-09-18T11:54:44.908312+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:44.908312+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10114294,"num_objects":1235,"num_object_clones":0,"num_object_copies":4940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11638,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11946","reported_seq":16299,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:44.197459+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:44.197459+0000","last_peered":"2024-09-18T11:54:44.197459+0000","last_clean":"2024-09-18T11:54:44.197459+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:44.197459+0000","last_undegraded":"2024-09-18T11:54:44.197459+0000","last_fullsized":"2024-09-18T11:54:44.197459+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10070508,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11943,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11409","reported_seq":11523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.472586+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:43.472586+0000","last_peered":"2024-09-18T11:54:43.472586+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:43.472586+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9704694,"num_objects":1185,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11407,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-778}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11785","reported_seq":11906,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.293721+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:44.293721+0000","last_peered":"2024-09-18T11:54:44.293721+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:44.293721+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10219377,"num_objects":1248,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11880","reported_seq":11973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.147511+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:44.147511+0000","last_peered":"2024-09-18T11:54:44.147511+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:44.147511+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9427579,"num_objects":1151,"num_object_clones":0,"num_object_copies":4604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-933}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11794","reported_seq":11892,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.694353+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:43.694353+0000","last_peered":"2024-09-18T11:54:43.694353+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:43.694353+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9607803,"num_objects":1173,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11826","reported_seq":11951,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.197269+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:44.197269+0000","last_peered":"2024-09-18T11:54:44.197269+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:44.197269+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10351862,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11765","reported_seq":16051,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:46.246382+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:46.246382+0000","last_peered":"2024-09-18T11:54:46.246382+0000","last_clean":"2024-09-18T11:54:46.246382+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:46.246382+0000","last_undegraded":"2024-09-18T11:54:46.246382+0000","last_fullsized":"2024-09-18T11:54:46.246382+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9319670,"num_objects":1138,"num_object_clones":0,"num_object_copies":4552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11763,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11488","reported_seq":11578,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.916521+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:44.916521+0000","last_peered":"2024-09-18T11:54:44.916521+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:44.916521+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983222,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11486,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":157001100,"num_objects":19171,"num_object_clones":0,"num_object_copies":76684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187916,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7964,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":313552896,"data_stored":313552896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992636,"kb_used_data":77428,"kb_used_omap":4,"kb_used_meta":915195,"kb_avail":93379204,"statfs":{"total":96636764160,"available":95620304896,"internally_reserved":0,"allocated":79286272,"data_stored":79102451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937160257},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.224},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.253}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":10.074}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":10.551},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":10.042}]}]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1005292,"kb_used_data":78116,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93366548,"statfs":{"total":96636764160,"available":95607345152,"internally_reserved":0,"allocated":79990784,"data_stored":79806963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":15.481},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":15.519}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.2080000000000002},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":15.802}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":15.827999999999999},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":9.9320000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662976,"kb_used_data":76792,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93708864,"statfs":{"total":96636764160,"available":95957876736,"internally_reserved":0,"allocated":78635008,"data_stored":78451187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":36.302999999999997},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":21.343}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":7.8639999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.35999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":4.5650000000000004},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":7.9029999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000864,"kb_used_data":77396,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93370976,"statfs":{"total":96636764160,"available":95611879424,"internally_reserved":0,"allocated":79253504,"data_stored":79069683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,0,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.9370000000000003},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":13.035}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":5.8849999999999998},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":11.507999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":5.8159999999999998},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":15.763999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":78266368,"data_stored":78266368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77524992,"data_stored":77524992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79036416,"data_stored":79036416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78725120,"data_stored":78725120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:48.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:49.201 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:49.202 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:49.219 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-18T11:54:48.073142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":157001100,"num_objects":19171,"num_object_clones":0,"num_object_copies":76684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187916,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7964,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3661768,"kb_used_data":309732,"kb_used_omap":14,"kb_used_meta":3351985,"kb_avail":373825592,"statfs":{"total":386547056640,"available":382797406208,"internally_reserved":0,"allocated":317165568,"data_stored":316430284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3432432667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,0,1,13,10,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17088512,"num_objects":-2086,"num_object_clones":0,"num_object_copies":-8344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003995"},"pg_stats":[{"pgid":"1.f","version":"37'12038","reported_seq":12139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.259493+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:44.259493+0000","last_peered":"2024-09-18T11:54:44.259493+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:44.259493+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9928704,"num_objects":1212,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11846","reported_seq":16138,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:46.172769+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:46.172769+0000","last_peered":"2024-09-18T11:54:46.172769+0000","last_clean":"2024-09-18T11:54:46.172769+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:46.172769+0000","last_undegraded":"2024-09-18T11:54:46.172769+0000","last_fullsized":"2024-09-18T11:54:46.172769+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9909494,"num_objects":1210,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11608","reported_seq":11725,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.850778+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:43.850778+0000","last_peered":"2024-09-18T11:54:43.850778+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:43.850778+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9829011,"num_objects":1201,"num_object_clones":0,"num_object_copies":4804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1201,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11607,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11717","reported_seq":11811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.777446+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:44.777446+0000","last_peered":"2024-09-18T11:54:44.777446+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:44.777446+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9901302,"num_objects":1209,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11711","reported_seq":11832,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.242765+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:44.242765+0000","last_peered":"2024-09-18T11:54:44.242765+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:44.242765+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9696502,"num_objects":1184,"num_object_clones":0,"num_object_copies":4736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11709,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11678","reported_seq":11797,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.936602+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:43.936602+0000","last_peered":"2024-09-18T11:54:43.936602+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:43.936602+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9076736,"num_objects":1108,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1011}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11812","reported_seq":11919,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.728420+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:44.728420+0000","last_peered":"2024-09-18T11:54:44.728420+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:44.728420+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9860342,"num_objects":1204,"num_object_clones":0,"num_object_copies":4816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11810,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-912}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11640","reported_seq":14400,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.908312+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:44.908312+0000","last_peered":"2024-09-18T11:54:44.908312+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:44.908312+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10114294,"num_objects":1235,"num_object_clones":0,"num_object_copies":4940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11638,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11946","reported_seq":16299,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:44.197459+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:44.197459+0000","last_peered":"2024-09-18T11:54:44.197459+0000","last_clean":"2024-09-18T11:54:44.197459+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:44.197459+0000","last_undegraded":"2024-09-18T11:54:44.197459+0000","last_fullsized":"2024-09-18T11:54:44.197459+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10070508,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11943,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11409","reported_seq":11523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.472586+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:43.472586+0000","last_peered":"2024-09-18T11:54:43.472586+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:43.472586+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9704694,"num_objects":1185,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11407,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-778}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11785","reported_seq":11906,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.293721+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:44.293721+0000","last_peered":"2024-09-18T11:54:44.293721+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:44.293721+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10219377,"num_objects":1248,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11880","reported_seq":11973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.147511+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:44.147511+0000","last_peered":"2024-09-18T11:54:44.147511+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:44.147511+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9427579,"num_objects":1151,"num_object_clones":0,"num_object_copies":4604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-933}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11794","reported_seq":11892,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:43.694353+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:43.694353+0000","last_peered":"2024-09-18T11:54:43.694353+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:43.694353+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9607803,"num_objects":1173,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11826","reported_seq":11951,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.197269+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:44.197269+0000","last_peered":"2024-09-18T11:54:44.197269+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:44.197269+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10351862,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11765","reported_seq":16051,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:46.246382+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:46.246382+0000","last_peered":"2024-09-18T11:54:46.246382+0000","last_clean":"2024-09-18T11:54:46.246382+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:46.246382+0000","last_undegraded":"2024-09-18T11:54:46.246382+0000","last_fullsized":"2024-09-18T11:54:46.246382+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9319670,"num_objects":1138,"num_object_clones":0,"num_object_copies":4552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11763,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11488","reported_seq":11578,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:44.916521+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:44.916521+0000","last_peered":"2024-09-18T11:54:44.916521+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:44.916521+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983222,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11486,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":157001100,"num_objects":19171,"num_object_clones":0,"num_object_copies":76684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187916,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7964,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":313552896,"data_stored":313552896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992636,"kb_used_data":77428,"kb_used_omap":4,"kb_used_meta":915195,"kb_avail":93379204,"statfs":{"total":96636764160,"available":95620304896,"internally_reserved":0,"allocated":79286272,"data_stored":79102451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937160257},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.224},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.253}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":10.074}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":10.551},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":10.042}]}]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1005292,"kb_used_data":78116,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93366548,"statfs":{"total":96636764160,"available":95607345152,"internally_reserved":0,"allocated":79990784,"data_stored":79806963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":15.481},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":15.519}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.2080000000000002},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":15.802}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":15.827999999999999},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":9.9320000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662976,"kb_used_data":76792,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93708864,"statfs":{"total":96636764160,"available":95957876736,"internally_reserved":0,"allocated":78635008,"data_stored":78451187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":36.302999999999997},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":21.343}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":7.8639999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.35999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":4.5650000000000004},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":7.9029999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000864,"kb_used_data":77396,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93370976,"statfs":{"total":96636764160,"available":95611879424,"internally_reserved":0,"allocated":79253504,"data_stored":79069683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,0,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.9370000000000003},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":13.035}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":5.8849999999999998},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":11.507999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":5.8159999999999998},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":15.763999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":78266368,"data_stored":78266368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77524992,"data_stored":77524992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79036416,"data_stored":79036416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78725120,"data_stored":78725120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:49.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:54:49.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12038", 'reported_seq': 12139, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:44.259493+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:44.259493+0000', 'last_peered': '2024-09-18T11:54:44.259493+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:44.259493+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9928704, 'num_objects': 1212, 'num_object_clones': 0, 'num_object_copies': 4848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1212, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12038, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -891}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:49.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:54:49.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11608", 'reported_seq': 11725, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:43.850778+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:43.850778+0000', 'last_peered': '2024-09-18T11:54:43.850778+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:43.850778+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9829011, 'num_objects': 1201, 'num_object_clones': 0, 'num_object_copies': 4804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1201, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11607, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -839}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:49.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:54:49.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11717", 'reported_seq': 11811, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:44.777446+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:54:44.777446+0000', 'last_peered': '2024-09-18T11:54:44.777446+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:54:44.777446+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9901302, 'num_objects': 1209, 'num_object_clones': 0, 'num_object_copies': 4836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11715, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -892}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:49.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:54:49.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11711", 'reported_seq': 11832, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:44.242765+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:44.242765+0000', 'last_peered': '2024-09-18T11:54:44.242765+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:44.242765+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9696502, 'num_objects': 1184, 'num_object_clones': 0, 'num_object_copies': 4736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11709, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -868}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:49.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:54:49.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11678", 'reported_seq': 11797, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:43.936602+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:43.936602+0000', 'last_peered': '2024-09-18T11:54:43.936602+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:43.936602+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9076736, 'num_objects': 1108, 'num_object_clones': 0, 'num_object_copies': 4432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1011}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:49.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:54:49.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11812", 'reported_seq': 11919, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:44.728420+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:44.728420+0000', 'last_peered': '2024-09-18T11:54:44.728420+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:44.728420+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9860342, 'num_objects': 1204, 'num_object_clones': 0, 'num_object_copies': 4816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11810, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -912}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:49.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:54:49.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11640", 'reported_seq': 14400, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:44.908312+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:54:44.908312+0000', 'last_peered': '2024-09-18T11:54:44.908312+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:54:44.908312+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10114294, 'num_objects': 1235, 'num_object_clones': 0, 'num_object_copies': 4940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 691, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11638, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1347, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 544}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:49.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:54:49.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11409", 'reported_seq': 11523, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:43.472586+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:43.472586+0000', 'last_peered': '2024-09-18T11:54:43.472586+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:43.472586+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9704694, 'num_objects': 1185, 'num_object_clones': 0, 'num_object_copies': 4740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11407, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -778}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:49.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:54:49.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11785", 'reported_seq': 11906, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:44.293721+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:44.293721+0000', 'last_peered': '2024-09-18T11:54:44.293721+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:44.293721+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10219377, 'num_objects': 1248, 'num_object_clones': 0, 'num_object_copies': 4992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11782, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -786}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:49.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:54:49.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11880", 'reported_seq': 11973, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:44.147511+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:44.147511+0000', 'last_peered': '2024-09-18T11:54:44.147511+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:44.147511+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9427579, 'num_objects': 1151, 'num_object_clones': 0, 'num_object_copies': 4604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -933}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:49.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:54:49.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11794", 'reported_seq': 11892, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:43.694353+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:43.694353+0000', 'last_peered': '2024-09-18T11:54:43.694353+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:43.694353+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9607803, 'num_objects': 1173, 'num_object_clones': 0, 'num_object_copies': 4692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -873}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:49.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:54:49.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11826", 'reported_seq': 11951, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:44.197269+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:44.197269+0000', 'last_peered': '2024-09-18T11:54:44.197269+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:44.197269+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10351862, 'num_objects': 1264, 'num_object_clones': 0, 'num_object_copies': 5056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11824, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:49.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:54:49.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11488", 'reported_seq': 11578, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:44.916521+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:44.916521+0000', 'last_peered': '2024-09-18T11:54:44.916521+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:44.916521+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9983222, 'num_objects': 1219, 'num_object_clones': 0, 'num_object_copies': 4876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11486, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:49.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:49.902 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:49.902 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25492/76684 objects degraded (33.243%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":687,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":19171,"data_bytes":157001100,"bytes_used":3749650432,"bytes_avail":382797406208,"bytes_total":386547056640,"degraded_objects":25492,"degraded_total":76684,"degraded_ratio":0.33242918992227843,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":173},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:49.931 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:49.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:50.498 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:50.524 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-18T11:54:50.073806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149742988,"num_objects":18285,"num_object_clones":0,"num_object_copies":73140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188802,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8013,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160829,"ondisk_log_size":160829,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3657740,"kb_used_data":297368,"kb_used_omap":14,"kb_used_meta":3360305,"kb_avail":373829620,"statfs":{"total":386547056640,"available":382801530880,"internally_reserved":0,"allocated":304504832,"data_stored":303769548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3440952347},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,4,14,15,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23412736,"num_objects":-2858,"num_object_clones":0,"num_object_copies":-11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2858,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004167"},"pg_stats":[{"pgid":"1.f","version":"37'12113","reported_seq":12214,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.210849+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:49.210849+0000","last_peered":"2024-09-18T11:54:49.210849+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:49.210849+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9314304,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12113,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11846","reported_seq":16138,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:46.172769+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:46.172769+0000","last_peered":"2024-09-18T11:54:46.172769+0000","last_clean":"2024-09-18T11:54:46.172769+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:46.172769+0000","last_undegraded":"2024-09-18T11:54:46.172769+0000","last_fullsized":"2024-09-18T11:54:46.172769+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9909494,"num_objects":1210,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11680","reported_seq":11797,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:48.641001+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:48.641001+0000","last_peered":"2024-09-18T11:54:48.641001+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:48.641001+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9239187,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11679,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11769","reported_seq":11863,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.995903+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:49.995903+0000","last_peered":"2024-09-18T11:54:49.995903+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:49.995903+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9475318,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11777","reported_seq":11898,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.048431+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:49.048431+0000","last_peered":"2024-09-18T11:54:49.048431+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:49.048431+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9155830,"num_objects":1118,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11733","reported_seq":11852,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.343342+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:49.343342+0000","last_peered":"2024-09-18T11:54:49.343342+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:49.343342+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8626176,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11879","reported_seq":11986,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.912382+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:49.912382+0000","last_peered":"2024-09-18T11:54:49.912382+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:49.912382+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311478,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-979}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11706","reported_seq":14563,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.934144+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:49.934144+0000","last_peered":"2024-09-18T11:54:49.934144+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:49.934144+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9573622,"num_objects":1169,"num_object_clones":0,"num_object_copies":4676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12012","reported_seq":16365,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:49.348384+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:49.348384+0000","last_peered":"2024-09-18T11:54:49.348384+0000","last_clean":"2024-09-18T11:54:49.348384+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:49.348384+0000","last_undegraded":"2024-09-18T11:54:49.348384+0000","last_fullsized":"2024-09-18T11:54:49.348384+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9529836,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12009,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11467","reported_seq":11581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:48.856998+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:48.856998+0000","last_peered":"2024-09-18T11:54:48.856998+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:48.856998+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9229558,"num_objects":1127,"num_object_clones":0,"num_object_copies":4508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11465,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-836}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11844","reported_seq":11965,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.126123+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:49.126123+0000","last_peered":"2024-09-18T11:54:49.126123+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:49.126123+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9736049,"num_objects":1189,"num_object_clones":0,"num_object_copies":4756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11841,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11939","reported_seq":12032,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.208980+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:49.208980+0000","last_peered":"2024-09-18T11:54:49.208980+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:49.208980+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8944251,"num_objects":1092,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11938,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11852","reported_seq":11950,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:48.587733+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:48.587733+0000","last_peered":"2024-09-18T11:54:48.587733+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:48.587733+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9132667,"num_objects":1115,"num_object_clones":0,"num_object_copies":4460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11884","reported_seq":12009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.156691+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:49.156691+0000","last_peered":"2024-09-18T11:54:49.156691+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:49.156691+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9876726,"num_objects":1206,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11882,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-917}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11765","reported_seq":16051,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:46.246382+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:46.246382+0000","last_peered":"2024-09-18T11:54:46.246382+0000","last_clean":"2024-09-18T11:54:46.246382+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:46.246382+0000","last_undegraded":"2024-09-18T11:54:46.246382+0000","last_fullsized":"2024-09-18T11:54:46.246382+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9319670,"num_objects":1138,"num_object_clones":0,"num_object_copies":4552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11763,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11563","reported_seq":11653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.933508+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:49.933508+0000","last_peered":"2024-09-18T11:54:49.933508+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:49.933508+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9368822,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11561,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149742988,"num_objects":18285,"num_object_clones":0,"num_object_copies":73140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188802,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8013,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300965888,"data_stored":300965888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160829,"ondisk_log_size":160829,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992676,"kb_used_data":73304,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93379164,"statfs":{"total":96636764160,"available":95620263936,"internally_reserved":0,"allocated":75063296,"data_stored":74879475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.25},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.2829999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":4.6689999999999996},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":1.8089999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.8280000000000003},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.101}]}]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1001116,"kb_used_data":73940,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93370724,"statfs":{"total":96636764160,"available":95611621376,"internally_reserved":0,"allocated":75714560,"data_stored":75530739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":10.430999999999999},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":10.464}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":5.4189999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":5.4640000000000004},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.5659999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662976,"kb_used_data":76792,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93708864,"statfs":{"total":96636764160,"available":95957876736,"internally_reserved":0,"allocated":78635008,"data_stored":78451187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":36.302999999999997},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":21.343}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":7.8639999999999999},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.35999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":4.5650000000000004},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":7.9029999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000972,"kb_used_data":73332,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93370868,"statfs":{"total":96636764160,"available":95611768832,"internally_reserved":0,"allocated":75091968,"data_stored":74908147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,4,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":2.9820000000000002},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.335}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.6379999999999999},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.3760000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":1.603},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.666}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74059776,"data_stored":74059776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77524992,"data_stored":77524992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":74874880,"data_stored":74874880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74506240,"data_stored":74506240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:53.527 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:54.111 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:54.136 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-18T11:54:54.075309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147031436,"num_objects":17954,"num_object_clones":0,"num_object_copies":71816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189133,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8013,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160660,"ondisk_log_size":160660,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3644816,"kb_used_data":288592,"kb_used_omap":14,"kb_used_meta":3356145,"kb_avail":373842544,"statfs":{"total":386547056640,"available":382814765056,"internally_reserved":0,"allocated":295518208,"data_stored":294782924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3436692507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,5,6,15,13,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18341888,"num_objects":-2239,"num_object_clones":0,"num_object_copies":-8956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2239,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004467"},"pg_stats":[{"pgid":"1.f","version":"37'12113","reported_seq":12214,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.210849+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:49.210849+0000","last_peered":"2024-09-18T11:54:49.210849+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:49.210849+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9314304,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12113,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11920","reported_seq":16212,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.291240+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:51.291240+0000","last_peered":"2024-09-18T11:54:51.291240+0000","last_clean":"2024-09-18T11:54:51.291240+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:51.291240+0000","last_undegraded":"2024-09-18T11:54:51.291240+0000","last_fullsized":"2024-09-18T11:54:51.291240+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9303286,"num_objects":1136,"num_object_clones":0,"num_object_copies":4544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11745","reported_seq":11862,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.942886+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:53.942886+0000","last_peered":"2024-09-18T11:54:53.942886+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:53.942886+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8706707,"num_objects":1064,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1064,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11744,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-976}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11769","reported_seq":11863,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.995903+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:49.995903+0000","last_peered":"2024-09-18T11:54:49.995903+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:49.995903+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9475318,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11777","reported_seq":11898,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.048431+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:49.048431+0000","last_peered":"2024-09-18T11:54:49.048431+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:49.048431+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9155830,"num_objects":1118,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11733","reported_seq":11852,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.343342+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:49.343342+0000","last_peered":"2024-09-18T11:54:49.343342+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:49.343342+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8626176,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11879","reported_seq":11986,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.912382+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:49.912382+0000","last_peered":"2024-09-18T11:54:49.912382+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:49.912382+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311478,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-979}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11706","reported_seq":14563,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.934144+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:49.934144+0000","last_peered":"2024-09-18T11:54:49.934144+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:49.934144+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9573622,"num_objects":1169,"num_object_clones":0,"num_object_copies":4676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12012","reported_seq":16365,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:49.348384+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:49.348384+0000","last_peered":"2024-09-18T11:54:49.348384+0000","last_clean":"2024-09-18T11:54:49.348384+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:49.348384+0000","last_undegraded":"2024-09-18T11:54:49.348384+0000","last_fullsized":"2024-09-18T11:54:49.348384+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9529836,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12009,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11534","reported_seq":11648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.745534+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:53.745534+0000","last_peered":"2024-09-18T11:54:53.745534+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:53.745534+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8680694,"num_objects":1060,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11532,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11844","reported_seq":11965,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.126123+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:49.126123+0000","last_peered":"2024-09-18T11:54:49.126123+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:49.126123+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9736049,"num_objects":1189,"num_object_clones":0,"num_object_copies":4756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11841,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11939","reported_seq":12032,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.208980+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:49.208980+0000","last_peered":"2024-09-18T11:54:49.208980+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:49.208980+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8944251,"num_objects":1092,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11938,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11918","reported_seq":12016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.749717+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:53.749717+0000","last_peered":"2024-09-18T11:54:53.749717+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:53.749717+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8591995,"num_objects":1049,"num_object_clones":0,"num_object_copies":4196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11917,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-997}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11884","reported_seq":12009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.156691+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:49.156691+0000","last_peered":"2024-09-18T11:54:49.156691+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:49.156691+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9876726,"num_objects":1206,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11882,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-917}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11824","reported_seq":16113,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.049770+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:51.049770+0000","last_peered":"2024-09-18T11:54:51.049770+0000","last_clean":"2024-09-18T11:54:51.049770+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:51.049770+0000","last_undegraded":"2024-09-18T11:54:51.049770+0000","last_fullsized":"2024-09-18T11:54:51.049770+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8836342,"num_objects":1079,"num_object_clones":0,"num_object_copies":4316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11822,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11563","reported_seq":11653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.933508+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:49.933508+0000","last_peered":"2024-09-18T11:54:49.933508+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:49.933508+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9368822,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11561,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":147031436,"num_objects":17954,"num_object_clones":0,"num_object_copies":71816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189133,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8013,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":292667392,"data_stored":292667392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160660,"ondisk_log_size":160660,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992676,"kb_used_data":73304,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93379164,"statfs":{"total":96636764160,"available":95620263936,"internally_reserved":0,"allocated":75063296,"data_stored":74879475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.25},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.2829999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":4.6689999999999996},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":1.8089999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.8280000000000003},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.101}]}]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":988848,"kb_used_data":69992,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93382992,"statfs":{"total":96636764160,"available":95624183808,"internally_reserved":0,"allocated":71671808,"data_stored":71487987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,5,3,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.1429999999999998},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.173}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.117999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.85399999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.0980000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.089}]}]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662320,"kb_used_data":71964,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93709520,"statfs":{"total":96636764160,"available":95958548480,"internally_reserved":0,"allocated":73691136,"data_stored":73507315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":10.627000000000001},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.3420000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":19.382000000000001},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":1.5189999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":1.423},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.484000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000972,"kb_used_data":73332,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93370868,"statfs":{"total":96636764160,"available":95611768832,"internally_reserved":0,"allocated":75091968,"data_stored":74908147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,4,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":2.9820000000000002},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.335}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.6379999999999999},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.3760000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":1.603},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.666}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74059776,"data_stored":74059776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73351168,"data_stored":73351168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":70750208,"data_stored":70750208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74506240,"data_stored":74506240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:54.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:54.686 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:54.686 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:54.706 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-18T11:54:54.075309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147031436,"num_objects":17954,"num_object_clones":0,"num_object_copies":71816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189133,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8013,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160660,"ondisk_log_size":160660,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3644816,"kb_used_data":288592,"kb_used_omap":14,"kb_used_meta":3356145,"kb_avail":373842544,"statfs":{"total":386547056640,"available":382814765056,"internally_reserved":0,"allocated":295518208,"data_stored":294782924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3436692507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,5,6,15,13,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18341888,"num_objects":-2239,"num_object_clones":0,"num_object_copies":-8956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2239,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004467"},"pg_stats":[{"pgid":"1.f","version":"37'12113","reported_seq":12214,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.210849+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:49.210849+0000","last_peered":"2024-09-18T11:54:49.210849+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:49.210849+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9314304,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12113,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11920","reported_seq":16212,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.291240+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:51.291240+0000","last_peered":"2024-09-18T11:54:51.291240+0000","last_clean":"2024-09-18T11:54:51.291240+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:51.291240+0000","last_undegraded":"2024-09-18T11:54:51.291240+0000","last_fullsized":"2024-09-18T11:54:51.291240+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9303286,"num_objects":1136,"num_object_clones":0,"num_object_copies":4544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11745","reported_seq":11862,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.942886+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:53.942886+0000","last_peered":"2024-09-18T11:54:53.942886+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:53.942886+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8706707,"num_objects":1064,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1064,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11744,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-976}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11769","reported_seq":11863,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.995903+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:49.995903+0000","last_peered":"2024-09-18T11:54:49.995903+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:49.995903+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9475318,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11777","reported_seq":11898,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.048431+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:49.048431+0000","last_peered":"2024-09-18T11:54:49.048431+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:49.048431+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9155830,"num_objects":1118,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11733","reported_seq":11852,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.343342+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:49.343342+0000","last_peered":"2024-09-18T11:54:49.343342+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:49.343342+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8626176,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11879","reported_seq":11986,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.912382+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:49.912382+0000","last_peered":"2024-09-18T11:54:49.912382+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:49.912382+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311478,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-979}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11706","reported_seq":14563,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.934144+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:49.934144+0000","last_peered":"2024-09-18T11:54:49.934144+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:49.934144+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9573622,"num_objects":1169,"num_object_clones":0,"num_object_copies":4676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12012","reported_seq":16365,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:49.348384+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:49.348384+0000","last_peered":"2024-09-18T11:54:49.348384+0000","last_clean":"2024-09-18T11:54:49.348384+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:49.348384+0000","last_undegraded":"2024-09-18T11:54:49.348384+0000","last_fullsized":"2024-09-18T11:54:49.348384+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9529836,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12009,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11534","reported_seq":11648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.745534+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:53.745534+0000","last_peered":"2024-09-18T11:54:53.745534+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:53.745534+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8680694,"num_objects":1060,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11532,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11844","reported_seq":11965,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.126123+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:49.126123+0000","last_peered":"2024-09-18T11:54:49.126123+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:49.126123+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9736049,"num_objects":1189,"num_object_clones":0,"num_object_copies":4756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11841,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11939","reported_seq":12032,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.208980+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:49.208980+0000","last_peered":"2024-09-18T11:54:49.208980+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:49.208980+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8944251,"num_objects":1092,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11938,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11918","reported_seq":12016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.749717+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:53.749717+0000","last_peered":"2024-09-18T11:54:53.749717+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:53.749717+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8591995,"num_objects":1049,"num_object_clones":0,"num_object_copies":4196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11917,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-997}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11884","reported_seq":12009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.156691+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:49.156691+0000","last_peered":"2024-09-18T11:54:49.156691+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:49.156691+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9876726,"num_objects":1206,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11882,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-917}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11824","reported_seq":16113,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.049770+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:51.049770+0000","last_peered":"2024-09-18T11:54:51.049770+0000","last_clean":"2024-09-18T11:54:51.049770+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:51.049770+0000","last_undegraded":"2024-09-18T11:54:51.049770+0000","last_fullsized":"2024-09-18T11:54:51.049770+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8836342,"num_objects":1079,"num_object_clones":0,"num_object_copies":4316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11822,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11563","reported_seq":11653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.933508+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:49.933508+0000","last_peered":"2024-09-18T11:54:49.933508+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:49.933508+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9368822,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11561,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":147031436,"num_objects":17954,"num_object_clones":0,"num_object_copies":71816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189133,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8013,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":292667392,"data_stored":292667392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160660,"ondisk_log_size":160660,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992676,"kb_used_data":73304,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93379164,"statfs":{"total":96636764160,"available":95620263936,"internally_reserved":0,"allocated":75063296,"data_stored":74879475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.25},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.2829999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":4.6689999999999996},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":1.8089999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.8280000000000003},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.101}]}]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":988848,"kb_used_data":69992,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93382992,"statfs":{"total":96636764160,"available":95624183808,"internally_reserved":0,"allocated":71671808,"data_stored":71487987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,5,3,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.1429999999999998},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.173}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.117999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.85399999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.0980000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.089}]}]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662320,"kb_used_data":71964,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93709520,"statfs":{"total":96636764160,"available":95958548480,"internally_reserved":0,"allocated":73691136,"data_stored":73507315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":10.627000000000001},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.3420000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":19.382000000000001},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":1.5189999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":1.423},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.484000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000972,"kb_used_data":73332,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93370868,"statfs":{"total":96636764160,"available":95611768832,"internally_reserved":0,"allocated":75091968,"data_stored":74908147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,4,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":2.9820000000000002},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.335}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.6379999999999999},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.3760000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":1.603},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.666}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74059776,"data_stored":74059776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73351168,"data_stored":73351168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":70750208,"data_stored":70750208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74506240,"data_stored":74506240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:54.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:54:54.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12113", 'reported_seq': 12214, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:49.210849+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:49.210849+0000', 'last_peered': '2024-09-18T11:54:49.210849+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:49.210849+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9314304, 'num_objects': 1137, 'num_object_clones': 0, 'num_object_copies': 4548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12113, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:54.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:54:54.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11745", 'reported_seq': 11862, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:53.942886+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:53.942886+0000', 'last_peered': '2024-09-18T11:54:53.942886+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:53.942886+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8706707, 'num_objects': 1064, 'num_object_clones': 0, 'num_object_copies': 4256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1064, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11744, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -976}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:54.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:54:54.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11769", 'reported_seq': 11863, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:49.995903+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:54:49.995903+0000', 'last_peered': '2024-09-18T11:54:49.995903+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:54:49.995903+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9475318, 'num_objects': 1157, 'num_object_clones': 0, 'num_object_copies': 4628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11767, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -944}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:54.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:54:54.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11777", 'reported_seq': 11898, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:49.048431+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:49.048431+0000', 'last_peered': '2024-09-18T11:54:49.048431+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:49.048431+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9155830, 'num_objects': 1118, 'num_object_clones': 0, 'num_object_copies': 4472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11775, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -934}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:54.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:54:54.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11733", 'reported_seq': 11852, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:49.343342+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:49.343342+0000', 'last_peered': '2024-09-18T11:54:49.343342+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:49.343342+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8626176, 'num_objects': 1053, 'num_object_clones': 0, 'num_object_copies': 4212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11733, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1066}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:54.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:54:54.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11879", 'reported_seq': 11986, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:49.912382+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:49.912382+0000', 'last_peered': '2024-09-18T11:54:49.912382+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:49.912382+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9311478, 'num_objects': 1137, 'num_object_clones': 0, 'num_object_copies': 4548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -979}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:54.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:54:54.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11706", 'reported_seq': 14563, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:49.934144+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:54:49.934144+0000', 'last_peered': '2024-09-18T11:54:49.934144+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:54:49.934144+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9573622, 'num_objects': 1169, 'num_object_clones': 0, 'num_object_copies': 4676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 642, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11704, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1396, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:54.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:54:54.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11534", 'reported_seq': 11648, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:53.745534+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:53.745534+0000', 'last_peered': '2024-09-18T11:54:53.745534+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:53.745534+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8680694, 'num_objects': 1060, 'num_object_clones': 0, 'num_object_copies': 4240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11532, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -903}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:54.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:54:54.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11844", 'reported_seq': 11965, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:49.126123+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:49.126123+0000', 'last_peered': '2024-09-18T11:54:49.126123+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:49.126123+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9736049, 'num_objects': 1189, 'num_object_clones': 0, 'num_object_copies': 4756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11841, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -845}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:54.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:54:54.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11939", 'reported_seq': 12032, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:49.208980+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:49.208980+0000', 'last_peered': '2024-09-18T11:54:49.208980+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:49.208980+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8944251, 'num_objects': 1092, 'num_object_clones': 0, 'num_object_copies': 4368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11938, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -992}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:54.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:54:54.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11918", 'reported_seq': 12016, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:53.749717+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:53.749717+0000', 'last_peered': '2024-09-18T11:54:53.749717+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:53.749717+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8591995, 'num_objects': 1049, 'num_object_clones': 0, 'num_object_copies': 4196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11917, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -997}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:54:54.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:54:54.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11884", 'reported_seq': 12009, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:49.156691+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:49.156691+0000', 'last_peered': '2024-09-18T11:54:49.156691+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:49.156691+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9876726, 'num_objects': 1206, 'num_object_clones': 0, 'num_object_copies': 4824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11882, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -917}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:54:54.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:54:54.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11563", 'reported_seq': 11653, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:49.933508+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:49.933508+0000', 'last_peered': '2024-09-18T11:54:49.933508+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:49.933508+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9368822, 'num_objects': 1144, 'num_object_clones': 0, 'num_object_copies': 4576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1144, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11561, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -858}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:54:54.710 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:55.379 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:55.379 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25443/71816 objects degraded (35.428%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":693,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":17954,"data_bytes":147031436,"bytes_used":3732291584,"bytes_avail":382814765056,"bytes_total":386547056640,"degraded_objects":25443,"degraded_total":71816,"degraded_ratio":0.354280383201515,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":186},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:54:55.413 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:54:55.414 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:55.935 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:55.936 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:55.972 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-18T11:54:54.075309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147031436,"num_objects":17954,"num_object_clones":0,"num_object_copies":71816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189133,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8013,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160660,"ondisk_log_size":160660,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3644816,"kb_used_data":288592,"kb_used_omap":14,"kb_used_meta":3356145,"kb_avail":373842544,"statfs":{"total":386547056640,"available":382814765056,"internally_reserved":0,"allocated":295518208,"data_stored":294782924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3436692507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,5,6,15,13,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18341888,"num_objects":-2239,"num_object_clones":0,"num_object_copies":-8956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2239,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004467"},"pg_stats":[{"pgid":"1.f","version":"37'12113","reported_seq":12214,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.210849+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:49.210849+0000","last_peered":"2024-09-18T11:54:49.210849+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:49.210849+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9314304,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12113,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11920","reported_seq":16212,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.291240+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:51.291240+0000","last_peered":"2024-09-18T11:54:51.291240+0000","last_clean":"2024-09-18T11:54:51.291240+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:51.291240+0000","last_undegraded":"2024-09-18T11:54:51.291240+0000","last_fullsized":"2024-09-18T11:54:51.291240+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9303286,"num_objects":1136,"num_object_clones":0,"num_object_copies":4544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11745","reported_seq":11862,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.942886+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:53.942886+0000","last_peered":"2024-09-18T11:54:53.942886+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:53.942886+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8706707,"num_objects":1064,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1064,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11744,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-976}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11769","reported_seq":11863,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.995903+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:49.995903+0000","last_peered":"2024-09-18T11:54:49.995903+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:49.995903+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9475318,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11777","reported_seq":11898,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.048431+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:49.048431+0000","last_peered":"2024-09-18T11:54:49.048431+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:49.048431+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9155830,"num_objects":1118,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11733","reported_seq":11852,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.343342+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:49.343342+0000","last_peered":"2024-09-18T11:54:49.343342+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:49.343342+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8626176,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11879","reported_seq":11986,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.912382+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:49.912382+0000","last_peered":"2024-09-18T11:54:49.912382+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:49.912382+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311478,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-979}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11706","reported_seq":14563,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.934144+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:49.934144+0000","last_peered":"2024-09-18T11:54:49.934144+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:49.934144+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9573622,"num_objects":1169,"num_object_clones":0,"num_object_copies":4676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12012","reported_seq":16365,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:49.348384+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:49.348384+0000","last_peered":"2024-09-18T11:54:49.348384+0000","last_clean":"2024-09-18T11:54:49.348384+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:49.348384+0000","last_undegraded":"2024-09-18T11:54:49.348384+0000","last_fullsized":"2024-09-18T11:54:49.348384+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9529836,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12009,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11534","reported_seq":11648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.745534+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:53.745534+0000","last_peered":"2024-09-18T11:54:53.745534+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:53.745534+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8680694,"num_objects":1060,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11532,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11844","reported_seq":11965,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.126123+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:49.126123+0000","last_peered":"2024-09-18T11:54:49.126123+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:49.126123+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9736049,"num_objects":1189,"num_object_clones":0,"num_object_copies":4756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11841,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11939","reported_seq":12032,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.208980+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:49.208980+0000","last_peered":"2024-09-18T11:54:49.208980+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:49.208980+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8944251,"num_objects":1092,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11938,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11918","reported_seq":12016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.749717+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:53.749717+0000","last_peered":"2024-09-18T11:54:53.749717+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:53.749717+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8591995,"num_objects":1049,"num_object_clones":0,"num_object_copies":4196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11917,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-997}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11884","reported_seq":12009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.156691+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:49.156691+0000","last_peered":"2024-09-18T11:54:49.156691+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:49.156691+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9876726,"num_objects":1206,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11882,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-917}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11824","reported_seq":16113,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.049770+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:51.049770+0000","last_peered":"2024-09-18T11:54:51.049770+0000","last_clean":"2024-09-18T11:54:51.049770+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:51.049770+0000","last_undegraded":"2024-09-18T11:54:51.049770+0000","last_fullsized":"2024-09-18T11:54:51.049770+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8836342,"num_objects":1079,"num_object_clones":0,"num_object_copies":4316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11822,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11563","reported_seq":11653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:49.933508+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:49.933508+0000","last_peered":"2024-09-18T11:54:49.933508+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:49.933508+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9368822,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11561,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":147031436,"num_objects":17954,"num_object_clones":0,"num_object_copies":71816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189133,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8013,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":292667392,"data_stored":292667392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160660,"ondisk_log_size":160660,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992676,"kb_used_data":73304,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93379164,"statfs":{"total":96636764160,"available":95620263936,"internally_reserved":0,"allocated":75063296,"data_stored":74879475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.25},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.2829999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":4.6689999999999996},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":1.8089999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.8280000000000003},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.101}]}]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":988848,"kb_used_data":69992,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93382992,"statfs":{"total":96636764160,"available":95624183808,"internally_reserved":0,"allocated":71671808,"data_stored":71487987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,5,3,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.1429999999999998},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.173}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.117999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.85399999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.0980000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.089}]}]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662320,"kb_used_data":71964,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93709520,"statfs":{"total":96636764160,"available":95958548480,"internally_reserved":0,"allocated":73691136,"data_stored":73507315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":10.627000000000001},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.3420000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":19.382000000000001},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":1.5189999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":1.423},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.484000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000972,"kb_used_data":73332,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93370868,"statfs":{"total":96636764160,"available":95611768832,"internally_reserved":0,"allocated":75091968,"data_stored":74908147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,4,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":2.9820000000000002},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.335}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.6379999999999999},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.3760000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":1.603},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.666}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74059776,"data_stored":74059776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73351168,"data_stored":73351168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":70750208,"data_stored":70750208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74506240,"data_stored":74506240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:58.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:59.516 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:54:59.517 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:54:59.542 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-18T11:54:58.076905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":140371340,"num_objects":17141,"num_object_clones":0,"num_object_copies":68564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189946,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8069,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3628656,"kb_used_data":276604,"kb_used_omap":14,"kb_used_meta":3351985,"kb_avail":373858704,"statfs":{"total":386547056640,"available":382831312896,"internally_reserved":0,"allocated":283242496,"data_stored":282507212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3432432667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,2,19,11,10,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17768448,"num_objects":-2169,"num_object_clones":0,"num_object_copies":-8676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004509"},"pg_stats":[{"pgid":"1.f","version":"37'12177","reported_seq":12278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.293435+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:54.293435+0000","last_peered":"2024-09-18T11:54:54.293435+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:54.293435+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8790016,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12177,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11987","reported_seq":16279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:56.409123+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:56.409123+0000","last_peered":"2024-09-18T11:54:56.409123+0000","last_clean":"2024-09-18T11:54:56.409123+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:56.409123+0000","last_undegraded":"2024-09-18T11:54:56.409123+0000","last_fullsized":"2024-09-18T11:54:56.409123+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8754422,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11745","reported_seq":11862,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.942886+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:53.942886+0000","last_peered":"2024-09-18T11:54:53.942886+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:53.942886+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8706707,"num_objects":1064,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1064,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11744,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-976}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11835","reported_seq":11929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.654198+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:54.654198+0000","last_peered":"2024-09-18T11:54:54.654198+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:54.654198+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8934646,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11833,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11833","reported_seq":11954,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.266123+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:54.266123+0000","last_peered":"2024-09-18T11:54:54.266123+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:54.266123+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8697078,"num_objects":1062,"num_object_clones":0,"num_object_copies":4248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11831,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-990}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11788","reported_seq":11907,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.330434+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:54.330434+0000","last_peered":"2024-09-18T11:54:54.330434+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:54.330434+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8175616,"num_objects":998,"num_object_clones":0,"num_object_copies":3992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11963","reported_seq":12070,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.957187+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:54.957187+0000","last_peered":"2024-09-18T11:54:54.957187+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:54.957187+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8623350,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11961,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1063}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11773","reported_seq":14742,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.979755+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:54.979755+0000","last_peered":"2024-09-18T11:54:54.979755+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:54.979755+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9024758,"num_objects":1102,"num_object_clones":0,"num_object_copies":4408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":586,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1452,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12063","reported_seq":16416,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:54.004189+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:54.004189+0000","last_peered":"2024-09-18T11:54:54.004189+0000","last_clean":"2024-09-18T11:54:54.004189+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:54.004189+0000","last_undegraded":"2024-09-18T11:54:54.004189+0000","last_fullsized":"2024-09-18T11:54:54.004189+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9112044,"num_objects":1113,"num_object_clones":0,"num_object_copies":4452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12060,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11534","reported_seq":11648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.745534+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:53.745534+0000","last_peered":"2024-09-18T11:54:53.745534+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:53.745534+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8680694,"num_objects":1060,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11532,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11907","reported_seq":12027,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.378872+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:54.378872+0000","last_peered":"2024-09-18T11:54:54.378872+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:54.378872+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9219953,"num_objects":1126,"num_object_clones":0,"num_object_copies":4504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11904,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11992","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.284490+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:54.284490+0000","last_peered":"2024-09-18T11:54:54.284490+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:54.284490+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8510075,"num_objects":1039,"num_object_clones":0,"num_object_copies":4156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11991,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1045}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11918","reported_seq":12016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:53.749717+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:53.749717+0000","last_peered":"2024-09-18T11:54:53.749717+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:53.749717+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8591995,"num_objects":1049,"num_object_clones":0,"num_object_copies":4196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11917,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-997}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11952","reported_seq":12076,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.354350+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:54.354350+0000","last_peered":"2024-09-18T11:54:54.354350+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:54.354350+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9319670,"num_objects":1138,"num_object_clones":0,"num_object_copies":4552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11950,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-985}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11890","reported_seq":16179,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:56.403776+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:56.403776+0000","last_peered":"2024-09-18T11:54:56.403776+0000","last_clean":"2024-09-18T11:54:56.403776+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:56.403776+0000","last_undegraded":"2024-09-18T11:54:56.403776+0000","last_fullsized":"2024-09-18T11:54:56.403776+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8295670,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11888,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11616","reported_seq":11706,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.731713+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:54.731713+0000","last_peered":"2024-09-18T11:54:54.731713+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:54.731713+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8934646,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":140371340,"num_objects":17141,"num_object_clones":0,"num_object_copies":68564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189946,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8069,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280309760,"data_stored":280309760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992776,"kb_used_data":69248,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93379064,"statfs":{"total":96636764160,"available":95620161536,"internally_reserved":0,"allocated":70909952,"data_stored":70726131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,11,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":3.5590000000000002},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":3.4849999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":4.2539999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":13.305999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":4.2240000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":988848,"kb_used_data":69992,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93382992,"statfs":{"total":96636764160,"available":95624183808,"internally_reserved":0,"allocated":71671808,"data_stored":71487987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,5,3,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.1429999999999998},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.173}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.117999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.85399999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.0980000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.089}]}]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662584,"kb_used_data":68072,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93709256,"statfs":{"total":96636764160,"available":95958278144,"internally_reserved":0,"allocated":69705728,"data_stored":69521907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":2.0089999999999999},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":5.1509999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":15.715},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.63600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":5.1150000000000002},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":15.675000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":984448,"kb_used_data":69292,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93387392,"statfs":{"total":96636764160,"available":95628689408,"internally_reserved":0,"allocated":70955008,"data_stored":70771187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,3,2,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":9.5890000000000004},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.8129999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.6980000000000004},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.843}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":9.5540000000000003},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.8650000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69971968,"data_stored":69971968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69181440,"data_stored":69181440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":70750208,"data_stored":70750208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70406144,"data_stored":70406144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:59.544 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:00.122 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:00.143 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-18T11:55:00.077717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135202188,"num_objects":16510,"num_object_clones":0,"num_object_copies":66040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190577,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8069,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3624868,"kb_used_data":268656,"kb_used_omap":14,"kb_used_meta":3356145,"kb_avail":373862492,"statfs":{"total":386547056640,"available":382835191808,"internally_reserved":0,"allocated":275103744,"data_stored":274368460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3436692507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,3,9,11,13,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21798912,"num_objects":-2661,"num_object_clones":0,"num_object_copies":-10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2661,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004574"},"pg_stats":[{"pgid":"1.f","version":"37'12237","reported_seq":12337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.342323+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:59.342323+0000","last_peered":"2024-09-18T11:54:59.342323+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:59.342323+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8298496,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11987","reported_seq":16279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:56.409123+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:56.409123+0000","last_peered":"2024-09-18T11:54:56.409123+0000","last_clean":"2024-09-18T11:54:56.409123+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:56.409123+0000","last_undegraded":"2024-09-18T11:54:56.409123+0000","last_fullsized":"2024-09-18T11:54:56.409123+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8754422,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11812","reported_seq":11929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.929100+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:58.929100+0000","last_peered":"2024-09-18T11:54:58.929100+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:58.929100+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8157843,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11811,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11835","reported_seq":11929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.654198+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:54.654198+0000","last_peered":"2024-09-18T11:54:54.654198+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:54.654198+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8934646,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11833,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11894","reported_seq":12015,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.482455+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:59.482455+0000","last_peered":"2024-09-18T11:54:59.482455+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:59.482455+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8197366,"num_objects":1001,"num_object_clones":0,"num_object_copies":4004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11892,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11845","reported_seq":11964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.462130+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:59.462130+0000","last_peered":"2024-09-18T11:54:59.462130+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:59.462130+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7708672,"num_objects":941,"num_object_clones":0,"num_object_copies":3764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11845,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11963","reported_seq":12070,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.957187+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:54.957187+0000","last_peered":"2024-09-18T11:54:54.957187+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:54.957187+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8623350,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11961,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1063}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11773","reported_seq":14742,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.979755+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:54.979755+0000","last_peered":"2024-09-18T11:54:54.979755+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:54.979755+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9024758,"num_objects":1102,"num_object_clones":0,"num_object_copies":4408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":586,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1452,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12134","reported_seq":16487,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:59.382120+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:59.382120+0000","last_peered":"2024-09-18T11:54:59.382120+0000","last_clean":"2024-09-18T11:54:59.382120+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:59.382120+0000","last_undegraded":"2024-09-18T11:54:59.382120+0000","last_fullsized":"2024-09-18T11:54:59.382120+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8530412,"num_objects":1042,"num_object_clones":0,"num_object_copies":4168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11597","reported_seq":11710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.785500+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:58.785500+0000","last_peered":"2024-09-18T11:54:58.785500+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:58.785500+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8164598,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-966}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11964","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.316033+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:59.316033+0000","last_peered":"2024-09-18T11:54:59.316033+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:59.316033+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8753009,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11961,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12056","reported_seq":12149,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.365749+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:59.365749+0000","last_peered":"2024-09-18T11:54:59.365749+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:59.365749+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7985787,"num_objects":975,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12055,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11979","reported_seq":12077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.976141+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:58.976141+0000","last_peered":"2024-09-18T11:54:58.976141+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:58.976141+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8092283,"num_objects":988,"num_object_clones":0,"num_object_copies":3952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1058}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12022","reported_seq":12147,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.324289+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:59.324289+0000","last_peered":"2024-09-18T11:54:59.324289+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:59.324289+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746230,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12020,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1055}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11890","reported_seq":16179,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:56.403776+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:56.403776+0000","last_peered":"2024-09-18T11:54:56.403776+0000","last_clean":"2024-09-18T11:54:56.403776+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:56.403776+0000","last_undegraded":"2024-09-18T11:54:56.403776+0000","last_fullsized":"2024-09-18T11:54:56.403776+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8295670,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11888,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11616","reported_seq":11706,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.731713+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:54.731713+0000","last_peered":"2024-09-18T11:54:54.731713+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:54.731713+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8934646,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135202188,"num_objects":16510,"num_object_clones":0,"num_object_copies":66040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190577,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8069,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":272027648,"data_stored":272027648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992964,"kb_used_data":65276,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93378876,"statfs":{"total":96636764160,"available":95619969024,"internally_reserved":0,"allocated":66842624,"data_stored":66658803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.609},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.53200000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.6489999999999991},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":0.95599999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":5.9809999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":984872,"kb_used_data":66016,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93386968,"statfs":{"total":96636764160,"available":95628255232,"internally_reserved":0,"allocated":67600384,"data_stored":67416563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,5,3,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.1429999999999998},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.173}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.117999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.85399999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.0980000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.089}]}]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662584,"kb_used_data":68072,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93709256,"statfs":{"total":96636764160,"available":95958278144,"internally_reserved":0,"allocated":69705728,"data_stored":69521907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":2.0089999999999999},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":5.1509999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":15.715},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.63600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":5.1150000000000002},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":15.675000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":984448,"kb_used_data":69292,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93387392,"statfs":{"total":96636764160,"available":95628689408,"internally_reserved":0,"allocated":70955008,"data_stored":70771187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,3,2,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":9.5890000000000004},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.8129999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.6980000000000004},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.843}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":9.5540000000000003},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.8650000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69971968,"data_stored":69971968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69181440,"data_stored":69181440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66633728,"data_stored":66633728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":66240512,"data_stored":66240512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:00.144 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:00.144 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12237", 'reported_seq': 12337, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.342323+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:59.342323+0000', 'last_peered': '2024-09-18T11:54:59.342323+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:59.342323+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8298496, 'num_objects': 1013, 'num_object_clones': 0, 'num_object_copies': 4052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12237, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1090}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:00.144 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:00.144 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11812", 'reported_seq': 11929, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:58.929100+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:58.929100+0000', 'last_peered': '2024-09-18T11:54:58.929100+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:58.929100+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8157843, 'num_objects': 997, 'num_object_clones': 0, 'num_object_copies': 3988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 997, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11811, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1043}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:00.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:00.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11835", 'reported_seq': 11929, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:54.654198+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:54:54.654198+0000', 'last_peered': '2024-09-18T11:54:54.654198+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:54:54.654198+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8934646, 'num_objects': 1091, 'num_object_clones': 0, 'num_object_copies': 4364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11833, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1010}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:00.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:00.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11894", 'reported_seq': 12015, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.482455+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:59.482455+0000', 'last_peered': '2024-09-18T11:54:59.482455+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:59.482455+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8197366, 'num_objects': 1001, 'num_object_clones': 0, 'num_object_copies': 4004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11892, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1051}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:00.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:00.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11845", 'reported_seq': 11964, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.462130+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:59.462130+0000', 'last_peered': '2024-09-18T11:54:59.462130+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:59.462130+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7708672, 'num_objects': 941, 'num_object_clones': 0, 'num_object_copies': 3764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11845, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1178}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:00.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:00.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11963", 'reported_seq': 12070, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:54.957187+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:54.957187+0000', 'last_peered': '2024-09-18T11:54:54.957187+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:54.957187+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8623350, 'num_objects': 1053, 'num_object_clones': 0, 'num_object_copies': 4212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11961, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1063}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:00.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:00.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11773", 'reported_seq': 14742, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:54.979755+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:54:54.979755+0000', 'last_peered': '2024-09-18T11:54:54.979755+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:54:54.979755+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9024758, 'num_objects': 1102, 'num_object_clones': 0, 'num_object_copies': 4408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 586, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11771, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1452, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 516}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11597", 'reported_seq': 11710, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:58.785500+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:58.785500+0000', 'last_peered': '2024-09-18T11:54:58.785500+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:58.785500+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8164598, 'num_objects': 997, 'num_object_clones': 0, 'num_object_copies': 3988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -966}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11964", 'reported_seq': 12085, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.316033+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:59.316033+0000', 'last_peered': '2024-09-18T11:54:59.316033+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:59.316033+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8753009, 'num_objects': 1069, 'num_object_clones': 0, 'num_object_copies': 4276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11961, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -965}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12056", 'reported_seq': 12149, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.365749+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:59.365749+0000', 'last_peered': '2024-09-18T11:54:59.365749+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:59.365749+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7985787, 'num_objects': 975, 'num_object_clones': 0, 'num_object_copies': 3900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12055, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11979", 'reported_seq': 12077, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:58.976141+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:58.976141+0000', 'last_peered': '2024-09-18T11:54:58.976141+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:58.976141+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8092283, 'num_objects': 988, 'num_object_clones': 0, 'num_object_copies': 3952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11978, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1058}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12022", 'reported_seq': 12147, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.324289+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:59.324289+0000', 'last_peered': '2024-09-18T11:54:59.324289+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:59.324289+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8746230, 'num_objects': 1068, 'num_object_clones': 0, 'num_object_copies': 4272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12020, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1055}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:00.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:00.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11616", 'reported_seq': 11706, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:54.731713+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:54.731713+0000', 'last_peered': '2024-09-18T11:54:54.731713+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:54.731713+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8934646, 'num_objects': 1091, 'num_object_clones': 0, 'num_object_copies': 4364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11614, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:00.147 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:00.832 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:00.832 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25387/66040 objects degraded (38.442%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":698,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16510,"data_bytes":135202188,"bytes_used":3711864832,"bytes_avail":382835191808,"bytes_total":386547056640,"degraded_objects":25387,"degraded_total":66040,"degraded_ratio":0.38441853422168376,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:00.854 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:00.855 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:01.395 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:01.420 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-18T11:55:00.077717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135202188,"num_objects":16510,"num_object_clones":0,"num_object_copies":66040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190577,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8069,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3624868,"kb_used_data":268656,"kb_used_omap":14,"kb_used_meta":3356145,"kb_avail":373862492,"statfs":{"total":386547056640,"available":382835191808,"internally_reserved":0,"allocated":275103744,"data_stored":274368460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3436692507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,3,9,11,13,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21798912,"num_objects":-2661,"num_object_clones":0,"num_object_copies":-10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2661,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004574"},"pg_stats":[{"pgid":"1.f","version":"37'12237","reported_seq":12337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.342323+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:59.342323+0000","last_peered":"2024-09-18T11:54:59.342323+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:59.342323+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8298496,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11987","reported_seq":16279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:56.409123+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:54:56.409123+0000","last_peered":"2024-09-18T11:54:56.409123+0000","last_clean":"2024-09-18T11:54:56.409123+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:54:56.409123+0000","last_undegraded":"2024-09-18T11:54:56.409123+0000","last_fullsized":"2024-09-18T11:54:56.409123+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8754422,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11812","reported_seq":11929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.929100+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:58.929100+0000","last_peered":"2024-09-18T11:54:58.929100+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:58.929100+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8157843,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11811,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11835","reported_seq":11929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.654198+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:54:54.654198+0000","last_peered":"2024-09-18T11:54:54.654198+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:54:54.654198+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8934646,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11833,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11894","reported_seq":12015,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.482455+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:59.482455+0000","last_peered":"2024-09-18T11:54:59.482455+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:59.482455+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8197366,"num_objects":1001,"num_object_clones":0,"num_object_copies":4004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11892,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11845","reported_seq":11964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.462130+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:59.462130+0000","last_peered":"2024-09-18T11:54:59.462130+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:59.462130+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7708672,"num_objects":941,"num_object_clones":0,"num_object_copies":3764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11845,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11963","reported_seq":12070,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.957187+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:54.957187+0000","last_peered":"2024-09-18T11:54:54.957187+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:54.957187+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8623350,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11961,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1063}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11773","reported_seq":14742,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.979755+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:54:54.979755+0000","last_peered":"2024-09-18T11:54:54.979755+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:54:54.979755+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9024758,"num_objects":1102,"num_object_clones":0,"num_object_copies":4408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":586,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1452,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12134","reported_seq":16487,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:59.382120+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:59.382120+0000","last_peered":"2024-09-18T11:54:59.382120+0000","last_clean":"2024-09-18T11:54:59.382120+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:59.382120+0000","last_undegraded":"2024-09-18T11:54:59.382120+0000","last_fullsized":"2024-09-18T11:54:59.382120+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8530412,"num_objects":1042,"num_object_clones":0,"num_object_copies":4168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11597","reported_seq":11710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.785500+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:58.785500+0000","last_peered":"2024-09-18T11:54:58.785500+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:58.785500+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8164598,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-966}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11964","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.316033+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:59.316033+0000","last_peered":"2024-09-18T11:54:59.316033+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:59.316033+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8753009,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11961,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12056","reported_seq":12149,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.365749+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:59.365749+0000","last_peered":"2024-09-18T11:54:59.365749+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:59.365749+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7985787,"num_objects":975,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12055,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11979","reported_seq":12077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.976141+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:58.976141+0000","last_peered":"2024-09-18T11:54:58.976141+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:58.976141+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8092283,"num_objects":988,"num_object_clones":0,"num_object_copies":3952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1058}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12022","reported_seq":12147,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.324289+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:59.324289+0000","last_peered":"2024-09-18T11:54:59.324289+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:59.324289+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746230,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12020,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1055}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11890","reported_seq":16179,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:56.403776+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:54:56.403776+0000","last_peered":"2024-09-18T11:54:56.403776+0000","last_clean":"2024-09-18T11:54:56.403776+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:54:56.403776+0000","last_undegraded":"2024-09-18T11:54:56.403776+0000","last_fullsized":"2024-09-18T11:54:56.403776+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8295670,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11888,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11616","reported_seq":11706,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:54.731713+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:54.731713+0000","last_peered":"2024-09-18T11:54:54.731713+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:54.731713+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8934646,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135202188,"num_objects":16510,"num_object_clones":0,"num_object_copies":66040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190577,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8069,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":272027648,"data_stored":272027648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992964,"kb_used_data":65276,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93378876,"statfs":{"total":96636764160,"available":95619969024,"internally_reserved":0,"allocated":66842624,"data_stored":66658803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.609},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.53200000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.6489999999999991},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":0.95599999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":5.9809999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":984872,"kb_used_data":66016,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93386968,"statfs":{"total":96636764160,"available":95628255232,"internally_reserved":0,"allocated":67600384,"data_stored":67416563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,5,3,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.1429999999999998},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.173}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.117999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.85399999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.0980000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.089}]}]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662584,"kb_used_data":68072,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93709256,"statfs":{"total":96636764160,"available":95958278144,"internally_reserved":0,"allocated":69705728,"data_stored":69521907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":2.0089999999999999},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":5.1509999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":15.715},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.63600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":5.1150000000000002},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":15.675000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":984448,"kb_used_data":69292,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93387392,"statfs":{"total":96636764160,"available":95628689408,"internally_reserved":0,"allocated":70955008,"data_stored":70771187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,3,2,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":9.5890000000000004},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":5.8129999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.6980000000000004},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.843}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":9.5540000000000003},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.8650000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69971968,"data_stored":69971968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69181440,"data_stored":69181440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66633728,"data_stored":66633728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":66240512,"data_stored":66240512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:04.423 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:04.950 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:04.975 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-18T11:55:04.078961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":132105612,"num_objects":16132,"num_object_clones":0,"num_object_copies":64528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190955,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8122,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3625128,"kb_used_data":260612,"kb_used_omap":14,"kb_used_meta":3364465,"kb_avail":373862232,"statfs":{"total":386547056640,"available":382834925568,"internally_reserved":0,"allocated":266866688,"data_stored":266131404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3445212187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,9,3,10,10,12,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16547840,"num_objects":-2020,"num_object_clones":0,"num_object_copies":-8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2020,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004478"},"pg_stats":[{"pgid":"1.f","version":"37'12237","reported_seq":12337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.342323+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:59.342323+0000","last_peered":"2024-09-18T11:54:59.342323+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:59.342323+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8298496,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12049","reported_seq":16341,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.518398+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:01.518398+0000","last_peered":"2024-09-18T11:55:01.518398+0000","last_clean":"2024-09-18T11:55:01.518398+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:01.518398+0000","last_undegraded":"2024-09-18T11:55:01.518398+0000","last_fullsized":"2024-09-18T11:55:01.518398+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8246518,"num_objects":1007,"num_object_clones":0,"num_object_copies":4028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12047,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11812","reported_seq":11929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.929100+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:58.929100+0000","last_peered":"2024-09-18T11:54:58.929100+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:58.929100+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8157843,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11811,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11886","reported_seq":11980,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:00.042763+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:00.042763+0000","last_peered":"2024-09-18T11:55:00.042763+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:00.042763+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8516854,"num_objects":1040,"num_object_clones":0,"num_object_copies":4160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11884,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11894","reported_seq":12015,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.482455+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:59.482455+0000","last_peered":"2024-09-18T11:54:59.482455+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:59.482455+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8197366,"num_objects":1001,"num_object_clones":0,"num_object_copies":4004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11892,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11845","reported_seq":11964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.462130+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:59.462130+0000","last_peered":"2024-09-18T11:54:59.462130+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:59.462130+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7708672,"num_objects":941,"num_object_clones":0,"num_object_copies":3764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11845,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12035","reported_seq":12142,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.577546+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:59.577546+0000","last_peered":"2024-09-18T11:54:59.577546+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:59.577546+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8033526,"num_objects":981,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1135}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11839","reported_seq":14914,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:00.095736+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:00.095736+0000","last_peered":"2024-09-18T11:55:00.095736+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:00.095736+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8484086,"num_objects":1036,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12134","reported_seq":16487,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:59.382120+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:59.382120+0000","last_peered":"2024-09-18T11:54:59.382120+0000","last_clean":"2024-09-18T11:54:59.382120+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:59.382120+0000","last_undegraded":"2024-09-18T11:54:59.382120+0000","last_fullsized":"2024-09-18T11:54:59.382120+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8530412,"num_objects":1042,"num_object_clones":0,"num_object_copies":4168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11597","reported_seq":11710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.785500+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:58.785500+0000","last_peered":"2024-09-18T11:54:58.785500+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:58.785500+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8164598,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-966}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11964","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.316033+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:59.316033+0000","last_peered":"2024-09-18T11:54:59.316033+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:59.316033+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8753009,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11961,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12056","reported_seq":12149,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.365749+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:59.365749+0000","last_peered":"2024-09-18T11:54:59.365749+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:59.365749+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7985787,"num_objects":975,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12055,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11979","reported_seq":12077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.976141+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:58.976141+0000","last_peered":"2024-09-18T11:54:58.976141+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:58.976141+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8092283,"num_objects":988,"num_object_clones":0,"num_object_copies":3952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1058}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12022","reported_seq":12147,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.324289+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:59.324289+0000","last_peered":"2024-09-18T11:54:59.324289+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:59.324289+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746230,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12020,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1055}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11949","reported_seq":16238,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.447711+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:01.447711+0000","last_peered":"2024-09-18T11:55:01.447711+0000","last_clean":"2024-09-18T11:55:01.447711+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:01.447711+0000","last_undegraded":"2024-09-18T11:55:01.447711+0000","last_fullsized":"2024-09-18T11:55:01.447711+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7812342,"num_objects":954,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11684","reported_seq":11774,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.883331+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:59.883331+0000","last_peered":"2024-09-18T11:54:59.883331+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:59.883331+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8377590,"num_objects":1023,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11682,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-979}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":132105612,"num_objects":16132,"num_object_clones":0,"num_object_copies":64528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190955,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8122,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263659520,"data_stored":263659520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992964,"kb_used_data":65276,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93378876,"statfs":{"total":96636764160,"available":95619969024,"internally_reserved":0,"allocated":66842624,"data_stored":66658803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.609},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.53200000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.6489999999999991},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":0.95599999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":5.9809999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":984872,"kb_used_data":66016,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93386968,"statfs":{"total":96636764160,"available":95628255232,"internally_reserved":0,"allocated":67600384,"data_stored":67416563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,5,3,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.1429999999999998},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.173}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.117999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.85399999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.0980000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.089}]}]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662672,"kb_used_data":64008,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93709168,"statfs":{"total":96636764160,"available":95958188032,"internally_reserved":0,"allocated":65544192,"data_stored":65360371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":17.602},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":4.3259999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":5.9660000000000002},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":17.643000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":14.768000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.023}]}]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":984620,"kb_used_data":65312,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93387220,"statfs":{"total":96636764160,"available":95628513280,"internally_reserved":0,"allocated":66879488,"data_stored":66695667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":5.4960000000000004},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":17.097999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":13.487},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":3.645}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":5.5250000000000004},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":3.6739999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65802240,"data_stored":65802240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":64983040,"data_stored":64983040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66633728,"data_stored":66633728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":66240512,"data_stored":66240512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:04.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:05.544 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:05.545 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:05.566 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-18T11:55:04.078961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":132105612,"num_objects":16132,"num_object_clones":0,"num_object_copies":64528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190955,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8122,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3625128,"kb_used_data":260612,"kb_used_omap":14,"kb_used_meta":3364465,"kb_avail":373862232,"statfs":{"total":386547056640,"available":382834925568,"internally_reserved":0,"allocated":266866688,"data_stored":266131404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3445212187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,9,3,10,10,12,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16547840,"num_objects":-2020,"num_object_clones":0,"num_object_copies":-8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2020,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004478"},"pg_stats":[{"pgid":"1.f","version":"37'12237","reported_seq":12337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.342323+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:54:59.342323+0000","last_peered":"2024-09-18T11:54:59.342323+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:54:59.342323+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8298496,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12049","reported_seq":16341,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.518398+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:01.518398+0000","last_peered":"2024-09-18T11:55:01.518398+0000","last_clean":"2024-09-18T11:55:01.518398+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:01.518398+0000","last_undegraded":"2024-09-18T11:55:01.518398+0000","last_fullsized":"2024-09-18T11:55:01.518398+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8246518,"num_objects":1007,"num_object_clones":0,"num_object_copies":4028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12047,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11812","reported_seq":11929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.929100+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:54:58.929100+0000","last_peered":"2024-09-18T11:54:58.929100+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:54:58.929100+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8157843,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11811,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11886","reported_seq":11980,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:00.042763+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:00.042763+0000","last_peered":"2024-09-18T11:55:00.042763+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:00.042763+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8516854,"num_objects":1040,"num_object_clones":0,"num_object_copies":4160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11884,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11894","reported_seq":12015,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.482455+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:54:59.482455+0000","last_peered":"2024-09-18T11:54:59.482455+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:54:59.482455+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8197366,"num_objects":1001,"num_object_clones":0,"num_object_copies":4004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11892,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11845","reported_seq":11964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.462130+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:54:59.462130+0000","last_peered":"2024-09-18T11:54:59.462130+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:54:59.462130+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7708672,"num_objects":941,"num_object_clones":0,"num_object_copies":3764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11845,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12035","reported_seq":12142,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.577546+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:54:59.577546+0000","last_peered":"2024-09-18T11:54:59.577546+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:54:59.577546+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8033526,"num_objects":981,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1135}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11839","reported_seq":14914,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:00.095736+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:00.095736+0000","last_peered":"2024-09-18T11:55:00.095736+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:00.095736+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8484086,"num_objects":1036,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12134","reported_seq":16487,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:54:59.382120+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:54:59.382120+0000","last_peered":"2024-09-18T11:54:59.382120+0000","last_clean":"2024-09-18T11:54:59.382120+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:54:59.382120+0000","last_undegraded":"2024-09-18T11:54:59.382120+0000","last_fullsized":"2024-09-18T11:54:59.382120+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8530412,"num_objects":1042,"num_object_clones":0,"num_object_copies":4168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11597","reported_seq":11710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.785500+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:54:58.785500+0000","last_peered":"2024-09-18T11:54:58.785500+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:54:58.785500+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8164598,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-966}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11964","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.316033+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:54:59.316033+0000","last_peered":"2024-09-18T11:54:59.316033+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:54:59.316033+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8753009,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11961,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12056","reported_seq":12149,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.365749+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:54:59.365749+0000","last_peered":"2024-09-18T11:54:59.365749+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:54:59.365749+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7985787,"num_objects":975,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12055,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11979","reported_seq":12077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:58.976141+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:54:58.976141+0000","last_peered":"2024-09-18T11:54:58.976141+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:54:58.976141+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8092283,"num_objects":988,"num_object_clones":0,"num_object_copies":3952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1058}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12022","reported_seq":12147,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.324289+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:54:59.324289+0000","last_peered":"2024-09-18T11:54:59.324289+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:54:59.324289+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746230,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12020,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1055}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11949","reported_seq":16238,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.447711+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:01.447711+0000","last_peered":"2024-09-18T11:55:01.447711+0000","last_clean":"2024-09-18T11:55:01.447711+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:01.447711+0000","last_undegraded":"2024-09-18T11:55:01.447711+0000","last_fullsized":"2024-09-18T11:55:01.447711+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7812342,"num_objects":954,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11684","reported_seq":11774,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:54:59.883331+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:54:59.883331+0000","last_peered":"2024-09-18T11:54:59.883331+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:54:59.883331+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8377590,"num_objects":1023,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11682,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-979}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":132105612,"num_objects":16132,"num_object_clones":0,"num_object_copies":64528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190955,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8122,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263659520,"data_stored":263659520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":992964,"kb_used_data":65276,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93378876,"statfs":{"total":96636764160,"available":95619969024,"internally_reserved":0,"allocated":66842624,"data_stored":66658803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.609},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.53200000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:53:58 2024","interfaces":[{"interface":"back","average":{"1min":8.4030000000000005,"5min":7.8869999999999996,"15min":6.2889999999999997},"min":{"1min":0.432,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":33.281999999999996,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.6489999999999991},{"interface":"front","average":{"1min":6.9119999999999999,"5min":6.8940000000000001,"15min":5.1669999999999998},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":25.013999999999999,"5min":57.372,"15min":57.372},"last":0.95599999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":5.9809999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":984872,"kb_used_data":66016,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93386968,"statfs":{"total":96636764160,"available":95628255232,"internally_reserved":0,"allocated":67600384,"data_stored":67416563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,5,3,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":5.1429999999999998},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":5.173}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":21.117999999999999},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.85399999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.0980000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.089}]}]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662672,"kb_used_data":64008,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93709168,"statfs":{"total":96636764160,"available":95958188032,"internally_reserved":0,"allocated":65544192,"data_stored":65360371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":17.602},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":4.3259999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":5.9660000000000002},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":17.643000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":14.768000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.023}]}]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":984620,"kb_used_data":65312,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93387220,"statfs":{"total":96636764160,"available":95628513280,"internally_reserved":0,"allocated":66879488,"data_stored":66695667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":5.4960000000000004},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":17.097999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":13.487},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":3.645}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":5.5250000000000004},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":3.6739999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65802240,"data_stored":65802240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":64983040,"data_stored":64983040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66633728,"data_stored":66633728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":66240512,"data_stored":66240512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:05.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:05.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12237", 'reported_seq': 12337, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.342323+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:54:59.342323+0000', 'last_peered': '2024-09-18T11:54:59.342323+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:54:59.342323+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8298496, 'num_objects': 1013, 'num_object_clones': 0, 'num_object_copies': 4052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12237, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1090}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:05.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:05.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11812", 'reported_seq': 11929, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:58.929100+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:54:58.929100+0000', 'last_peered': '2024-09-18T11:54:58.929100+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:54:58.929100+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8157843, 'num_objects': 997, 'num_object_clones': 0, 'num_object_copies': 3988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 997, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11811, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1043}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:05.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:05.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11886", 'reported_seq': 11980, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:00.042763+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:00.042763+0000', 'last_peered': '2024-09-18T11:55:00.042763+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:00.042763+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8516854, 'num_objects': 1040, 'num_object_clones': 0, 'num_object_copies': 4160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1040, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11884, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1061}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:05.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:05.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11894", 'reported_seq': 12015, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.482455+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:54:59.482455+0000', 'last_peered': '2024-09-18T11:54:59.482455+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:54:59.482455+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8197366, 'num_objects': 1001, 'num_object_clones': 0, 'num_object_copies': 4004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11892, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1051}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:05.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:05.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11845", 'reported_seq': 11964, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.462130+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:54:59.462130+0000', 'last_peered': '2024-09-18T11:54:59.462130+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:54:59.462130+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7708672, 'num_objects': 941, 'num_object_clones': 0, 'num_object_copies': 3764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11845, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1178}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:05.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:05.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12035", 'reported_seq': 12142, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.577546+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:54:59.577546+0000', 'last_peered': '2024-09-18T11:54:59.577546+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:54:59.577546+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8033526, 'num_objects': 981, 'num_object_clones': 0, 'num_object_copies': 3924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12033, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1135}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:05.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:05.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11839", 'reported_seq': 14914, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:00.095736+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:55:00.095736+0000', 'last_peered': '2024-09-18T11:55:00.095736+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:55:00.095736+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8484086, 'num_objects': 1036, 'num_object_clones': 0, 'num_object_copies': 4144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 533, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11837, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1505, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:05.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:05.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11597", 'reported_seq': 11710, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:58.785500+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:54:58.785500+0000', 'last_peered': '2024-09-18T11:54:58.785500+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:54:58.785500+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8164598, 'num_objects': 997, 'num_object_clones': 0, 'num_object_copies': 3988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -966}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:05.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:05.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11964", 'reported_seq': 12085, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.316033+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:54:59.316033+0000', 'last_peered': '2024-09-18T11:54:59.316033+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:54:59.316033+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8753009, 'num_objects': 1069, 'num_object_clones': 0, 'num_object_copies': 4276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11961, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -965}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:05.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:05.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12056", 'reported_seq': 12149, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.365749+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:54:59.365749+0000', 'last_peered': '2024-09-18T11:54:59.365749+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:54:59.365749+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7985787, 'num_objects': 975, 'num_object_clones': 0, 'num_object_copies': 3900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12055, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:05.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:05.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11979", 'reported_seq': 12077, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:58.976141+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:54:58.976141+0000', 'last_peered': '2024-09-18T11:54:58.976141+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:54:58.976141+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8092283, 'num_objects': 988, 'num_object_clones': 0, 'num_object_copies': 3952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11978, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1058}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:05.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:05.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12022", 'reported_seq': 12147, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.324289+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:54:59.324289+0000', 'last_peered': '2024-09-18T11:54:59.324289+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:54:59.324289+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8746230, 'num_objects': 1068, 'num_object_clones': 0, 'num_object_copies': 4272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12020, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1055}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:05.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:05.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11684", 'reported_seq': 11774, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:54:59.883331+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:54:59.883331+0000', 'last_peered': '2024-09-18T11:54:59.883331+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:54:59.883331+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8377590, 'num_objects': 1023, 'num_object_clones': 0, 'num_object_copies': 4092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1023, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11682, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -979}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:05.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:06.302 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:06.303 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25281/60904 objects degraded (41.510%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":704,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":15226,"data_bytes":124683660,"bytes_used":3694743552,"bytes_avail":382852313088,"bytes_total":386547056640,"degraded_objects":25281,"degraded_total":60904,"degraded_ratio":0.41509588861158547,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":227},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:06.329 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:06.329 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:06.892 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:06.892 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:06.918 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-18T11:55:06.079779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":124683660,"num_objects":15226,"num_object_clones":0,"num_object_copies":60904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15226,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191861,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8175,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160588,"ondisk_log_size":160588,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3608148,"kb_used_data":247764,"kb_used_omap":14,"kb_used_meta":3360305,"kb_avail":373879212,"statfs":{"total":386547056640,"available":382852313088,"internally_reserved":0,"allocated":253710336,"data_stored":252975052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3440952347},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,5,15,9,4,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22347776,"num_objects":-2728,"num_object_clones":0,"num_object_copies":-10912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2728,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004469"},"pg_stats":[{"pgid":"1.f","version":"37'12305","reported_seq":12406,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.421074+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:04.421074+0000","last_peered":"2024-09-18T11:55:04.421074+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:04.421074+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7741440,"num_objects":945,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12305,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12049","reported_seq":16341,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.518398+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:01.518398+0000","last_peered":"2024-09-18T11:55:01.518398+0000","last_clean":"2024-09-18T11:55:01.518398+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:01.518398+0000","last_undegraded":"2024-09-18T11:55:01.518398+0000","last_fullsized":"2024-09-18T11:55:01.518398+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8246518,"num_objects":1007,"num_object_clones":0,"num_object_copies":4028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12047,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11886","reported_seq":12003,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.059955+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:04.059955+0000","last_peered":"2024-09-18T11:55:04.059955+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:04.059955+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7551635,"num_objects":923,"num_object_clones":0,"num_object_copies":3692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11885,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1117}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11946","reported_seq":12040,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.855318+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:04.855318+0000","last_peered":"2024-09-18T11:55:04.855318+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:04.855318+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8025334,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11944,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11945","reported_seq":12065,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.436363+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:04.436363+0000","last_peered":"2024-09-18T11:55:04.436363+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:04.436363+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7779574,"num_objects":950,"num_object_clones":0,"num_object_copies":3800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11943,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11905","reported_seq":12024,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.425138+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:04.425138+0000","last_peered":"2024-09-18T11:55:04.425138+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:04.425138+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7217152,"num_objects":881,"num_object_clones":0,"num_object_copies":3524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11905,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12103","reported_seq":12208,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:05.085954+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:05.085954+0000","last_peered":"2024-09-18T11:55:05.085954+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:05.085954+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7476470,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12101,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1203}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11903","reported_seq":15083,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:05.130141+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:05.130141+0000","last_peered":"2024-09-18T11:55:05.130141+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:05.130141+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7959798,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11901,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1558,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12210","reported_seq":16563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.539178+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:04.539178+0000","last_peered":"2024-09-18T11:55:04.539178+0000","last_clean":"2024-09-18T11:55:04.539178+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:04.539178+0000","last_undegraded":"2024-09-18T11:55:04.539178+0000","last_fullsized":"2024-09-18T11:55:04.539178+0000","mapping_epoch":36,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7907820,"num_objects":966,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11658","reported_seq":11772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:03.990571+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:03.990571+0000","last_peered":"2024-09-18T11:55:03.990571+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:03.990571+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7664886,"num_objects":936,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11656,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1027}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12034","reported_seq":12152,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.404253+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:04.404253+0000","last_peered":"2024-09-18T11:55:04.404253+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:04.404253+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8179569,"num_objects":999,"num_object_clones":0,"num_object_copies":3996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1035}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12122","reported_seq":12215,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.236484+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:04.236484+0000","last_peered":"2024-09-18T11:55:04.236484+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:04.236484+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7445115,"num_objects":909,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12121,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1175}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12046","reported_seq":12144,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.000334+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:04.000334+0000","last_peered":"2024-09-18T11:55:04.000334+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:04.000334+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7543419,"num_objects":921,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12045,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1125}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12087","reported_seq":12212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.541623+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:04.541623+0000","last_peered":"2024-09-18T11:55:04.541623+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:04.541623+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8213750,"num_objects":1003,"num_object_clones":0,"num_object_copies":4012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12085,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1120}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11949","reported_seq":16238,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.447711+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:01.447711+0000","last_peered":"2024-09-18T11:55:01.447711+0000","last_clean":"2024-09-18T11:55:01.447711+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:01.447711+0000","last_undegraded":"2024-09-18T11:55:01.447711+0000","last_fullsized":"2024-09-18T11:55:01.447711+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7812342,"num_objects":954,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11740","reported_seq":11830,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:05.061778+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:05.061778+0000","last_peered":"2024-09-18T11:55:05.061778+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:05.061778+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7918838,"num_objects":967,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11738,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":124683660,"num_objects":15226,"num_object_clones":0,"num_object_copies":60904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15226,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191861,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8175,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":251043840,"data_stored":251043840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160588,"ondisk_log_size":160588,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":976468,"kb_used_data":61240,"kb_used_omap":4,"kb_used_meta":915195,"kb_avail":93395372,"statfs":{"total":96636764160,"available":95636860928,"internally_reserved":0,"allocated":62709760,"data_stored":62525939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937160257},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,1,1,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":5.6879999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":11.257}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":4.9699999999999998},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.95899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":985080,"kb_used_data":62052,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93386760,"statfs":{"total":96636764160,"available":95628042240,"internally_reserved":0,"allocated":63541248,"data_stored":63357427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":1.712},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":1.7470000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":9.3780000000000001,"5min":8.1240000000000006,"15min":6.633},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":46.793999999999997,"5min":46.793999999999997,"15min":46.793999999999997},"last":16.556000000000001},{"interface":"front","average":{"1min":7.4619999999999997,"5min":5.4340000000000002,"15min":4.3789999999999996},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":15.898999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.9760000000000009}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":19.992999999999999},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":2.9649999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662672,"kb_used_data":64008,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93709168,"statfs":{"total":96636764160,"available":95958188032,"internally_reserved":0,"allocated":65544192,"data_stored":65360371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":17.602},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":4.3259999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":5.9660000000000002},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":17.643000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":14.768000000000001},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.023}]}]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":983928,"kb_used_data":60464,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93387912,"statfs":{"total":96636764160,"available":95629221888,"internally_reserved":0,"allocated":61915136,"data_stored":61731315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":5.319},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.3610000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":2.399},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":4.609}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":5.2830000000000004},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61579264,"data_stored":61579264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":64983040,"data_stored":64983040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62435328,"data_stored":62435328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62046208,"data_stored":62046208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:09.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:10.429 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:10.429 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:10.459 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-18T11:55:10.081444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118506892,"num_objects":14472,"num_object_clones":0,"num_object_copies":57888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192615,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8175,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160642,"ondisk_log_size":160642,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3927016,"kb_used_data":235644,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373560344,"statfs":{"total":386547056640,"available":382525792256,"internally_reserved":0,"allocated":241299456,"data_stored":240564172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,8,12,10,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21864448,"num_objects":-2669,"num_object_clones":0,"num_object_copies":-10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004538"},"pg_stats":[{"pgid":"1.f","version":"37'12357","reported_seq":12458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.184051+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:09.184051+0000","last_peered":"2024-09-18T11:55:09.184051+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:09.184051+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7315456,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12116","reported_seq":16408,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:06.777476+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:06.777476+0000","last_peered":"2024-09-18T11:55:06.777476+0000","last_clean":"2024-09-18T11:55:06.777476+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:06.777476+0000","last_undegraded":"2024-09-18T11:55:06.777476+0000","last_fullsized":"2024-09-18T11:55:06.777476+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7697654,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11937","reported_seq":12054,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:08.944591+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:08.944591+0000","last_peered":"2024-09-18T11:55:08.944591+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:08.944591+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7133843,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11936,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11946","reported_seq":12040,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.855318+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:04.855318+0000","last_peered":"2024-09-18T11:55:04.855318+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:04.855318+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8025334,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11944,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12019","reported_seq":12138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.486107+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:09.486107+0000","last_peered":"2024-09-18T11:55:09.486107+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:09.486107+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7173366,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11969","reported_seq":12086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.385953+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:09.385953+0000","last_peered":"2024-09-18T11:55:09.385953+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:09.385953+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11969,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12103","reported_seq":12208,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:05.085954+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:05.085954+0000","last_peered":"2024-09-18T11:55:05.085954+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:05.085954+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7476470,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12101,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1203}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11903","reported_seq":15083,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:05.130141+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:05.130141+0000","last_peered":"2024-09-18T11:55:05.130141+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:05.130141+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7959798,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11901,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1558,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12283","reported_seq":16636,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:09.414295+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:09.414295+0000","last_peered":"2024-09-18T11:55:09.414295+0000","last_clean":"2024-09-18T11:55:09.414295+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:09.414295+0000","last_undegraded":"2024-09-18T11:55:09.414295+0000","last_fullsized":"2024-09-18T11:55:09.414295+0000","mapping_epoch":36,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7309804,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12280,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11730","reported_seq":11843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.066793+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:09.066793+0000","last_peered":"2024-09-18T11:55:09.066793+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:09.066793+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075062,"num_objects":864,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11728,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1099}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12105","reported_seq":12225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.515610+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:09.515610+0000","last_peered":"2024-09-18T11:55:09.515610+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:09.515610+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7597937,"num_objects":928,"num_object_clones":0,"num_object_copies":3712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12102,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12185","reported_seq":12278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.512026+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:09.512026+0000","last_peered":"2024-09-18T11:55:09.512026+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:09.512026+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6929019,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1238}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12099","reported_seq":12197,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.084436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:09.084436+0000","last_peered":"2024-09-18T11:55:09.084436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:09.084436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7109243,"num_objects":868,"num_object_clones":0,"num_object_copies":3472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1178}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12129","reported_seq":12254,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.484972+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:09.484972+0000","last_peered":"2024-09-18T11:55:09.484972+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:09.484972+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7869686,"num_objects":961,"num_object_clones":0,"num_object_copies":3844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12021","reported_seq":16310,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:06.892206+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:06.892206+0000","last_peered":"2024-09-18T11:55:06.892206+0000","last_clean":"2024-09-18T11:55:06.892206+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:06.892206+0000","last_undegraded":"2024-09-18T11:55:06.892206+0000","last_fullsized":"2024-09-18T11:55:06.892206+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7222518,"num_objects":882,"num_object_clones":0,"num_object_copies":3528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12019,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11740","reported_seq":11830,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:05.061778+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:05.061778+0000","last_peered":"2024-09-18T11:55:05.061778+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:05.061778+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7918838,"num_objects":967,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11738,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":118506892,"num_objects":14472,"num_object_clones":0,"num_object_copies":57888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192615,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8175,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":238333952,"data_stored":238333952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160642,"ondisk_log_size":160642,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":976676,"kb_used_data":57304,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93395164,"statfs":{"total":96636764160,"available":95636647936,"internally_reserved":0,"allocated":58679296,"data_stored":58495475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.4280000000000008},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":8.4600000000000009}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.4909999999999997},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":6.2060000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.2669999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.2229999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":985148,"kb_used_data":57972,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93386692,"statfs":{"total":96636764160,"available":95627972608,"internally_reserved":0,"allocated":59363328,"data_stored":59179507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":4.1870000000000003},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":4.2199999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.35499999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":0.32400000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":0.58699999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":981264,"kb_used_data":59904,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93390576,"statfs":{"total":96636764160,"available":95631949824,"internally_reserved":0,"allocated":61341696,"data_stored":61157875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":4.8380000000000001},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":11.118}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":12.271000000000001},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.9409999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":43.334000000000003},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":2.0950000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":983928,"kb_used_data":60464,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93387912,"statfs":{"total":96636764160,"available":95629221888,"internally_reserved":0,"allocated":61915136,"data_stored":61731315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":5.319},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.3610000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":2.399},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":4.609}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":5.2830000000000004},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61579264,"data_stored":61579264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60571648,"data_stored":60571648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58298368,"data_stored":58298368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57884672,"data_stored":57884672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:10.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:11.007 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:11.007 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:11.026 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-18T11:55:10.081444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118506892,"num_objects":14472,"num_object_clones":0,"num_object_copies":57888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192615,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8175,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160642,"ondisk_log_size":160642,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3927016,"kb_used_data":235644,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373560344,"statfs":{"total":386547056640,"available":382525792256,"internally_reserved":0,"allocated":241299456,"data_stored":240564172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,8,12,10,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21864448,"num_objects":-2669,"num_object_clones":0,"num_object_copies":-10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004538"},"pg_stats":[{"pgid":"1.f","version":"37'12357","reported_seq":12458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.184051+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:09.184051+0000","last_peered":"2024-09-18T11:55:09.184051+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:09.184051+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7315456,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12116","reported_seq":16408,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:06.777476+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:06.777476+0000","last_peered":"2024-09-18T11:55:06.777476+0000","last_clean":"2024-09-18T11:55:06.777476+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:06.777476+0000","last_undegraded":"2024-09-18T11:55:06.777476+0000","last_fullsized":"2024-09-18T11:55:06.777476+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7697654,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11937","reported_seq":12054,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:08.944591+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:08.944591+0000","last_peered":"2024-09-18T11:55:08.944591+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:08.944591+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7133843,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11936,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11946","reported_seq":12040,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:04.855318+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:04.855318+0000","last_peered":"2024-09-18T11:55:04.855318+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:04.855318+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8025334,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11944,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12019","reported_seq":12138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.486107+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:09.486107+0000","last_peered":"2024-09-18T11:55:09.486107+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:09.486107+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7173366,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11969","reported_seq":12086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.385953+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:09.385953+0000","last_peered":"2024-09-18T11:55:09.385953+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:09.385953+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11969,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12103","reported_seq":12208,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:05.085954+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:05.085954+0000","last_peered":"2024-09-18T11:55:05.085954+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:05.085954+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7476470,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12101,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1203}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11903","reported_seq":15083,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:05.130141+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:05.130141+0000","last_peered":"2024-09-18T11:55:05.130141+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:05.130141+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7959798,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11901,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1558,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12283","reported_seq":16636,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:09.414295+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:09.414295+0000","last_peered":"2024-09-18T11:55:09.414295+0000","last_clean":"2024-09-18T11:55:09.414295+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:09.414295+0000","last_undegraded":"2024-09-18T11:55:09.414295+0000","last_fullsized":"2024-09-18T11:55:09.414295+0000","mapping_epoch":36,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7309804,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12280,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11730","reported_seq":11843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.066793+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:09.066793+0000","last_peered":"2024-09-18T11:55:09.066793+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:09.066793+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075062,"num_objects":864,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11728,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1099}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12105","reported_seq":12225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.515610+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:09.515610+0000","last_peered":"2024-09-18T11:55:09.515610+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:09.515610+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7597937,"num_objects":928,"num_object_clones":0,"num_object_copies":3712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12102,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12185","reported_seq":12278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.512026+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:09.512026+0000","last_peered":"2024-09-18T11:55:09.512026+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:09.512026+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6929019,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1238}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12099","reported_seq":12197,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.084436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:09.084436+0000","last_peered":"2024-09-18T11:55:09.084436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:09.084436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7109243,"num_objects":868,"num_object_clones":0,"num_object_copies":3472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1178}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12129","reported_seq":12254,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.484972+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:09.484972+0000","last_peered":"2024-09-18T11:55:09.484972+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:09.484972+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7869686,"num_objects":961,"num_object_clones":0,"num_object_copies":3844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12021","reported_seq":16310,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:06.892206+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:06.892206+0000","last_peered":"2024-09-18T11:55:06.892206+0000","last_clean":"2024-09-18T11:55:06.892206+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:06.892206+0000","last_undegraded":"2024-09-18T11:55:06.892206+0000","last_fullsized":"2024-09-18T11:55:06.892206+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7222518,"num_objects":882,"num_object_clones":0,"num_object_copies":3528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12019,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11740","reported_seq":11830,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:05.061778+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:05.061778+0000","last_peered":"2024-09-18T11:55:05.061778+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:05.061778+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7918838,"num_objects":967,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11738,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":118506892,"num_objects":14472,"num_object_clones":0,"num_object_copies":57888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192615,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8175,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":238333952,"data_stored":238333952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160642,"ondisk_log_size":160642,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":976676,"kb_used_data":57304,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93395164,"statfs":{"total":96636764160,"available":95636647936,"internally_reserved":0,"allocated":58679296,"data_stored":58495475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.4280000000000008},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":8.4600000000000009}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.4909999999999997},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":6.2060000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.2669999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.2229999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":985148,"kb_used_data":57972,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93386692,"statfs":{"total":96636764160,"available":95627972608,"internally_reserved":0,"allocated":59363328,"data_stored":59179507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":4.1870000000000003},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":4.2199999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.35499999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":0.32400000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":0.58699999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":981264,"kb_used_data":59904,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93390576,"statfs":{"total":96636764160,"available":95631949824,"internally_reserved":0,"allocated":61341696,"data_stored":61157875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":6.9690000000000003,"5min":6.7640000000000002,"15min":4.6390000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.159},"max":{"1min":29.690000000000001,"5min":32.371000000000002,"15min":45.462000000000003},"last":4.8380000000000001},{"interface":"front","average":{"1min":7.5229999999999997,"5min":6.9960000000000004,"15min":5.6319999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":27.09,"5min":54.710999999999999,"15min":54.710999999999999},"last":11.118}]},{"osd":2,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":9.7149999999999999,"5min":7.6840000000000002,"15min":5.5119999999999996},"min":{"1min":0.56000000000000005,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":45.701999999999998,"5min":45.701999999999998,"15min":45.701999999999998},"last":12.271000000000001},{"interface":"front","average":{"1min":5.5789999999999997,"5min":5.5609999999999999,"15min":4.548},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":25.481000000000002,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.9409999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:05 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":6.1840000000000002,"15min":4.8929999999999998},"min":{"1min":0.27500000000000002,"5min":0.19,"15min":0.19},"max":{"1min":28.451000000000001,"5min":28.451000000000001,"15min":28.451000000000001},"last":43.334000000000003},{"interface":"front","average":{"1min":10.010999999999999,"5min":7.5730000000000004,"15min":5.8239999999999998},"min":{"1min":0.46300000000000002,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":45.786999999999999,"5min":45.786999999999999,"15min":45.786999999999999},"last":2.0950000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":983928,"kb_used_data":60464,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93387912,"statfs":{"total":96636764160,"available":95629221888,"internally_reserved":0,"allocated":61915136,"data_stored":61731315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:54:03 2024","interfaces":[{"interface":"back","average":{"1min":8.2449999999999992,"5min":7.7409999999999997,"15min":5.1139999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.960999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":5.319},{"interface":"front","average":{"1min":6.0460000000000003,"5min":6.016,"15min":4.6470000000000002},"min":{"1min":0.28100000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.798999999999999,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.3610000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":2.399},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":4.609}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":5.2830000000000004},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61579264,"data_stored":61579264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60571648,"data_stored":60571648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58298368,"data_stored":58298368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57884672,"data_stored":57884672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:11.027 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:11.027 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12357", 'reported_seq': 12458, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:09.184051+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:55:09.184051+0000', 'last_peered': '2024-09-18T11:55:09.184051+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:55:09.184051+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7315456, 'num_objects': 893, 'num_object_clones': 0, 'num_object_copies': 3572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 893, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12357, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1210}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:11.028 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:11.028 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11937", 'reported_seq': 12054, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:08.944591+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:55:08.944591+0000', 'last_peered': '2024-09-18T11:55:08.944591+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:55:08.944591+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7133843, 'num_objects': 872, 'num_object_clones': 0, 'num_object_copies': 3488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 872, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11936, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1168}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:11.028 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:11.028 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11946", 'reported_seq': 12040, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:04.855318+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:04.855318+0000', 'last_peered': '2024-09-18T11:55:04.855318+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:04.855318+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8025334, 'num_objects': 980, 'num_object_clones': 0, 'num_object_copies': 3920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11944, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:11.028 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:11.028 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12019", 'reported_seq': 12138, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:09.486107+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:55:09.486107+0000', 'last_peered': '2024-09-18T11:55:09.486107+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:55:09.486107+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7173366, 'num_objects': 876, 'num_object_clones': 0, 'num_object_copies': 3504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12017, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1176}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:11.028 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:11.028 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11969", 'reported_seq': 12086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:09.385953+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:55:09.385953+0000', 'last_peered': '2024-09-18T11:55:09.385953+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:55:09.385953+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6692864, 'num_objects': 817, 'num_object_clones': 0, 'num_object_copies': 3268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11969, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1302}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:11.028 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:11.028 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12103", 'reported_seq': 12208, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:05.085954+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:55:05.085954+0000', 'last_peered': '2024-09-18T11:55:05.085954+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:55:05.085954+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7476470, 'num_objects': 913, 'num_object_clones': 0, 'num_object_copies': 3652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12101, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1203}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:11.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:11.029 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11903", 'reported_seq': 15083, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:05.130141+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:55:05.130141+0000', 'last_peered': '2024-09-18T11:55:05.130141+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:55:05.130141+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7959798, 'num_objects': 972, 'num_object_clones': 0, 'num_object_copies': 3888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 480, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11901, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1558, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 492}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:11.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:11.029 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11730", 'reported_seq': 11843, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:09.066793+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:55:09.066793+0000', 'last_peered': '2024-09-18T11:55:09.066793+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:55:09.066793+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7075062, 'num_objects': 864, 'num_object_clones': 0, 'num_object_copies': 3456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11728, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1099}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:11.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:11.029 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12105", 'reported_seq': 12225, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:09.515610+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:55:09.515610+0000', 'last_peered': '2024-09-18T11:55:09.515610+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:55:09.515610+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7597937, 'num_objects': 928, 'num_object_clones': 0, 'num_object_copies': 3712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12102, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1106}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:11.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:11.029 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12185", 'reported_seq': 12278, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:09.512026+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:55:09.512026+0000', 'last_peered': '2024-09-18T11:55:09.512026+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:55:09.512026+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6929019, 'num_objects': 846, 'num_object_clones': 0, 'num_object_copies': 3384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12184, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:11.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:11.030 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12099", 'reported_seq': 12197, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:09.084436+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:55:09.084436+0000', 'last_peered': '2024-09-18T11:55:09.084436+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:55:09.084436+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7109243, 'num_objects': 868, 'num_object_clones': 0, 'num_object_copies': 3472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:11.030 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:11.030 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12129", 'reported_seq': 12254, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:09.484972+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:55:09.484972+0000', 'last_peered': '2024-09-18T11:55:09.484972+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:55:09.484972+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7869686, 'num_objects': 961, 'num_object_clones': 0, 'num_object_copies': 3844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12127, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:11.030 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:11.030 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11740", 'reported_seq': 11830, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:05.061778+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:55:05.061778+0000', 'last_peered': '2024-09-18T11:55:05.061778+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:55:05.061778+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7918838, 'num_objects': 967, 'num_object_clones': 0, 'num_object_copies': 3868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11738, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1035}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:11.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:11.759 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:11.759 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25281/57888 objects degraded (43.672%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":709,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14472,"data_bytes":118506892,"bytes_used":4021264384,"bytes_avail":382525792256,"bytes_total":386547056640,"degraded_objects":25281,"degraded_total":57888,"degraded_ratio":0.43672263681592033,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":222},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:11.795 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:11.795 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:12.373 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:12.391 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-09-18T11:55:12.082290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115279244,"num_objects":14078,"num_object_clones":0,"num_object_copies":56312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14078,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193009,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8226,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160836,"ondisk_log_size":160836,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3927176,"kb_used_data":227488,"kb_used_omap":14,"kb_used_meta":3699633,"kb_avail":373560184,"statfs":{"total":386547056640,"available":382525628416,"internally_reserved":0,"allocated":232947712,"data_stored":232212428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788424219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,4,16,11,15,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19922944,"num_objects":-2432,"num_object_clones":0,"num_object_copies":-9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2432,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004572"},"pg_stats":[{"pgid":"1.f","version":"37'12357","reported_seq":12458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.184051+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:09.184051+0000","last_peered":"2024-09-18T11:55:09.184051+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:09.184051+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7315456,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12180","reported_seq":16472,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:12.011280+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:12.011280+0000","last_peered":"2024-09-18T11:55:12.011280+0000","last_clean":"2024-09-18T11:55:12.011280+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:12.011280+0000","last_undegraded":"2024-09-18T11:55:12.011280+0000","last_fullsized":"2024-09-18T11:55:12.011280+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7173366,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12178,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11937","reported_seq":12054,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:08.944591+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:08.944591+0000","last_peered":"2024-09-18T11:55:08.944591+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:08.944591+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7133843,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11936,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12020","reported_seq":12114,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:10.123914+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:10.123914+0000","last_peered":"2024-09-18T11:55:10.123914+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:10.123914+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7419126,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12018,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1195}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12019","reported_seq":12138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.486107+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:09.486107+0000","last_peered":"2024-09-18T11:55:09.486107+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:09.486107+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7173366,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11969","reported_seq":12086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.385953+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:09.385953+0000","last_peered":"2024-09-18T11:55:09.385953+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:09.385953+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11969,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12168","reported_seq":12275,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.915469+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:09.915469+0000","last_peered":"2024-09-18T11:55:09.915469+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:09.915469+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6943990,"num_objects":848,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11978","reported_seq":15261,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:10.174602+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:10.174602+0000","last_peered":"2024-09-18T11:55:10.174602+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:10.174602+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7345398,"num_objects":897,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11976,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1609,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12283","reported_seq":16636,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:09.414295+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:09.414295+0000","last_peered":"2024-09-18T11:55:09.414295+0000","last_clean":"2024-09-18T11:55:09.414295+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:09.414295+0000","last_undegraded":"2024-09-18T11:55:09.414295+0000","last_fullsized":"2024-09-18T11:55:09.414295+0000","mapping_epoch":36,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7309804,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12280,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11730","reported_seq":11843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.066793+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:09.066793+0000","last_peered":"2024-09-18T11:55:09.066793+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:09.066793+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075062,"num_objects":864,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11728,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1099}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12105","reported_seq":12225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.515610+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:09.515610+0000","last_peered":"2024-09-18T11:55:09.515610+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:09.515610+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7597937,"num_objects":928,"num_object_clones":0,"num_object_copies":3712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12102,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12185","reported_seq":12278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.512026+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:09.512026+0000","last_peered":"2024-09-18T11:55:09.512026+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:09.512026+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6929019,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1238}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12099","reported_seq":12197,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.084436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:09.084436+0000","last_peered":"2024-09-18T11:55:09.084436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:09.084436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7109243,"num_objects":868,"num_object_clones":0,"num_object_copies":3472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1178}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12129","reported_seq":12254,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.484972+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:09.484972+0000","last_peered":"2024-09-18T11:55:09.484972+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:09.484972+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7869686,"num_objects":961,"num_object_clones":0,"num_object_copies":3844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12068","reported_seq":16357,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:11.971226+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:11.971226+0000","last_peered":"2024-09-18T11:55:11.971226+0000","last_clean":"2024-09-18T11:55:11.971226+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:11.971226+0000","last_undegraded":"2024-09-18T11:55:11.971226+0000","last_fullsized":"2024-09-18T11:55:11.971226+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6837494,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12066,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11809","reported_seq":11895,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:10.072198+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:10.072198+0000","last_peered":"2024-09-18T11:55:10.072198+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:10.072198+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7353590,"num_objects":898,"num_object_clones":0,"num_object_copies":3592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":115279244,"num_objects":14078,"num_object_clones":0,"num_object_copies":56312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14078,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193009,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8226,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":229822464,"data_stored":229822464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160836,"ondisk_log_size":160836,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":976676,"kb_used_data":57304,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93395164,"statfs":{"total":96636764160,"available":95636647936,"internally_reserved":0,"allocated":58679296,"data_stored":58495475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.4280000000000008},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":8.4600000000000009}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.4909999999999997},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":6.2060000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.2669999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.2229999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":985148,"kb_used_data":57972,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93386692,"statfs":{"total":96636764160,"available":95627972608,"internally_reserved":0,"allocated":59363328,"data_stored":59179507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":4.1870000000000003},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":4.2199999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.35499999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":0.32400000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":0.58699999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":981428,"kb_used_data":55912,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93390412,"statfs":{"total":96636764160,"available":95631781888,"internally_reserved":0,"allocated":57253888,"data_stored":57070067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":9.2739999999999991},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.006}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":1.3560000000000001},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.91100000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":15.880000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":1.387}]}]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":983924,"kb_used_data":56300,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93387916,"statfs":{"total":96636764160,"available":95629225984,"internally_reserved":0,"allocated":57651200,"data_stored":57467379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,4,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":7.0940000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.95699999999999996},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":7.141}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57327616,"data_stored":57327616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":56311808,"data_stored":56311808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58298368,"data_stored":58298368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57884672,"data_stored":57884672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:15.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:15.932 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:15.933 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:15.958 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-18T11:55:14.083011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115279244,"num_objects":14078,"num_object_clones":0,"num_object_copies":56312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14078,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193009,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8226,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160836,"ondisk_log_size":160836,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3927176,"kb_used_data":227488,"kb_used_omap":14,"kb_used_meta":3699633,"kb_avail":373560184,"statfs":{"total":386547056640,"available":382525628416,"internally_reserved":0,"allocated":232947712,"data_stored":232212428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788424219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,4,16,11,15,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16826368,"num_objects":-2054,"num_object_clones":0,"num_object_copies":-8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004694"},"pg_stats":[{"pgid":"1.f","version":"37'12357","reported_seq":12458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.184051+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:09.184051+0000","last_peered":"2024-09-18T11:55:09.184051+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:09.184051+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7315456,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12180","reported_seq":16472,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:12.011280+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:12.011280+0000","last_peered":"2024-09-18T11:55:12.011280+0000","last_clean":"2024-09-18T11:55:12.011280+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:12.011280+0000","last_undegraded":"2024-09-18T11:55:12.011280+0000","last_fullsized":"2024-09-18T11:55:12.011280+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7173366,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12178,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11937","reported_seq":12054,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:08.944591+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:08.944591+0000","last_peered":"2024-09-18T11:55:08.944591+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:08.944591+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7133843,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11936,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12020","reported_seq":12114,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:10.123914+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:10.123914+0000","last_peered":"2024-09-18T11:55:10.123914+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:10.123914+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7419126,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12018,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1195}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12019","reported_seq":12138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.486107+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:09.486107+0000","last_peered":"2024-09-18T11:55:09.486107+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:09.486107+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7173366,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11969","reported_seq":12086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.385953+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:09.385953+0000","last_peered":"2024-09-18T11:55:09.385953+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:09.385953+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11969,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12168","reported_seq":12275,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.915469+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:09.915469+0000","last_peered":"2024-09-18T11:55:09.915469+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:09.915469+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6943990,"num_objects":848,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11978","reported_seq":15261,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:10.174602+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:10.174602+0000","last_peered":"2024-09-18T11:55:10.174602+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:10.174602+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7345398,"num_objects":897,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11976,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1609,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12283","reported_seq":16636,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:09.414295+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:09.414295+0000","last_peered":"2024-09-18T11:55:09.414295+0000","last_clean":"2024-09-18T11:55:09.414295+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:09.414295+0000","last_undegraded":"2024-09-18T11:55:09.414295+0000","last_fullsized":"2024-09-18T11:55:09.414295+0000","mapping_epoch":36,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7309804,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12280,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11730","reported_seq":11843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.066793+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:09.066793+0000","last_peered":"2024-09-18T11:55:09.066793+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:09.066793+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075062,"num_objects":864,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11728,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1099}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12105","reported_seq":12225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.515610+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:09.515610+0000","last_peered":"2024-09-18T11:55:09.515610+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:09.515610+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7597937,"num_objects":928,"num_object_clones":0,"num_object_copies":3712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12102,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12185","reported_seq":12278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.512026+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:09.512026+0000","last_peered":"2024-09-18T11:55:09.512026+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:09.512026+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6929019,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1238}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12099","reported_seq":12197,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.084436+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:09.084436+0000","last_peered":"2024-09-18T11:55:09.084436+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:09.084436+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7109243,"num_objects":868,"num_object_clones":0,"num_object_copies":3472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1178}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12129","reported_seq":12254,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:09.484972+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:09.484972+0000","last_peered":"2024-09-18T11:55:09.484972+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:09.484972+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7869686,"num_objects":961,"num_object_clones":0,"num_object_copies":3844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12068","reported_seq":16357,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:11.971226+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:11.971226+0000","last_peered":"2024-09-18T11:55:11.971226+0000","last_clean":"2024-09-18T11:55:11.971226+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:11.971226+0000","last_undegraded":"2024-09-18T11:55:11.971226+0000","last_fullsized":"2024-09-18T11:55:11.971226+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6837494,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12066,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11809","reported_seq":11895,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:10.072198+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:10.072198+0000","last_peered":"2024-09-18T11:55:10.072198+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:10.072198+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7353590,"num_objects":898,"num_object_clones":0,"num_object_copies":3592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":115279244,"num_objects":14078,"num_object_clones":0,"num_object_copies":56312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14078,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193009,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8226,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":229822464,"data_stored":229822464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160836,"ondisk_log_size":160836,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":976676,"kb_used_data":57304,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93395164,"statfs":{"total":96636764160,"available":95636647936,"internally_reserved":0,"allocated":58679296,"data_stored":58495475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":8.4280000000000008},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":8.4600000000000009}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.4909999999999997},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":6.2060000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.2669999999999999},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":2.2229999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":985148,"kb_used_data":57972,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93386692,"statfs":{"total":96636764160,"available":95627972608,"internally_reserved":0,"allocated":59363328,"data_stored":59179507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":4.1870000000000003},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":4.2199999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.35499999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":0.32400000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":0.58699999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":981428,"kb_used_data":55912,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93390412,"statfs":{"total":96636764160,"available":95631781888,"internally_reserved":0,"allocated":57253888,"data_stored":57070067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":9.2739999999999991},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.006}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":1.3560000000000001},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.91100000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":15.880000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":1.387}]}]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":983924,"kb_used_data":56300,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93387916,"statfs":{"total":96636764160,"available":95629225984,"internally_reserved":0,"allocated":57651200,"data_stored":57467379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,4,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":7.0940000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.95699999999999996},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":7.141}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57327616,"data_stored":57327616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":56311808,"data_stored":56311808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58298368,"data_stored":58298368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57884672,"data_stored":57884672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:15.959 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:16.550 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:16.551 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:16.571 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-18T11:55:16.083722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107931020,"num_objects":13181,"num_object_clones":0,"num_object_copies":52724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13181,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193906,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8272,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3893552,"kb_used_data":214632,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373593808,"statfs":{"total":386547056640,"available":382560059392,"internally_reserved":0,"allocated":219783168,"data_stored":219047884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,5,15,7,8,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24174592,"num_objects":-2951,"num_object_clones":0,"num_object_copies":-11804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2951,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004761"},"pg_stats":[{"pgid":"1.f","version":"37'12433","reported_seq":12534,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.437234+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:14.437234+0000","last_peered":"2024-09-18T11:55:14.437234+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:14.437234+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12433,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12180","reported_seq":16472,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:12.011280+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:12.011280+0000","last_peered":"2024-09-18T11:55:12.011280+0000","last_clean":"2024-09-18T11:55:12.011280+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:12.011280+0000","last_undegraded":"2024-09-18T11:55:12.011280+0000","last_fullsized":"2024-09-18T11:55:12.011280+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7173366,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12178,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11993","reported_seq":12109,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.097983+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:14.097983+0000","last_peered":"2024-09-18T11:55:14.097983+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:14.097983+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6675091,"num_objects":816,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11992,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12081","reported_seq":12175,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.087208+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:15.087208+0000","last_peered":"2024-09-18T11:55:15.087208+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:15.087208+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6919414,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12079,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12076","reported_seq":12197,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.564401+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:14.564401+0000","last_peered":"2024-09-18T11:55:14.564401+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:14.564401+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6706422,"num_objects":819,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12074,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12027","reported_seq":12146,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.508736+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:14.508736+0000","last_peered":"2024-09-18T11:55:14.508736+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:14.508736+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6217728,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12230","reported_seq":12337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.199847+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:15.199847+0000","last_peered":"2024-09-18T11:55:15.199847+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:15.199847+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6436086,"num_objects":786,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1330}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12050","reported_seq":15426,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.157081+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:15.157081+0000","last_peered":"2024-09-18T11:55:15.157081+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:15.157081+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6755574,"num_objects":825,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12339","reported_seq":16692,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:14.088946+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:14.088946+0000","last_peered":"2024-09-18T11:55:14.088946+0000","last_clean":"2024-09-18T11:55:14.088946+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:14.088946+0000","last_undegraded":"2024-09-18T11:55:14.088946+0000","last_fullsized":"2024-09-18T11:55:14.088946+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6851052,"num_objects":837,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12336,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11785","reported_seq":11899,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.024871+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:14.024871+0000","last_peered":"2024-09-18T11:55:14.024871+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:14.024871+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6624502,"num_objects":809,"num_object_clones":0,"num_object_copies":3236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11783,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12163","reported_seq":12284,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.519614+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:14.519614+0000","last_peered":"2024-09-18T11:55:14.519614+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:14.519614+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7122801,"num_objects":870,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12160,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12256","reported_seq":12348,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.459230+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:14.459230+0000","last_peered":"2024-09-18T11:55:14.459230+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:14.459230+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6347387,"num_objects":775,"num_object_clones":0,"num_object_copies":3100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12255,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12154","reported_seq":12251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.111828+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:14.111828+0000","last_peered":"2024-09-18T11:55:14.111828+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:14.111828+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6658683,"num_objects":813,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12153,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1233}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12209","reported_seq":12333,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.458529+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:14.458529+0000","last_peered":"2024-09-18T11:55:14.458529+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:14.458529+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7214326,"num_objects":881,"num_object_clones":0,"num_object_copies":3524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12068","reported_seq":16357,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:11.971226+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:11.971226+0000","last_peered":"2024-09-18T11:55:11.971226+0000","last_clean":"2024-09-18T11:55:11.971226+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:11.971226+0000","last_undegraded":"2024-09-18T11:55:11.971226+0000","last_fullsized":"2024-09-18T11:55:11.971226+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6837494,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12066,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11889","reported_seq":11979,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.130857+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:15.130857+0000","last_peered":"2024-09-18T11:55:15.130857+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:15.130857+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6698230,"num_objects":818,"num_object_clones":0,"num_object_copies":3272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11887,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107931020,"num_objects":13181,"num_object_clones":0,"num_object_copies":52724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13181,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193906,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8272,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217362432,"data_stored":217362432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":976796,"kb_used_data":53256,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93395044,"statfs":{"total":96636764160,"available":95636525056,"internally_reserved":0,"allocated":54534144,"data_stored":54350323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":3.6259999999999999},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.316}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":10.202999999999999},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":6.4820000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.90000000000000002},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.80300000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967856,"kb_used_data":53148,"kb_used_omap":1,"kb_used_meta":914686,"kb_avail":93403984,"statfs":{"total":96636764160,"available":95645679616,"internally_reserved":0,"allocated":54423552,"data_stored":54239731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936638808},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,1,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":10.25},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":10.278}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":31.425999999999998},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":10.307}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":10.327999999999999},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":18.033999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":981428,"kb_used_data":55912,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93390412,"statfs":{"total":96636764160,"available":95631781888,"internally_reserved":0,"allocated":57253888,"data_stored":57070067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":9.2739999999999991},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.006}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":1.3560000000000001},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.91100000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":15.880000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":1.387}]}]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967472,"kb_used_data":52316,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93404368,"statfs":{"total":96636764160,"available":95646072832,"internally_reserved":0,"allocated":53571584,"data_stored":53387763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,7,1,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.71},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.0540000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.585},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":11.460000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":7.7460000000000004},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":11.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53198848,"data_stored":53198848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":56311808,"data_stored":56311808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54079488,"data_stored":54079488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53772288,"data_stored":53772288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:16.572 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:16.572 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12433", 'reported_seq': 12534, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:14.437234+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:55:14.437234+0000', 'last_peered': '2024-09-18T11:55:14.437234+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:55:14.437234+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6692864, 'num_objects': 817, 'num_object_clones': 0, 'num_object_copies': 3268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12433, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1286}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:16.572 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:16.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11993", 'reported_seq': 12109, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:14.097983+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:55:14.097983+0000', 'last_peered': '2024-09-18T11:55:14.097983+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:55:14.097983+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6675091, 'num_objects': 816, 'num_object_clones': 0, 'num_object_copies': 3264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 816, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11992, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1224}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:16.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:16.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12081", 'reported_seq': 12175, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:15.087208+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:15.087208+0000', 'last_peered': '2024-09-18T11:55:15.087208+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:15.087208+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6919414, 'num_objects': 845, 'num_object_clones': 0, 'num_object_copies': 3380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12079, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:16.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:16.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12076", 'reported_seq': 12197, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:14.564401+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:55:14.564401+0000', 'last_peered': '2024-09-18T11:55:14.564401+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:55:14.564401+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6706422, 'num_objects': 819, 'num_object_clones': 0, 'num_object_copies': 3276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12074, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1233}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:16.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:16.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12027", 'reported_seq': 12146, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:14.508736+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:55:14.508736+0000', 'last_peered': '2024-09-18T11:55:14.508736+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:55:14.508736+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6217728, 'num_objects': 759, 'num_object_clones': 0, 'num_object_copies': 3036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12027, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1360}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:16.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:16.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12230", 'reported_seq': 12337, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:15.199847+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:55:15.199847+0000', 'last_peered': '2024-09-18T11:55:15.199847+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:55:15.199847+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6436086, 'num_objects': 786, 'num_object_clones': 0, 'num_object_copies': 3144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12228, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1330}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:16.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:16.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12050", 'reported_seq': 15426, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:15.157081+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:55:15.157081+0000', 'last_peered': '2024-09-18T11:55:15.157081+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:55:15.157081+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6755574, 'num_objects': 825, 'num_object_clones': 0, 'num_object_copies': 3300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 383, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12048, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1655, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 442}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:16.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:16.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11785", 'reported_seq': 11899, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:14.024871+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:55:14.024871+0000', 'last_peered': '2024-09-18T11:55:14.024871+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:55:14.024871+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6624502, 'num_objects': 809, 'num_object_clones': 0, 'num_object_copies': 3236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11783, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1154}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:16.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:16.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12163", 'reported_seq': 12284, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:14.519614+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:55:14.519614+0000', 'last_peered': '2024-09-18T11:55:14.519614+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:55:14.519614+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7122801, 'num_objects': 870, 'num_object_clones': 0, 'num_object_copies': 3480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12160, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1164}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:16.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:16.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12256", 'reported_seq': 12348, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:14.459230+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:55:14.459230+0000', 'last_peered': '2024-09-18T11:55:14.459230+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:55:14.459230+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6347387, 'num_objects': 775, 'num_object_clones': 0, 'num_object_copies': 3100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12255, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:16.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:16.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12154", 'reported_seq': 12251, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:14.111828+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:55:14.111828+0000', 'last_peered': '2024-09-18T11:55:14.111828+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:55:14.111828+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6658683, 'num_objects': 813, 'num_object_clones': 0, 'num_object_copies': 3252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12153, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1233}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:16.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:16.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12209", 'reported_seq': 12333, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:14.458529+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:55:14.458529+0000', 'last_peered': '2024-09-18T11:55:14.458529+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:55:14.458529+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7214326, 'num_objects': 881, 'num_object_clones': 0, 'num_object_copies': 3524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12207, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1242}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:16.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:16.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11889", 'reported_seq': 11979, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:15.130857+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:55:15.130857+0000', 'last_peered': '2024-09-18T11:55:15.130857+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:55:15.130857+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6698230, 'num_objects': 818, 'num_object_clones': 0, 'num_object_copies': 3272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11887, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1184}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:16.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:17.284 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:17.284 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25184/52724 objects degraded (47.766%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":715,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13181,"data_bytes":107931020,"bytes_used":3986997248,"bytes_avail":382560059392,"bytes_total":386547056640,"degraded_objects":25184,"degraded_total":52724,"degraded_ratio":0.47765723389727638,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":245},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:17.304 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:17.305 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:17.951 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:17.951 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:17.977 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-18T11:55:16.083722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107931020,"num_objects":13181,"num_object_clones":0,"num_object_copies":52724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13181,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193906,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8272,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3893552,"kb_used_data":214632,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373593808,"statfs":{"total":386547056640,"available":382560059392,"internally_reserved":0,"allocated":219783168,"data_stored":219047884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,5,15,7,8,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24174592,"num_objects":-2951,"num_object_clones":0,"num_object_copies":-11804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2951,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004761"},"pg_stats":[{"pgid":"1.f","version":"37'12433","reported_seq":12534,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.437234+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:14.437234+0000","last_peered":"2024-09-18T11:55:14.437234+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:14.437234+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12433,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12180","reported_seq":16472,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:12.011280+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:12.011280+0000","last_peered":"2024-09-18T11:55:12.011280+0000","last_clean":"2024-09-18T11:55:12.011280+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:12.011280+0000","last_undegraded":"2024-09-18T11:55:12.011280+0000","last_fullsized":"2024-09-18T11:55:12.011280+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7173366,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12178,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11993","reported_seq":12109,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.097983+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:14.097983+0000","last_peered":"2024-09-18T11:55:14.097983+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:14.097983+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6675091,"num_objects":816,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11992,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12081","reported_seq":12175,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.087208+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:15.087208+0000","last_peered":"2024-09-18T11:55:15.087208+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:15.087208+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6919414,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12079,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12076","reported_seq":12197,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.564401+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:14.564401+0000","last_peered":"2024-09-18T11:55:14.564401+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:14.564401+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6706422,"num_objects":819,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12074,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12027","reported_seq":12146,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.508736+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:14.508736+0000","last_peered":"2024-09-18T11:55:14.508736+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:14.508736+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6217728,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12230","reported_seq":12337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.199847+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:15.199847+0000","last_peered":"2024-09-18T11:55:15.199847+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:15.199847+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6436086,"num_objects":786,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1330}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12050","reported_seq":15426,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.157081+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:15.157081+0000","last_peered":"2024-09-18T11:55:15.157081+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:15.157081+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6755574,"num_objects":825,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12339","reported_seq":16692,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:14.088946+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:14.088946+0000","last_peered":"2024-09-18T11:55:14.088946+0000","last_clean":"2024-09-18T11:55:14.088946+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:14.088946+0000","last_undegraded":"2024-09-18T11:55:14.088946+0000","last_fullsized":"2024-09-18T11:55:14.088946+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6851052,"num_objects":837,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12336,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11785","reported_seq":11899,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.024871+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:14.024871+0000","last_peered":"2024-09-18T11:55:14.024871+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:14.024871+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6624502,"num_objects":809,"num_object_clones":0,"num_object_copies":3236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11783,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12163","reported_seq":12284,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.519614+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:14.519614+0000","last_peered":"2024-09-18T11:55:14.519614+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:14.519614+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7122801,"num_objects":870,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12160,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12256","reported_seq":12348,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.459230+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:14.459230+0000","last_peered":"2024-09-18T11:55:14.459230+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:14.459230+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6347387,"num_objects":775,"num_object_clones":0,"num_object_copies":3100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12255,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12154","reported_seq":12251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.111828+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:14.111828+0000","last_peered":"2024-09-18T11:55:14.111828+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:14.111828+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6658683,"num_objects":813,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12153,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1233}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12209","reported_seq":12333,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:14.458529+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:14.458529+0000","last_peered":"2024-09-18T11:55:14.458529+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:14.458529+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7214326,"num_objects":881,"num_object_clones":0,"num_object_copies":3524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12068","reported_seq":16357,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:11.971226+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:11.971226+0000","last_peered":"2024-09-18T11:55:11.971226+0000","last_clean":"2024-09-18T11:55:11.971226+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:11.971226+0000","last_undegraded":"2024-09-18T11:55:11.971226+0000","last_fullsized":"2024-09-18T11:55:11.971226+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6837494,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12066,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11889","reported_seq":11979,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.130857+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:15.130857+0000","last_peered":"2024-09-18T11:55:15.130857+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:15.130857+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6698230,"num_objects":818,"num_object_clones":0,"num_object_copies":3272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11887,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107931020,"num_objects":13181,"num_object_clones":0,"num_object_copies":52724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13181,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193906,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8272,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217362432,"data_stored":217362432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":976796,"kb_used_data":53256,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93395044,"statfs":{"total":96636764160,"available":95636525056,"internally_reserved":0,"allocated":54534144,"data_stored":54350323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":3.6259999999999999},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.316}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":10.202999999999999},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":6.4820000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.90000000000000002},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.80300000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967856,"kb_used_data":53148,"kb_used_omap":1,"kb_used_meta":914686,"kb_avail":93403984,"statfs":{"total":96636764160,"available":95645679616,"internally_reserved":0,"allocated":54423552,"data_stored":54239731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936638808},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,1,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:16 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":5.1639999999999997,"15min":3.7109999999999999},"min":{"1min":0.24299999999999999,"5min":0.153,"15min":0.153},"max":{"1min":15.929,"5min":24.965,"15min":24.965},"last":10.25},{"interface":"front","average":{"1min":7.2530000000000001,"5min":5.3079999999999998,"15min":3.347},"min":{"1min":0.53100000000000003,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":15.949,"5min":24.995000000000001,"15min":24.995000000000001},"last":10.278}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":31.425999999999998},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":10.307}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":10.327999999999999},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":18.033999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":981428,"kb_used_data":55912,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93390412,"statfs":{"total":96636764160,"available":95631781888,"internally_reserved":0,"allocated":57253888,"data_stored":57070067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":9.2739999999999991},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":1.006}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":1.3560000000000001},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.91100000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":15.880000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":1.387}]}]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967472,"kb_used_data":52316,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93404368,"statfs":{"total":96636764160,"available":95646072832,"internally_reserved":0,"allocated":53571584,"data_stored":53387763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,7,1,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.71},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.0540000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.585},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":11.460000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":7.7460000000000004},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":11.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53198848,"data_stored":53198848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":56311808,"data_stored":56311808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54079488,"data_stored":54079488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53772288,"data_stored":53772288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:20.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:21.509 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:21.510 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:21.538 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-09-18T11:55:20.085281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102057356,"num_objects":12464,"num_object_clones":0,"num_object_copies":49856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194623,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8272,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3893468,"kb_used_data":202076,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373593892,"statfs":{"total":386547056640,"available":382560145408,"internally_reserved":0,"allocated":206925824,"data_stored":206190540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,16,12,14,9,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21487616,"num_objects":-2623,"num_object_clones":0,"num_object_copies":-10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2623,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004670"},"pg_stats":[{"pgid":"1.f","version":"37'12488","reported_seq":12589,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.415021+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:19.415021+0000","last_peered":"2024-09-18T11:55:19.415021+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:19.415021+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6242304,"num_objects":762,"num_object_clones":0,"num_object_copies":3048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12488,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1341}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12251","reported_seq":16543,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:17.021650+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:17.021650+0000","last_peered":"2024-09-18T11:55:17.021650+0000","last_clean":"2024-09-18T11:55:17.021650+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:17.021650+0000","last_undegraded":"2024-09-18T11:55:17.021650+0000","last_fullsized":"2024-09-18T11:55:17.021650+0000","mapping_epoch":34,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6591734,"num_objects":805,"num_object_clones":0,"num_object_copies":3220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12050","reported_seq":12167,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.069312+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:19.069312+0000","last_peered":"2024-09-18T11:55:19.069312+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:19.069312+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6208147,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12049,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1281}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12081","reported_seq":12175,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.087208+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:15.087208+0000","last_peered":"2024-09-18T11:55:15.087208+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:15.087208+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6919414,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12079,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12130","reported_seq":12250,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.623572+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:19.623572+0000","last_peered":"2024-09-18T11:55:19.623572+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:19.623572+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6264054,"num_objects":765,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12128,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12084","reported_seq":12203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.490969+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:19.490969+0000","last_peered":"2024-09-18T11:55:19.490969+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:19.490969+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5750784,"num_objects":702,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12084,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12230","reported_seq":12337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.199847+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:15.199847+0000","last_peered":"2024-09-18T11:55:15.199847+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:15.199847+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6436086,"num_objects":786,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1330}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12050","reported_seq":15426,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.157081+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:15.157081+0000","last_peered":"2024-09-18T11:55:15.157081+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:15.157081+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6755574,"num_objects":825,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12406","reported_seq":16759,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:19.516846+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:19.516846+0000","last_peered":"2024-09-18T11:55:19.516846+0000","last_clean":"2024-09-18T11:55:19.516846+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:19.516846+0000","last_undegraded":"2024-09-18T11:55:19.516846+0000","last_fullsized":"2024-09-18T11:55:19.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302188,"num_objects":770,"num_object_clones":0,"num_object_copies":3080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12403,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11864","reported_seq":11978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.220184+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:19.220184+0000","last_peered":"2024-09-18T11:55:19.220184+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:19.220184+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5977334,"num_objects":730,"num_object_clones":0,"num_object_copies":2920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11862,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1233}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12218","reported_seq":12339,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.502222+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:19.502222+0000","last_peered":"2024-09-18T11:55:19.502222+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:19.502222+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672241,"num_objects":815,"num_object_clones":0,"num_object_copies":3260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12215,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12309","reported_seq":12402,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.590811+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:19.590811+0000","last_peered":"2024-09-18T11:55:19.590811+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:19.590811+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5913211,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12308,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1362}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12206","reported_seq":12304,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:18.990591+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:18.990591+0000","last_peered":"2024-09-18T11:55:18.990591+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:18.990591+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6232699,"num_objects":761,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12205,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1285}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12269","reported_seq":12394,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.529785+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:19.529785+0000","last_peered":"2024-09-18T11:55:19.529785+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:19.529785+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722806,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12267,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1302}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12125","reported_seq":16414,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:16.984769+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:16.984769+0000","last_peered":"2024-09-18T11:55:16.984769+0000","last_clean":"2024-09-18T11:55:16.984769+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:16.984769+0000","last_undegraded":"2024-09-18T11:55:16.984769+0000","last_fullsized":"2024-09-18T11:55:16.984769+0000","mapping_epoch":32,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6370550,"num_objects":778,"num_object_clones":0,"num_object_copies":3112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12123,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11889","reported_seq":11979,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:15.130857+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:15.130857+0000","last_peered":"2024-09-18T11:55:15.130857+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:15.130857+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6698230,"num_objects":818,"num_object_clones":0,"num_object_copies":3272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11887,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":102057356,"num_objects":12464,"num_object_clones":0,"num_object_copies":49856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194623,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8272,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204906496,"data_stored":204906496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":976980,"kb_used_data":49292,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93394860,"statfs":{"total":96636764160,"available":95636336640,"internally_reserved":0,"allocated":50475008,"data_stored":50291187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":15.881},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.008}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":4.9800000000000004},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":12.960000000000001},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.56999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968112,"kb_used_data":49236,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93403728,"statfs":{"total":96636764160,"available":95645417472,"internally_reserved":0,"allocated":50417664,"data_stored":50233843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,2,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":4.7789999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":4.8129999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":1.391},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":4.8490000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.3420000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":5.7889999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":980904,"kb_used_data":51232,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93390936,"statfs":{"total":96636764160,"available":95632318464,"internally_reserved":0,"allocated":52461568,"data_stored":52277747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":7.1260000000000003},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":7.7400000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.17799999999999999},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":7.093}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":9.2840000000000007},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.5960000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967472,"kb_used_data":52316,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93404368,"statfs":{"total":96636764160,"available":95646072832,"internally_reserved":0,"allocated":53571584,"data_stored":53387763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,7,1,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.71},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":6.0540000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.585},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":11.460000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":7.7460000000000004},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":11.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53198848,"data_stored":53198848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":52072448,"data_stored":52072448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49975296,"data_stored":49975296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49659904,"data_stored":49659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:21.540 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:22.126 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:22.126 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:22.145 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-18T11:55:22.085951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99829132,"num_objects":12192,"num_object_clones":0,"num_object_copies":48768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194895,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8319,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3893568,"kb_used_data":198028,"kb_used_omap":14,"kb_used_meta":3695473,"kb_avail":373593792,"statfs":{"total":386547056640,"available":382560043008,"internally_reserved":0,"allocated":202780672,"data_stored":202045388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3784164379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,17,10,12,10,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18677760,"num_objects":-2280,"num_object_clones":0,"num_object_copies":-9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2280,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004507"},"pg_stats":[{"pgid":"1.f","version":"37'12488","reported_seq":12589,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.415021+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:19.415021+0000","last_peered":"2024-09-18T11:55:19.415021+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:19.415021+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6242304,"num_objects":762,"num_object_clones":0,"num_object_copies":3048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12488,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1341}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12251","reported_seq":16543,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:17.021650+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:17.021650+0000","last_peered":"2024-09-18T11:55:17.021650+0000","last_clean":"2024-09-18T11:55:17.021650+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:17.021650+0000","last_undegraded":"2024-09-18T11:55:17.021650+0000","last_fullsized":"2024-09-18T11:55:17.021650+0000","mapping_epoch":34,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6591734,"num_objects":805,"num_object_clones":0,"num_object_copies":3220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12050","reported_seq":12167,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.069312+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:19.069312+0000","last_peered":"2024-09-18T11:55:19.069312+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:19.069312+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6208147,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12049,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1281}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12143","reported_seq":12237,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:20.244008+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:20.244008+0000","last_peered":"2024-09-18T11:55:20.244008+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:20.244008+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6411510,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12141,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12130","reported_seq":12250,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.623572+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:19.623572+0000","last_peered":"2024-09-18T11:55:19.623572+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:19.623572+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6264054,"num_objects":765,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12128,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12084","reported_seq":12203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.490969+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:19.490969+0000","last_peered":"2024-09-18T11:55:19.490969+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:19.490969+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5750784,"num_objects":702,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12084,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12290","reported_seq":12397,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:20.244554+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:20.244554+0000","last_peered":"2024-09-18T11:55:20.244554+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:20.244554+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5944566,"num_objects":726,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12288,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1390}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12133","reported_seq":15603,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:20.298662+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:20.298662+0000","last_peered":"2024-09-18T11:55:20.298662+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:20.298662+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6075638,"num_objects":742,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1702,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12406","reported_seq":16759,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:19.516846+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:19.516846+0000","last_peered":"2024-09-18T11:55:19.516846+0000","last_clean":"2024-09-18T11:55:19.516846+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:19.516846+0000","last_undegraded":"2024-09-18T11:55:19.516846+0000","last_fullsized":"2024-09-18T11:55:19.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302188,"num_objects":770,"num_object_clones":0,"num_object_copies":3080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12403,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11864","reported_seq":11978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.220184+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:19.220184+0000","last_peered":"2024-09-18T11:55:19.220184+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:19.220184+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5977334,"num_objects":730,"num_object_clones":0,"num_object_copies":2920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11862,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1233}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12218","reported_seq":12339,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.502222+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:19.502222+0000","last_peered":"2024-09-18T11:55:19.502222+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:19.502222+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672241,"num_objects":815,"num_object_clones":0,"num_object_copies":3260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12215,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12309","reported_seq":12402,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.590811+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:19.590811+0000","last_peered":"2024-09-18T11:55:19.590811+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:19.590811+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5913211,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12308,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1362}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12206","reported_seq":12304,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:18.990591+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:18.990591+0000","last_peered":"2024-09-18T11:55:18.990591+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:18.990591+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6232699,"num_objects":761,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12205,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1285}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12269","reported_seq":12394,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.529785+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:19.529785+0000","last_peered":"2024-09-18T11:55:19.529785+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:19.529785+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722806,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12267,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1302}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12125","reported_seq":16414,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:16.984769+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:16.984769+0000","last_peered":"2024-09-18T11:55:16.984769+0000","last_clean":"2024-09-18T11:55:16.984769+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:16.984769+0000","last_undegraded":"2024-09-18T11:55:16.984769+0000","last_fullsized":"2024-09-18T11:55:16.984769+0000","mapping_epoch":32,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6370550,"num_objects":778,"num_object_clones":0,"num_object_copies":3112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12123,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11956","reported_seq":12046,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.903990+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:19.903990+0000","last_peered":"2024-09-18T11:55:19.903990+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:19.903990+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6149366,"num_objects":751,"num_object_clones":0,"num_object_copies":3004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":99829132,"num_objects":12192,"num_object_clones":0,"num_object_copies":48768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194895,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8319,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200753152,"data_stored":200753152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":976980,"kb_used_data":49292,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93394860,"statfs":{"total":96636764160,"available":95636336640,"internally_reserved":0,"allocated":50475008,"data_stored":50291187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":15.881},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.008}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":4.9800000000000004},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":12.960000000000001},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.56999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968112,"kb_used_data":49236,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93403728,"statfs":{"total":96636764160,"available":95645417472,"internally_reserved":0,"allocated":50417664,"data_stored":50233843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,2,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":4.7789999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":4.8129999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":1.391},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":4.8490000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.3420000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":5.7889999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":980904,"kb_used_data":51232,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93390936,"statfs":{"total":96636764160,"available":95632318464,"internally_reserved":0,"allocated":52461568,"data_stored":52277747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":7.1260000000000003},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":7.7400000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.17799999999999999},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":7.093}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":9.2840000000000007},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.5960000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967572,"kb_used_data":48268,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93404268,"statfs":{"total":96636764160,"available":95645970432,"internally_reserved":0,"allocated":49426432,"data_stored":49242611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,5,2,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":2.1789999999999998},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":3.1749999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":8.3460000000000001},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.7949999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":2.0409999999999999},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49045504,"data_stored":49045504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":52072448,"data_stored":52072448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49975296,"data_stored":49975296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49659904,"data_stored":49659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:22.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:22.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12488", 'reported_seq': 12589, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:19.415021+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:55:19.415021+0000', 'last_peered': '2024-09-18T11:55:19.415021+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:55:19.415021+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6242304, 'num_objects': 762, 'num_object_clones': 0, 'num_object_copies': 3048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12488, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1341}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:22.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:22.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12050", 'reported_seq': 12167, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:19.069312+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:55:19.069312+0000', 'last_peered': '2024-09-18T11:55:19.069312+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:55:19.069312+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6208147, 'num_objects': 759, 'num_object_clones': 0, 'num_object_copies': 3036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 759, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12049, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1281}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:22.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:22.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12143", 'reported_seq': 12237, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:20.244008+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:20.244008+0000', 'last_peered': '2024-09-18T11:55:20.244008+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:20.244008+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6411510, 'num_objects': 783, 'num_object_clones': 0, 'num_object_copies': 3132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12141, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:22.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:22.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12130", 'reported_seq': 12250, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:19.623572+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:55:19.623572+0000', 'last_peered': '2024-09-18T11:55:19.623572+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:55:19.623572+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6264054, 'num_objects': 765, 'num_object_clones': 0, 'num_object_copies': 3060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12128, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1287}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:22.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:22.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12084", 'reported_seq': 12203, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:19.490969+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:55:19.490969+0000', 'last_peered': '2024-09-18T11:55:19.490969+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:55:19.490969+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5750784, 'num_objects': 702, 'num_object_clones': 0, 'num_object_copies': 2808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12084, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1417}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:22.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:22.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12290", 'reported_seq': 12397, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:20.244554+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:55:20.244554+0000', 'last_peered': '2024-09-18T11:55:20.244554+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:55:20.244554+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5944566, 'num_objects': 726, 'num_object_clones': 0, 'num_object_copies': 2904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 726, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12288, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1390}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:22.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:22.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12133", 'reported_seq': 15603, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:20.298662+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:55:20.298662+0000', 'last_peered': '2024-09-18T11:55:20.298662+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:55:20.298662+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6075638, 'num_objects': 742, 'num_object_clones': 0, 'num_object_copies': 2968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12131, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1702, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 406}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:22.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:22.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11864", 'reported_seq': 11978, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:19.220184+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:55:19.220184+0000', 'last_peered': '2024-09-18T11:55:19.220184+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:55:19.220184+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5977334, 'num_objects': 730, 'num_object_clones': 0, 'num_object_copies': 2920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11862, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1233}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:22.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:22.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12218", 'reported_seq': 12339, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:19.502222+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:55:19.502222+0000', 'last_peered': '2024-09-18T11:55:19.502222+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:55:19.502222+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6672241, 'num_objects': 815, 'num_object_clones': 0, 'num_object_copies': 3260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12215, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:22.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:22.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12309", 'reported_seq': 12402, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:19.590811+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:55:19.590811+0000', 'last_peered': '2024-09-18T11:55:19.590811+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:55:19.590811+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5913211, 'num_objects': 722, 'num_object_clones': 0, 'num_object_copies': 2888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12308, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1362}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:22.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:22.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12206", 'reported_seq': 12304, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:18.990591+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:55:18.990591+0000', 'last_peered': '2024-09-18T11:55:18.990591+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:55:18.990591+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6232699, 'num_objects': 761, 'num_object_clones': 0, 'num_object_copies': 3044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12205, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1285}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:22.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:22.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12269", 'reported_seq': 12394, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:19.529785+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:55:19.529785+0000', 'last_peered': '2024-09-18T11:55:19.529785+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:55:19.529785+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6722806, 'num_objects': 821, 'num_object_clones': 0, 'num_object_copies': 3284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12267, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1302}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:22.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:22.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11956", 'reported_seq': 12046, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:19.903990+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:55:19.903990+0000', 'last_peered': '2024-09-18T11:55:19.903990+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:55:19.903990+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6149366, 'num_objects': 751, 'num_object_clones': 0, 'num_object_copies': 3004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1251}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:22.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:22.941 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:22.941 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25184/49856 objects degraded (50.513%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":721,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12464,"data_bytes":102057356,"bytes_used":3986911232,"bytes_avail":382560145408,"bytes_total":386547056640,"degraded_objects":25184,"degraded_total":49856,"degraded_ratio":0.50513478818998714,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":218},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:22.966 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:22.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:23.526 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:23.527 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:23.549 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-18T11:55:22.085951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99829132,"num_objects":12192,"num_object_clones":0,"num_object_copies":48768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194895,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8319,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3893568,"kb_used_data":198028,"kb_used_omap":14,"kb_used_meta":3695473,"kb_avail":373593792,"statfs":{"total":386547056640,"available":382560043008,"internally_reserved":0,"allocated":202780672,"data_stored":202045388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3784164379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,17,10,12,10,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18677760,"num_objects":-2280,"num_object_clones":0,"num_object_copies":-9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2280,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004507"},"pg_stats":[{"pgid":"1.f","version":"37'12488","reported_seq":12589,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.415021+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:19.415021+0000","last_peered":"2024-09-18T11:55:19.415021+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:19.415021+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6242304,"num_objects":762,"num_object_clones":0,"num_object_copies":3048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12488,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1341}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12251","reported_seq":16543,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:17.021650+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:17.021650+0000","last_peered":"2024-09-18T11:55:17.021650+0000","last_clean":"2024-09-18T11:55:17.021650+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:17.021650+0000","last_undegraded":"2024-09-18T11:55:17.021650+0000","last_fullsized":"2024-09-18T11:55:17.021650+0000","mapping_epoch":34,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6591734,"num_objects":805,"num_object_clones":0,"num_object_copies":3220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12050","reported_seq":12167,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.069312+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:19.069312+0000","last_peered":"2024-09-18T11:55:19.069312+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:19.069312+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6208147,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12049,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1281}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12143","reported_seq":12237,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:20.244008+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:20.244008+0000","last_peered":"2024-09-18T11:55:20.244008+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:20.244008+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6411510,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12141,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12130","reported_seq":12250,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.623572+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:19.623572+0000","last_peered":"2024-09-18T11:55:19.623572+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:19.623572+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6264054,"num_objects":765,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12128,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12084","reported_seq":12203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.490969+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:19.490969+0000","last_peered":"2024-09-18T11:55:19.490969+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:19.490969+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5750784,"num_objects":702,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12084,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12290","reported_seq":12397,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:20.244554+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:20.244554+0000","last_peered":"2024-09-18T11:55:20.244554+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:20.244554+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5944566,"num_objects":726,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12288,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1390}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12133","reported_seq":15603,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:20.298662+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:20.298662+0000","last_peered":"2024-09-18T11:55:20.298662+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:20.298662+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6075638,"num_objects":742,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1702,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12406","reported_seq":16759,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:19.516846+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:19.516846+0000","last_peered":"2024-09-18T11:55:19.516846+0000","last_clean":"2024-09-18T11:55:19.516846+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:19.516846+0000","last_undegraded":"2024-09-18T11:55:19.516846+0000","last_fullsized":"2024-09-18T11:55:19.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302188,"num_objects":770,"num_object_clones":0,"num_object_copies":3080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12403,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11864","reported_seq":11978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.220184+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:19.220184+0000","last_peered":"2024-09-18T11:55:19.220184+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:19.220184+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5977334,"num_objects":730,"num_object_clones":0,"num_object_copies":2920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11862,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1233}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12218","reported_seq":12339,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.502222+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:19.502222+0000","last_peered":"2024-09-18T11:55:19.502222+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:19.502222+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672241,"num_objects":815,"num_object_clones":0,"num_object_copies":3260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12215,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12309","reported_seq":12402,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.590811+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:19.590811+0000","last_peered":"2024-09-18T11:55:19.590811+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:19.590811+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5913211,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12308,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1362}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12206","reported_seq":12304,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:18.990591+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:18.990591+0000","last_peered":"2024-09-18T11:55:18.990591+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:18.990591+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6232699,"num_objects":761,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12205,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1285}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12269","reported_seq":12394,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.529785+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:19.529785+0000","last_peered":"2024-09-18T11:55:19.529785+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:19.529785+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722806,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12267,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1302}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12125","reported_seq":16414,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:16.984769+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:16.984769+0000","last_peered":"2024-09-18T11:55:16.984769+0000","last_clean":"2024-09-18T11:55:16.984769+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:16.984769+0000","last_undegraded":"2024-09-18T11:55:16.984769+0000","last_fullsized":"2024-09-18T11:55:16.984769+0000","mapping_epoch":32,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6370550,"num_objects":778,"num_object_clones":0,"num_object_copies":3112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12123,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'11956","reported_seq":12046,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:19.903990+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:19.903990+0000","last_peered":"2024-09-18T11:55:19.903990+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:19.903990+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6149366,"num_objects":751,"num_object_clones":0,"num_object_copies":3004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":99829132,"num_objects":12192,"num_object_clones":0,"num_object_copies":48768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194895,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8319,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200753152,"data_stored":200753152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":976980,"kb_used_data":49292,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93394860,"statfs":{"total":96636764160,"available":95636336640,"internally_reserved":0,"allocated":50475008,"data_stored":50291187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":15.881},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.008}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":4.9800000000000004},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":12.960000000000001},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":0.56999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968112,"kb_used_data":49236,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93403728,"statfs":{"total":96636764160,"available":95645417472,"internally_reserved":0,"allocated":50417664,"data_stored":50233843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,2,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":4.7789999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":4.8129999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":1.391},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":4.8490000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":1.3420000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":5.7889999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":980904,"kb_used_data":51232,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93390936,"statfs":{"total":96636764160,"available":95632318464,"internally_reserved":0,"allocated":52461568,"data_stored":52277747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":7.1260000000000003},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":7.7400000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.17799999999999999},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":7.093}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":9.2840000000000007},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.5960000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967572,"kb_used_data":48268,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93404268,"statfs":{"total":96636764160,"available":95645970432,"internally_reserved":0,"allocated":49426432,"data_stored":49242611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,5,2,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":2.1789999999999998},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":3.1749999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:54:18 2024","interfaces":[{"interface":"back","average":{"1min":6.4109999999999996,"5min":6.5880000000000001,"15min":4.0179999999999998},"min":{"1min":0.311,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":18.751000000000001,"5min":28.969000000000001,"15min":32.795000000000002},"last":8.3460000000000001},{"interface":"front","average":{"1min":6.4550000000000001,"5min":5.9509999999999996,"15min":3.6040000000000001},"min":{"1min":0.35699999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.099,"5min":36.697000000000003,"15min":36.697000000000003},"last":1.7949999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:54:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":6.4660000000000002,"15min":4.0620000000000003},"min":{"1min":0.28199999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":37.414000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":2.0409999999999999},{"interface":"front","average":{"1min":13.278,"5min":8.9169999999999998,"15min":4.9450000000000003},"min":{"1min":1.0780000000000001,"5min":0.26100000000000001,"15min":0.17299999999999999},"max":{"1min":48.185000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49045504,"data_stored":49045504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":52072448,"data_stored":52072448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49975296,"data_stored":49975296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49659904,"data_stored":49659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:26.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:27.165 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:27.165 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:27.188 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-18T11:55:26.087279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91497868,"num_objects":11175,"num_object_clones":0,"num_object_copies":44700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195912,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8373,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3894140,"kb_used_data":181972,"kb_used_omap":14,"kb_used_meta":3712113,"kb_avail":373593220,"statfs":{"total":386547056640,"available":382559457280,"internally_reserved":0,"allocated":186339328,"data_stored":185604044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3801203739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,10,13,16,14,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23781376,"num_objects":-2903,"num_object_clones":0,"num_object_copies":-11612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2903,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004268"},"pg_stats":[{"pgid":"1.f","version":"37'12554","reported_seq":12655,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.574814+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:24.574814+0000","last_peered":"2024-09-18T11:55:24.574814+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:24.574814+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":696,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12324","reported_seq":16616,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:22.350044+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:22.350044+0000","last_peered":"2024-09-18T11:55:22.350044+0000","last_clean":"2024-09-18T11:55:22.350044+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:22.350044+0000","last_undegraded":"2024-09-18T11:55:22.350044+0000","last_fullsized":"2024-09-18T11:55:22.350044+0000","mapping_epoch":34,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5993718,"num_objects":732,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12322,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12110","reported_seq":12227,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.285770+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:24.285770+0000","last_peered":"2024-09-18T11:55:24.285770+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:24.285770+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5716627,"num_objects":699,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12109,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12208","reported_seq":12302,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.221377+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:25.221377+0000","last_peered":"2024-09-18T11:55:25.221377+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:25.221377+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5879030,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12206,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12186","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.579860+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:24.579860+0000","last_peered":"2024-09-18T11:55:24.579860+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:24.579860+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805302,"num_objects":709,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1343}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12150","reported_seq":12269,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.506197+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:24.506197+0000","last_peered":"2024-09-18T11:55:24.506197+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:24.506197+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210112,"num_objects":636,"num_object_clones":0,"num_object_copies":2544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12353","reported_seq":12460,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.182785+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:25.182785+0000","last_peered":"2024-09-18T11:55:25.182785+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:25.182785+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428470,"num_objects":663,"num_object_clones":0,"num_object_copies":2652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1453}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12197","reported_seq":15775,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.348150+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:25.348150+0000","last_peered":"2024-09-18T11:55:25.348150+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:25.348150+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5551350,"num_objects":678,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12195,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12458","reported_seq":16811,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:24.620191+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:24.620191+0000","last_peered":"2024-09-18T11:55:24.620191+0000","last_clean":"2024-09-18T11:55:24.620191+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:24.620191+0000","last_undegraded":"2024-09-18T11:55:24.620191+0000","last_fullsized":"2024-09-18T11:55:24.620191+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5876204,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12455,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11915","reported_seq":12029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.242854+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:24.242854+0000","last_peered":"2024-09-18T11:55:24.242854+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:24.242854+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5559542,"num_objects":679,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11913,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1284}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12289","reported_seq":12410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.577484+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:24.577484+0000","last_peered":"2024-09-18T11:55:24.577484+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:24.577484+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6090609,"num_objects":744,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":12476,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.557630+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:24.557630+0000","last_peered":"2024-09-18T11:55:24.557630+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:24.557630+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5298811,"num_objects":647,"num_object_clones":0,"num_object_copies":2588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12383,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1437}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12267","reported_seq":12365,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.093053+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:24.093053+0000","last_peered":"2024-09-18T11:55:24.093053+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:24.093053+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5732987,"num_objects":700,"num_object_clones":0,"num_object_copies":2800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12266,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1346}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12337","reported_seq":12462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.512782+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:24.512782+0000","last_peered":"2024-09-18T11:55:24.512782+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:24.512782+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6165750,"num_objects":753,"num_object_clones":0,"num_object_copies":3012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12335,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1370}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12185","reported_seq":16474,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:22.043847+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:22.043847+0000","last_peered":"2024-09-18T11:55:22.043847+0000","last_clean":"2024-09-18T11:55:22.043847+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:22.043847+0000","last_undegraded":"2024-09-18T11:55:22.043847+0000","last_fullsized":"2024-09-18T11:55:22.043847+0000","mapping_epoch":32,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5879030,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12022","reported_seq":12112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.236407+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:25.236407+0000","last_peered":"2024-09-18T11:55:25.236407+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:25.236407+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5608694,"num_objects":685,"num_object_clones":0,"num_object_copies":2740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12020,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1317}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":91497868,"num_objects":11175,"num_object_clones":0,"num_object_copies":44700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195912,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8373,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184299520,"data_stored":184299520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":977172,"kb_used_data":45324,"kb_used_omap":4,"kb_used_meta":931835,"kb_avail":93394668,"statfs":{"total":96636764160,"available":95636140032,"internally_reserved":0,"allocated":46411776,"data_stored":46227955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954199617},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,5,2,5,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":16.913},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.54400000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.7279999999999998},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":14.680999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":14.784000000000001},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":4.9409999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968336,"kb_used_data":45316,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93403504,"statfs":{"total":96636764160,"available":95645188096,"internally_reserved":0,"allocated":46403584,"data_stored":46219763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.73499999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.77300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.183},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":7.1920000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":22.047000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":1.728}]}]},{"osd":1,"up_from":29,"seq":124554051711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":980852,"kb_used_data":47020,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93390988,"statfs":{"total":96636764160,"available":95632371712,"internally_reserved":0,"allocated":48148480,"data_stored":47964659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":13.263999999999999},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":12.497999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":13.526999999999999},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":11.093999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":6.4779999999999998},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":11.259}]}]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967780,"kb_used_data":44312,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93404060,"statfs":{"total":96636764160,"available":95645757440,"internally_reserved":0,"allocated":45375488,"data_stored":45191667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,5,7,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":4.7720000000000002},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.3580000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.7709999999999999},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":2.3250000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":3.4449999999999998},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":6.282}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44990464,"data_stored":44990464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47820800,"data_stored":47820800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45883392,"data_stored":45883392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45604864,"data_stored":45604864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:27.189 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:27.719 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:27.721 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:27.742 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-18T11:55:26.087279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91497868,"num_objects":11175,"num_object_clones":0,"num_object_copies":44700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195912,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8373,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3894140,"kb_used_data":181972,"kb_used_omap":14,"kb_used_meta":3712113,"kb_avail":373593220,"statfs":{"total":386547056640,"available":382559457280,"internally_reserved":0,"allocated":186339328,"data_stored":185604044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3801203739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,10,13,16,14,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23781376,"num_objects":-2903,"num_object_clones":0,"num_object_copies":-11612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2903,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004268"},"pg_stats":[{"pgid":"1.f","version":"37'12554","reported_seq":12655,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.574814+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:24.574814+0000","last_peered":"2024-09-18T11:55:24.574814+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:24.574814+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":696,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12324","reported_seq":16616,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:22.350044+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:22.350044+0000","last_peered":"2024-09-18T11:55:22.350044+0000","last_clean":"2024-09-18T11:55:22.350044+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:22.350044+0000","last_undegraded":"2024-09-18T11:55:22.350044+0000","last_fullsized":"2024-09-18T11:55:22.350044+0000","mapping_epoch":34,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5993718,"num_objects":732,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12322,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12110","reported_seq":12227,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.285770+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:24.285770+0000","last_peered":"2024-09-18T11:55:24.285770+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:24.285770+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5716627,"num_objects":699,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12109,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12208","reported_seq":12302,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.221377+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:25.221377+0000","last_peered":"2024-09-18T11:55:25.221377+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:25.221377+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5879030,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12206,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12186","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.579860+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:24.579860+0000","last_peered":"2024-09-18T11:55:24.579860+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:24.579860+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805302,"num_objects":709,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1343}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12150","reported_seq":12269,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.506197+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:24.506197+0000","last_peered":"2024-09-18T11:55:24.506197+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:24.506197+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210112,"num_objects":636,"num_object_clones":0,"num_object_copies":2544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12353","reported_seq":12460,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.182785+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:25.182785+0000","last_peered":"2024-09-18T11:55:25.182785+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:25.182785+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428470,"num_objects":663,"num_object_clones":0,"num_object_copies":2652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1453}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12197","reported_seq":15775,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.348150+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:25.348150+0000","last_peered":"2024-09-18T11:55:25.348150+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:25.348150+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5551350,"num_objects":678,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12195,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12458","reported_seq":16811,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:24.620191+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:24.620191+0000","last_peered":"2024-09-18T11:55:24.620191+0000","last_clean":"2024-09-18T11:55:24.620191+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:24.620191+0000","last_undegraded":"2024-09-18T11:55:24.620191+0000","last_fullsized":"2024-09-18T11:55:24.620191+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5876204,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12455,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11915","reported_seq":12029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.242854+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:24.242854+0000","last_peered":"2024-09-18T11:55:24.242854+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:24.242854+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5559542,"num_objects":679,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11913,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1284}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12289","reported_seq":12410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.577484+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:24.577484+0000","last_peered":"2024-09-18T11:55:24.577484+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:24.577484+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6090609,"num_objects":744,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":12476,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.557630+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:24.557630+0000","last_peered":"2024-09-18T11:55:24.557630+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:24.557630+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5298811,"num_objects":647,"num_object_clones":0,"num_object_copies":2588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12383,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1437}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12267","reported_seq":12365,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.093053+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:24.093053+0000","last_peered":"2024-09-18T11:55:24.093053+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:24.093053+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5732987,"num_objects":700,"num_object_clones":0,"num_object_copies":2800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12266,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1346}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12337","reported_seq":12462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.512782+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:24.512782+0000","last_peered":"2024-09-18T11:55:24.512782+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:24.512782+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6165750,"num_objects":753,"num_object_clones":0,"num_object_copies":3012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12335,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1370}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12185","reported_seq":16474,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:22.043847+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:22.043847+0000","last_peered":"2024-09-18T11:55:22.043847+0000","last_clean":"2024-09-18T11:55:22.043847+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:22.043847+0000","last_undegraded":"2024-09-18T11:55:22.043847+0000","last_fullsized":"2024-09-18T11:55:22.043847+0000","mapping_epoch":32,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5879030,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12022","reported_seq":12112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.236407+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:25.236407+0000","last_peered":"2024-09-18T11:55:25.236407+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:25.236407+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5608694,"num_objects":685,"num_object_clones":0,"num_object_copies":2740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12020,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1317}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":91497868,"num_objects":11175,"num_object_clones":0,"num_object_copies":44700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195912,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8373,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184299520,"data_stored":184299520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":977172,"kb_used_data":45324,"kb_used_omap":4,"kb_used_meta":931835,"kb_avail":93394668,"statfs":{"total":96636764160,"available":95636140032,"internally_reserved":0,"allocated":46411776,"data_stored":46227955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954199617},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,5,2,5,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":16.913},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.54400000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.7279999999999998},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":14.680999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":14.784000000000001},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":4.9409999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968336,"kb_used_data":45316,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93403504,"statfs":{"total":96636764160,"available":95645188096,"internally_reserved":0,"allocated":46403584,"data_stored":46219763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.73499999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.77300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.183},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":7.1920000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":22.047000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":1.728}]}]},{"osd":1,"up_from":29,"seq":124554051711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":980852,"kb_used_data":47020,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93390988,"statfs":{"total":96636764160,"available":95632371712,"internally_reserved":0,"allocated":48148480,"data_stored":47964659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":13.263999999999999},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":12.497999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":13.526999999999999},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":11.093999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":6.4779999999999998},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":11.259}]}]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967780,"kb_used_data":44312,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93404060,"statfs":{"total":96636764160,"available":95645757440,"internally_reserved":0,"allocated":45375488,"data_stored":45191667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,5,7,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":4.7720000000000002},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.3580000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.7709999999999999},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":2.3250000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":3.4449999999999998},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":6.282}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44990464,"data_stored":44990464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47820800,"data_stored":47820800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45883392,"data_stored":45883392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45604864,"data_stored":45604864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:27.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:27.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12554", 'reported_seq': 12655, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:24.574814+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:55:24.574814+0000', 'last_peered': '2024-09-18T11:55:24.574814+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:55:24.574814+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5701632, 'num_objects': 696, 'num_object_clones': 0, 'num_object_copies': 2784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12554, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1407}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:27.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:27.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12110", 'reported_seq': 12227, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:24.285770+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:55:24.285770+0000', 'last_peered': '2024-09-18T11:55:24.285770+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:55:24.285770+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5716627, 'num_objects': 699, 'num_object_clones': 0, 'num_object_copies': 2796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 699, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12109, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1341}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:27.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:27.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12208", 'reported_seq': 12302, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:25.221377+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:25.221377+0000', 'last_peered': '2024-09-18T11:55:25.221377+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:25.221377+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5879030, 'num_objects': 718, 'num_object_clones': 0, 'num_object_copies': 2872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12206, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:27.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12186", 'reported_seq': 12307, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:24.579860+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:55:24.579860+0000', 'last_peered': '2024-09-18T11:55:24.579860+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:55:24.579860+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5805302, 'num_objects': 709, 'num_object_clones': 0, 'num_object_copies': 2836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12184, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1343}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12150", 'reported_seq': 12269, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:24.506197+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:55:24.506197+0000', 'last_peered': '2024-09-18T11:55:24.506197+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:55:24.506197+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5210112, 'num_objects': 636, 'num_object_clones': 0, 'num_object_copies': 2544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12150, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1483}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12353", 'reported_seq': 12460, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:25.182785+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:55:25.182785+0000', 'last_peered': '2024-09-18T11:55:25.182785+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:55:25.182785+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5428470, 'num_objects': 663, 'num_object_clones': 0, 'num_object_copies': 2652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12351, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12197", 'reported_seq': 15775, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:25.348150+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:55:25.348150+0000', 'last_peered': '2024-09-18T11:55:25.348150+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:55:25.348150+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5551350, 'num_objects': 678, 'num_object_clones': 0, 'num_object_copies': 2712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 282, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12195, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1756, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11915", 'reported_seq': 12029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:24.242854+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:55:24.242854+0000', 'last_peered': '2024-09-18T11:55:24.242854+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:55:24.242854+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5559542, 'num_objects': 679, 'num_object_clones': 0, 'num_object_copies': 2716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11913, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1284}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:27.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12289", 'reported_seq': 12410, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:24.577484+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:55:24.577484+0000', 'last_peered': '2024-09-18T11:55:24.577484+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:55:24.577484+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6090609, 'num_objects': 744, 'num_object_clones': 0, 'num_object_copies': 2976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12286, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1290}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:27.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:27.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12384", 'reported_seq': 12476, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:24.557630+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:55:24.557630+0000', 'last_peered': '2024-09-18T11:55:24.557630+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:55:24.557630+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5298811, 'num_objects': 647, 'num_object_clones': 0, 'num_object_copies': 2588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12383, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1437}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:27.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:27.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12267", 'reported_seq': 12365, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:24.093053+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:55:24.093053+0000', 'last_peered': '2024-09-18T11:55:24.093053+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:55:24.093053+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5732987, 'num_objects': 700, 'num_object_clones': 0, 'num_object_copies': 2800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 700, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12266, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:27.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:27.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12337", 'reported_seq': 12462, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:24.512782+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:55:24.512782+0000', 'last_peered': '2024-09-18T11:55:24.512782+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:55:24.512782+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6165750, 'num_objects': 753, 'num_object_clones': 0, 'num_object_copies': 3012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 753, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12335, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1370}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:27.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:27.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12022", 'reported_seq': 12112, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:25.236407+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:55:25.236407+0000', 'last_peered': '2024-09-18T11:55:25.236407+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:55:25.236407+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5608694, 'num_objects': 685, 'num_object_clones': 0, 'num_object_copies': 2740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12020, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1317}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:27.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:28.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:28.459 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25083/44252 objects degraded (56.682%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":726,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11063,"data_bytes":90580364,"bytes_used":3974938624,"bytes_avail":382572118016,"bytes_total":386547056640,"degraded_objects":25083,"degraded_total":44252,"degraded_ratio":0.56682183856096902,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":176},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:28.478 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:28.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:29.041 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:29.042 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:29.067 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-18T11:55:28.087911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":90580364,"num_objects":11063,"num_object_clones":0,"num_object_copies":44252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11063,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196024,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8373,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3881776,"kb_used_data":177928,"kb_used_omap":14,"kb_used_meta":3703793,"kb_avail":373605584,"statfs":{"total":386547056640,"available":382572118016,"internally_reserved":0,"allocated":182198272,"data_stored":181462988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792684059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,10,11,11,9,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17350656,"num_objects":-2118,"num_object_clones":0,"num_object_copies":-8472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2118,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004189"},"pg_stats":[{"pgid":"1.f","version":"37'12554","reported_seq":12655,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.574814+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:24.574814+0000","last_peered":"2024-09-18T11:55:24.574814+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:24.574814+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":696,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12384","reported_seq":16676,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:27.321344+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:27.321344+0000","last_peered":"2024-09-18T11:55:27.321344+0000","last_clean":"2024-09-18T11:55:27.321344+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:27.321344+0000","last_undegraded":"2024-09-18T11:55:27.321344+0000","last_fullsized":"2024-09-18T11:55:27.321344+0000","mapping_epoch":34,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5502198,"num_objects":672,"num_object_clones":0,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12110","reported_seq":12227,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.285770+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:24.285770+0000","last_peered":"2024-09-18T11:55:24.285770+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:24.285770+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5716627,"num_objects":699,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12109,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12208","reported_seq":12302,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.221377+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:25.221377+0000","last_peered":"2024-09-18T11:55:25.221377+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:25.221377+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5879030,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12206,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12186","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.579860+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:24.579860+0000","last_peered":"2024-09-18T11:55:24.579860+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:24.579860+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805302,"num_objects":709,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1343}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12150","reported_seq":12269,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.506197+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:24.506197+0000","last_peered":"2024-09-18T11:55:24.506197+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:24.506197+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210112,"num_objects":636,"num_object_clones":0,"num_object_copies":2544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12353","reported_seq":12460,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.182785+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:25.182785+0000","last_peered":"2024-09-18T11:55:25.182785+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:25.182785+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428470,"num_objects":663,"num_object_clones":0,"num_object_copies":2652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1453}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12197","reported_seq":15775,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.348150+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:25.348150+0000","last_peered":"2024-09-18T11:55:25.348150+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:25.348150+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5551350,"num_objects":678,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12195,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12458","reported_seq":16811,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:24.620191+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:24.620191+0000","last_peered":"2024-09-18T11:55:24.620191+0000","last_clean":"2024-09-18T11:55:24.620191+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:24.620191+0000","last_undegraded":"2024-09-18T11:55:24.620191+0000","last_fullsized":"2024-09-18T11:55:24.620191+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5876204,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12455,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11915","reported_seq":12029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.242854+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:24.242854+0000","last_peered":"2024-09-18T11:55:24.242854+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:24.242854+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5559542,"num_objects":679,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11913,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1284}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12289","reported_seq":12410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.577484+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:24.577484+0000","last_peered":"2024-09-18T11:55:24.577484+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:24.577484+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6090609,"num_objects":744,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":12476,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.557630+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:24.557630+0000","last_peered":"2024-09-18T11:55:24.557630+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:24.557630+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5298811,"num_objects":647,"num_object_clones":0,"num_object_copies":2588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12383,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1437}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12267","reported_seq":12365,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.093053+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:24.093053+0000","last_peered":"2024-09-18T11:55:24.093053+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:24.093053+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5732987,"num_objects":700,"num_object_clones":0,"num_object_copies":2800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12266,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1346}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12337","reported_seq":12462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:24.512782+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:24.512782+0000","last_peered":"2024-09-18T11:55:24.512782+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:24.512782+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6165750,"num_objects":753,"num_object_clones":0,"num_object_copies":3012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12335,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1370}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12237","reported_seq":16526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:26.881701+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:26.881701+0000","last_peered":"2024-09-18T11:55:26.881701+0000","last_clean":"2024-09-18T11:55:26.881701+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:26.881701+0000","last_undegraded":"2024-09-18T11:55:26.881701+0000","last_fullsized":"2024-09-18T11:55:26.881701+0000","mapping_epoch":32,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5453046,"num_objects":666,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12235,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12022","reported_seq":12112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:25.236407+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:25.236407+0000","last_peered":"2024-09-18T11:55:25.236407+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:25.236407+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5608694,"num_objects":685,"num_object_clones":0,"num_object_copies":2740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12020,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1317}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":90580364,"num_objects":11063,"num_object_clones":0,"num_object_copies":44252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11063,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196024,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8373,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180150272,"data_stored":180150272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":977172,"kb_used_data":45324,"kb_used_omap":4,"kb_used_meta":931835,"kb_avail":93394668,"statfs":{"total":96636764160,"available":95636140032,"internally_reserved":0,"allocated":46411776,"data_stored":46227955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954199617},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,5,2,5,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7629999999999999,"5min":7.1660000000000004,"15min":4.7930000000000001},"min":{"1min":0.97399999999999998,"5min":0.186,"15min":0.155},"max":{"1min":21.527999999999999,"5min":23.504000000000001,"15min":23.504000000000001},"last":16.913},{"interface":"front","average":{"1min":6.7679999999999998,"5min":6.4690000000000003,"15min":4.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":21.352,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.54400000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":8.7279999999999998},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":14.680999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:54:22 2024","interfaces":[{"interface":"back","average":{"1min":7.9550000000000001,"5min":7.673,"15min":4.1909999999999998},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":22.190999999999999,"5min":42.752000000000002,"15min":42.752000000000002},"last":14.784000000000001},{"interface":"front","average":{"1min":4.5030000000000001,"5min":4.2480000000000002,"15min":3.75},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":24.974,"5min":24.974,"15min":24.974},"last":4.9409999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968336,"kb_used_data":45316,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93403504,"statfs":{"total":96636764160,"available":95645188096,"internally_reserved":0,"allocated":46403584,"data_stored":46219763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.73499999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.77300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.183},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":7.1920000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:54:25 2024","interfaces":[{"interface":"back","average":{"1min":9.4280000000000008,"5min":5.7160000000000002,"15min":3.9609999999999999},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.73,"5min":22.73,"15min":22.73},"last":22.047000000000001},{"interface":"front","average":{"1min":6.9429999999999996,"5min":5.5650000000000004,"15min":3.3889999999999998},"min":{"1min":0.30199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":20.460000000000001,"5min":20.728999999999999,"15min":20.728999999999999},"last":1.728}]}]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968488,"kb_used_data":42976,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93403352,"statfs":{"total":96636764160,"available":95645032448,"internally_reserved":0,"allocated":44007424,"data_stored":43823603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,2,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":7.9370000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":2.5859999999999999},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":6.4530000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":6.758},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.3959999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967780,"kb_used_data":44312,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93404060,"statfs":{"total":96636764160,"available":95645757440,"internally_reserved":0,"allocated":45375488,"data_stored":45191667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,5,7,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":4.7720000000000002},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":2.3580000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.7709999999999999},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":2.3250000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":3.4449999999999998},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":6.282}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44990464,"data_stored":44990464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43671552,"data_stored":43671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45883392,"data_stored":45883392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45604864,"data_stored":45604864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:32.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:32.647 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:32.648 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:32.681 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-18T11:55:32.092158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83297676,"num_objects":10174,"num_object_clones":0,"num_object_copies":40696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196913,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8424,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3865476,"kb_used_data":165796,"kb_used_omap":14,"kb_used_meta":3699633,"kb_avail":373621884,"statfs":{"total":386547056640,"available":382588809216,"internally_reserved":0,"allocated":169775104,"data_stored":169039820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788424219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,6,4,5,10,10,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18759680,"num_objects":-2290,"num_object_clones":0,"num_object_copies":-9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2290,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006877"},"pg_stats":[{"pgid":"1.f","version":"37'12617","reported_seq":12715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.662418+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:29.662418+0000","last_peered":"2024-09-18T11:55:29.662418+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:29.662418+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5185536,"num_objects":633,"num_object_clones":0,"num_object_copies":2532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12617,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12384","reported_seq":16676,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:27.321344+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:27.321344+0000","last_peered":"2024-09-18T11:55:27.321344+0000","last_clean":"2024-09-18T11:55:27.321344+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:27.321344+0000","last_undegraded":"2024-09-18T11:55:27.321344+0000","last_fullsized":"2024-09-18T11:55:27.321344+0000","mapping_epoch":34,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5502198,"num_objects":672,"num_object_clones":0,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12182","reported_seq":12299,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.363811+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:29.363811+0000","last_peered":"2024-09-18T11:55:29.363811+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:29.363811+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126803,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12181,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12267","reported_seq":12361,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.312394+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:30.312394+0000","last_peered":"2024-09-18T11:55:30.312394+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:30.312394+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395702,"num_objects":659,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12265,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12248","reported_seq":12369,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.404978+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:29.404978+0000","last_peered":"2024-09-18T11:55:29.404978+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:29.404978+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5297398,"num_objects":647,"num_object_clones":0,"num_object_copies":2588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12246,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1405}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12207","reported_seq":12326,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.385766+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:29.385766+0000","last_peered":"2024-09-18T11:55:29.385766+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:29.385766+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4743168,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12409","reported_seq":12516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.369178+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:30.369178+0000","last_peered":"2024-09-18T11:55:30.369178+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:30.369178+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4969718,"num_objects":607,"num_object_clones":0,"num_object_copies":2428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12407,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1509}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12251","reported_seq":15931,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.393413+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:30.393413+0000","last_peered":"2024-09-18T11:55:30.393413+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:30.393413+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5108982,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12526","reported_seq":16879,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:29.507928+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:29.507928+0000","last_peered":"2024-09-18T11:55:29.507928+0000","last_clean":"2024-09-18T11:55:29.507928+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:29.507928+0000","last_undegraded":"2024-09-18T11:55:29.507928+0000","last_fullsized":"2024-09-18T11:55:29.507928+0000","mapping_epoch":36,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5319148,"num_objects":650,"num_object_clones":0,"num_object_copies":2600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12523,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11980","reported_seq":12093,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.348507+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:29.348507+0000","last_peered":"2024-09-18T11:55:29.348507+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:29.348507+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5027062,"num_objects":614,"num_object_clones":0,"num_object_copies":2456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12354","reported_seq":12474,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.637218+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:29.637218+0000","last_peered":"2024-09-18T11:55:29.637218+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:29.637218+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5558129,"num_objects":679,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12446","reported_seq":12539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.380069+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:29.380069+0000","last_peered":"2024-09-18T11:55:29.380069+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:29.380069+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4790907,"num_objects":585,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1499}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12327","reported_seq":12425,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.054330+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:29.054330+0000","last_peered":"2024-09-18T11:55:29.054330+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:29.054330+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5241467,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12326,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1406}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12407","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.543004+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:29.543004+0000","last_peered":"2024-09-18T11:55:29.543004+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:29.543004+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5592310,"num_objects":683,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12237","reported_seq":16526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:26.881701+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:26.881701+0000","last_peered":"2024-09-18T11:55:26.881701+0000","last_clean":"2024-09-18T11:55:26.881701+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:26.881701+0000","last_undegraded":"2024-09-18T11:55:26.881701+0000","last_fullsized":"2024-09-18T11:55:26.881701+0000","mapping_epoch":32,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5453046,"num_objects":666,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12235,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12098","reported_seq":12188,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.096876+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:30.096876+0000","last_peered":"2024-09-18T11:55:30.096876+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:30.096876+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4986102,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12096,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":83297676,"num_objects":10174,"num_object_clones":0,"num_object_copies":40696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196913,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8424,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167669760,"data_stored":167669760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":960732,"kb_used_data":41364,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93411108,"statfs":{"total":96636764160,"available":95652974592,"internally_reserved":0,"allocated":42356736,"data_stored":42172915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,5,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.1719999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.139},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.1349999999999998},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":2.1190000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968416,"kb_used_data":41240,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93403424,"statfs":{"total":96636764160,"available":95645106176,"internally_reserved":0,"allocated":42229760,"data_stored":42045939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,0,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":33.225000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":33.259}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":9.3859999999999992},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":15.291}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":5.2060000000000004},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":2.8340000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968488,"kb_used_data":42976,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93403352,"statfs":{"total":96636764160,"available":95645032448,"internally_reserved":0,"allocated":44007424,"data_stored":43823603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,2,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":7.9370000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":2.5859999999999999},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":6.4530000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":6.758},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.3959999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967840,"kb_used_data":40216,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93404000,"statfs":{"total":96636764160,"available":95645696000,"internally_reserved":0,"allocated":41181184,"data_stored":40997363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.1689999999999996},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":9.9640000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.7210000000000001},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":9.9990000000000006}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":8.609},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.021000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40828928,"data_stored":40828928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43671552,"data_stored":43671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41734144,"data_stored":41734144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41435136,"data_stored":41435136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:32.683 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:33.274 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:33.275 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:33.307 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-18T11:55:32.092158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83297676,"num_objects":10174,"num_object_clones":0,"num_object_copies":40696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196913,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8424,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3865476,"kb_used_data":165796,"kb_used_omap":14,"kb_used_meta":3699633,"kb_avail":373621884,"statfs":{"total":386547056640,"available":382588809216,"internally_reserved":0,"allocated":169775104,"data_stored":169039820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788424219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,6,4,5,10,10,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18759680,"num_objects":-2290,"num_object_clones":0,"num_object_copies":-9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2290,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006877"},"pg_stats":[{"pgid":"1.f","version":"37'12617","reported_seq":12715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.662418+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:29.662418+0000","last_peered":"2024-09-18T11:55:29.662418+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:29.662418+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5185536,"num_objects":633,"num_object_clones":0,"num_object_copies":2532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12617,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12384","reported_seq":16676,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:27.321344+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:27.321344+0000","last_peered":"2024-09-18T11:55:27.321344+0000","last_clean":"2024-09-18T11:55:27.321344+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:27.321344+0000","last_undegraded":"2024-09-18T11:55:27.321344+0000","last_fullsized":"2024-09-18T11:55:27.321344+0000","mapping_epoch":34,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5502198,"num_objects":672,"num_object_clones":0,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12182","reported_seq":12299,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.363811+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:29.363811+0000","last_peered":"2024-09-18T11:55:29.363811+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:29.363811+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126803,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12181,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12267","reported_seq":12361,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.312394+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:30.312394+0000","last_peered":"2024-09-18T11:55:30.312394+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:30.312394+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395702,"num_objects":659,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12265,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12248","reported_seq":12369,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.404978+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:29.404978+0000","last_peered":"2024-09-18T11:55:29.404978+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:29.404978+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5297398,"num_objects":647,"num_object_clones":0,"num_object_copies":2588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12246,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1405}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12207","reported_seq":12326,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.385766+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:29.385766+0000","last_peered":"2024-09-18T11:55:29.385766+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:29.385766+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4743168,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12409","reported_seq":12516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.369178+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:30.369178+0000","last_peered":"2024-09-18T11:55:30.369178+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:30.369178+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4969718,"num_objects":607,"num_object_clones":0,"num_object_copies":2428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12407,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1509}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12251","reported_seq":15931,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.393413+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:30.393413+0000","last_peered":"2024-09-18T11:55:30.393413+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:30.393413+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5108982,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12526","reported_seq":16879,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:29.507928+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:29.507928+0000","last_peered":"2024-09-18T11:55:29.507928+0000","last_clean":"2024-09-18T11:55:29.507928+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:29.507928+0000","last_undegraded":"2024-09-18T11:55:29.507928+0000","last_fullsized":"2024-09-18T11:55:29.507928+0000","mapping_epoch":36,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5319148,"num_objects":650,"num_object_clones":0,"num_object_copies":2600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12523,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11980","reported_seq":12093,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.348507+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:29.348507+0000","last_peered":"2024-09-18T11:55:29.348507+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:29.348507+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5027062,"num_objects":614,"num_object_clones":0,"num_object_copies":2456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12354","reported_seq":12474,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.637218+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:29.637218+0000","last_peered":"2024-09-18T11:55:29.637218+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:29.637218+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5558129,"num_objects":679,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12446","reported_seq":12539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.380069+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:29.380069+0000","last_peered":"2024-09-18T11:55:29.380069+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:29.380069+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4790907,"num_objects":585,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1499}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12327","reported_seq":12425,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.054330+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:29.054330+0000","last_peered":"2024-09-18T11:55:29.054330+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:29.054330+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5241467,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12326,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1406}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12407","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.543004+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:29.543004+0000","last_peered":"2024-09-18T11:55:29.543004+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:29.543004+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5592310,"num_objects":683,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12237","reported_seq":16526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:26.881701+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:26.881701+0000","last_peered":"2024-09-18T11:55:26.881701+0000","last_clean":"2024-09-18T11:55:26.881701+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:26.881701+0000","last_undegraded":"2024-09-18T11:55:26.881701+0000","last_fullsized":"2024-09-18T11:55:26.881701+0000","mapping_epoch":32,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5453046,"num_objects":666,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12235,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12098","reported_seq":12188,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.096876+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:30.096876+0000","last_peered":"2024-09-18T11:55:30.096876+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:30.096876+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4986102,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12096,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":83297676,"num_objects":10174,"num_object_clones":0,"num_object_copies":40696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196913,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8424,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167669760,"data_stored":167669760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":960732,"kb_used_data":41364,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93411108,"statfs":{"total":96636764160,"available":95652974592,"internally_reserved":0,"allocated":42356736,"data_stored":42172915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,5,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.1719999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.139},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.1349999999999998},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":2.1190000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968416,"kb_used_data":41240,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93403424,"statfs":{"total":96636764160,"available":95645106176,"internally_reserved":0,"allocated":42229760,"data_stored":42045939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,0,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":33.225000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":33.259}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":9.3859999999999992},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":15.291}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":5.2060000000000004},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":2.8340000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968488,"kb_used_data":42976,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93403352,"statfs":{"total":96636764160,"available":95645032448,"internally_reserved":0,"allocated":44007424,"data_stored":43823603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,2,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":7.9370000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":2.5859999999999999},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":6.4530000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":6.758},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.3959999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967840,"kb_used_data":40216,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93404000,"statfs":{"total":96636764160,"available":95645696000,"internally_reserved":0,"allocated":41181184,"data_stored":40997363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.1689999999999996},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":9.9640000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.7210000000000001},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":9.9990000000000006}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":8.609},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.021000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40828928,"data_stored":40828928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43671552,"data_stored":43671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41734144,"data_stored":41734144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41435136,"data_stored":41435136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:33.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:33.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12617", 'reported_seq': 12715, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:29.662418+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:55:29.662418+0000', 'last_peered': '2024-09-18T11:55:29.662418+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:55:29.662418+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5185536, 'num_objects': 633, 'num_object_clones': 0, 'num_object_copies': 2532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12617, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1470}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:33.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:33.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12182", 'reported_seq': 12299, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:29.363811+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:55:29.363811+0000', 'last_peered': '2024-09-18T11:55:29.363811+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:55:29.363811+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5126803, 'num_objects': 627, 'num_object_clones': 0, 'num_object_copies': 2508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 627, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12181, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1413}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:33.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:33.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12267", 'reported_seq': 12361, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:30.312394+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:30.312394+0000', 'last_peered': '2024-09-18T11:55:30.312394+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:30.312394+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5395702, 'num_objects': 659, 'num_object_clones': 0, 'num_object_copies': 2636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12265, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1442}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:33.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:33.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12248", 'reported_seq': 12369, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:29.404978+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:55:29.404978+0000', 'last_peered': '2024-09-18T11:55:29.404978+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:55:29.404978+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5297398, 'num_objects': 647, 'num_object_clones': 0, 'num_object_copies': 2588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12246, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1405}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:33.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:33.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12207", 'reported_seq': 12326, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:29.385766+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:55:29.385766+0000', 'last_peered': '2024-09-18T11:55:29.385766+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:55:29.385766+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4743168, 'num_objects': 579, 'num_object_clones': 0, 'num_object_copies': 2316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12207, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1540}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:33.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:33.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12409", 'reported_seq': 12516, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:30.369178+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:55:30.369178+0000', 'last_peered': '2024-09-18T11:55:30.369178+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:55:30.369178+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4969718, 'num_objects': 607, 'num_object_clones': 0, 'num_object_copies': 2428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12407, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1509}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:33.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:33.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12251", 'reported_seq': 15931, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:30.393413+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:55:30.393413+0000', 'last_peered': '2024-09-18T11:55:30.393413+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:55:30.393413+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5108982, 'num_objects': 624, 'num_object_clones': 0, 'num_object_copies': 2496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12249, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1807, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 393}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:33.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:33.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11980", 'reported_seq': 12093, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:29.348507+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:55:29.348507+0000', 'last_peered': '2024-09-18T11:55:29.348507+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:55:29.348507+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5027062, 'num_objects': 614, 'num_object_clones': 0, 'num_object_copies': 2456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11978, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1349}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:33.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:33.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12354", 'reported_seq': 12474, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:29.637218+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:55:29.637218+0000', 'last_peered': '2024-09-18T11:55:29.637218+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:55:29.637218+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5558129, 'num_objects': 679, 'num_object_clones': 0, 'num_object_copies': 2716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12351, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1355}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:33.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:33.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12446", 'reported_seq': 12539, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:29.380069+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:55:29.380069+0000', 'last_peered': '2024-09-18T11:55:29.380069+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:55:29.380069+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4790907, 'num_objects': 585, 'num_object_clones': 0, 'num_object_copies': 2340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1499}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:33.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:33.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12327", 'reported_seq': 12425, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:29.054330+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:55:29.054330+0000', 'last_peered': '2024-09-18T11:55:29.054330+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:55:29.054330+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5241467, 'num_objects': 640, 'num_object_clones': 0, 'num_object_copies': 2560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12326, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1406}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:33.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:33.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12407", 'reported_seq': 12532, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:29.543004+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:55:29.543004+0000', 'last_peered': '2024-09-18T11:55:29.543004+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:55:29.543004+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5592310, 'num_objects': 683, 'num_object_clones': 0, 'num_object_copies': 2732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1440}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:33.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:33.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12098", 'reported_seq': 12188, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:30.096876+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:55:30.096876+0000', 'last_peered': '2024-09-18T11:55:30.096876+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:55:30.096876+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4986102, 'num_objects': 609, 'num_object_clones': 0, 'num_object_copies': 2436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12096, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1393}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:33.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:33.987 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25032/40696 objects degraded (61.510%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":732,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10174,"data_bytes":83297676,"bytes_used":3958247424,"bytes_avail":382588809216,"bytes_total":386547056640,"degraded_objects":25032,"degraded_total":40696,"degraded_ratio":0.61509730686062514,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":190},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:34.016 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:34.016 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:34.586 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:34.603 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-18T11:55:34.092613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":82191756,"num_objects":10039,"num_object_clones":0,"num_object_copies":40156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10039,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197048,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8424,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160675,"ondisk_log_size":160675,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3865588,"kb_used_data":161748,"kb_used_omap":14,"kb_used_meta":3703793,"kb_avail":373621772,"statfs":{"total":386547056640,"available":382588694528,"internally_reserved":0,"allocated":165629952,"data_stored":164894668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792684059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,6,4,6,9,13,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17637376,"num_objects":-2153,"num_object_clones":0,"num_object_copies":-8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2153,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006661"},"pg_stats":[{"pgid":"1.f","version":"37'12617","reported_seq":12715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.662418+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:29.662418+0000","last_peered":"2024-09-18T11:55:29.662418+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:29.662418+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5185536,"num_objects":633,"num_object_clones":0,"num_object_copies":2532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12617,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12453","reported_seq":16744,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:32.477893+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:32.477893+0000","last_peered":"2024-09-18T11:55:32.477893+0000","last_clean":"2024-09-18T11:55:32.477893+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:32.477893+0000","last_undegraded":"2024-09-18T11:55:32.477893+0000","last_fullsized":"2024-09-18T11:55:32.477893+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4936950,"num_objects":603,"num_object_clones":0,"num_object_copies":2412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12451,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12182","reported_seq":12299,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.363811+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:29.363811+0000","last_peered":"2024-09-18T11:55:29.363811+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:29.363811+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126803,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12181,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12267","reported_seq":12361,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.312394+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:30.312394+0000","last_peered":"2024-09-18T11:55:30.312394+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:30.312394+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395702,"num_objects":659,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12265,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12248","reported_seq":12369,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.404978+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:29.404978+0000","last_peered":"2024-09-18T11:55:29.404978+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:29.404978+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5297398,"num_objects":647,"num_object_clones":0,"num_object_copies":2588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12246,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1405}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12207","reported_seq":12326,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.385766+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:29.385766+0000","last_peered":"2024-09-18T11:55:29.385766+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:29.385766+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4743168,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12409","reported_seq":12516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.369178+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:30.369178+0000","last_peered":"2024-09-18T11:55:30.369178+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:30.369178+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4969718,"num_objects":607,"num_object_clones":0,"num_object_copies":2428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12407,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1509}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12251","reported_seq":15931,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.393413+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:30.393413+0000","last_peered":"2024-09-18T11:55:30.393413+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:30.393413+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5108982,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12526","reported_seq":16879,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:29.507928+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:29.507928+0000","last_peered":"2024-09-18T11:55:29.507928+0000","last_clean":"2024-09-18T11:55:29.507928+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:29.507928+0000","last_undegraded":"2024-09-18T11:55:29.507928+0000","last_fullsized":"2024-09-18T11:55:29.507928+0000","mapping_epoch":36,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5319148,"num_objects":650,"num_object_clones":0,"num_object_copies":2600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12523,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11980","reported_seq":12093,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.348507+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:29.348507+0000","last_peered":"2024-09-18T11:55:29.348507+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:29.348507+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5027062,"num_objects":614,"num_object_clones":0,"num_object_copies":2456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12354","reported_seq":12474,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.637218+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:29.637218+0000","last_peered":"2024-09-18T11:55:29.637218+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:29.637218+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5558129,"num_objects":679,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12446","reported_seq":12539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.380069+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:29.380069+0000","last_peered":"2024-09-18T11:55:29.380069+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:29.380069+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4790907,"num_objects":585,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1499}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12327","reported_seq":12425,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.054330+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:29.054330+0000","last_peered":"2024-09-18T11:55:29.054330+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:29.054330+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5241467,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12326,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1406}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12407","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:29.543004+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:29.543004+0000","last_peered":"2024-09-18T11:55:29.543004+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:29.543004+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5592310,"num_objects":683,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12303","reported_seq":16591,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:32.441687+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:32.441687+0000","last_peered":"2024-09-18T11:55:32.441687+0000","last_clean":"2024-09-18T11:55:32.441687+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:32.441687+0000","last_undegraded":"2024-09-18T11:55:32.441687+0000","last_fullsized":"2024-09-18T11:55:32.441687+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4912374,"num_objects":600,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12301,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12098","reported_seq":12188,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:30.096876+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:30.096876+0000","last_peered":"2024-09-18T11:55:30.096876+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:30.096876+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4986102,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12096,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":82191756,"num_objects":10039,"num_object_clones":0,"num_object_copies":40156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10039,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197048,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8424,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163422208,"data_stored":163422208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160675,"ondisk_log_size":160675,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":960732,"kb_used_data":41364,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93411108,"statfs":{"total":96636764160,"available":95652974592,"internally_reserved":0,"allocated":42356736,"data_stored":42172915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,5,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.1719999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.139},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":2.1349999999999998},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":2.1190000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968416,"kb_used_data":41240,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93403424,"statfs":{"total":96636764160,"available":95645106176,"internally_reserved":0,"allocated":42229760,"data_stored":42045939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,0,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":33.225000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":33.259}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":9.3859999999999992},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":15.291}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":5.2060000000000004},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":2.8340000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":968600,"kb_used_data":38928,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93403240,"statfs":{"total":96636764160,"available":95644917760,"internally_reserved":0,"allocated":39862272,"data_stored":39678451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,0,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":4.8360000000000003},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":5.0460000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":5.6820000000000004},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.8710000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":5.0190000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":13.01}]}]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967840,"kb_used_data":40216,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93404000,"statfs":{"total":96636764160,"available":95645696000,"internally_reserved":0,"allocated":41181184,"data_stored":40997363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.1689999999999996},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":9.9640000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":7.7210000000000001},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":9.9990000000000006}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":8.609},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.021000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40828928,"data_stored":40828928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39424000,"data_stored":39424000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41734144,"data_stored":41734144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41435136,"data_stored":41435136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:37.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:38.210 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:38.211 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:38.236 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-18T11:55:38.097303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":73844108,"num_objects":9020,"num_object_clones":0,"num_object_copies":36080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9020,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198067,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8471,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160794,"ondisk_log_size":160794,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3845188,"kb_used_data":145484,"kb_used_omap":14,"kb_used_meta":3699633,"kb_avail":373642172,"statfs":{"total":386547056640,"available":382609584128,"internally_reserved":0,"allocated":148975616,"data_stored":148240332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788424219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,6,9,13,6,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17653760,"num_objects":-2155,"num_object_clones":0,"num_object_copies":-8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2155,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010023"},"pg_stats":[{"pgid":"1.f","version":"37'12677","reported_seq":12778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.380246+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:34.380246+0000","last_peered":"2024-09-18T11:55:34.380246+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:34.380246+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4694016,"num_objects":573,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12523","reported_seq":16815,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:37.311187+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:37.311187+0000","last_peered":"2024-09-18T11:55:37.311187+0000","last_clean":"2024-09-18T11:55:37.311187+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:37.311187+0000","last_undegraded":"2024-09-18T11:55:37.311187+0000","last_fullsized":"2024-09-18T11:55:37.311187+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4363510,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12243","reported_seq":12360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.072665+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:34.072665+0000","last_peered":"2024-09-18T11:55:34.072665+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:34.072665+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4627091,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12242,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1474}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12325","reported_seq":12419,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.411850+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:35.411850+0000","last_peered":"2024-09-18T11:55:35.411850+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:35.411850+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4920566,"num_objects":601,"num_object_clones":0,"num_object_copies":2404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12313","reported_seq":12434,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.709696+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:34.709696+0000","last_peered":"2024-09-18T11:55:34.709696+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:34.709696+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4764918,"num_objects":582,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12311,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12264","reported_seq":12383,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.740446+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:34.740446+0000","last_peered":"2024-09-18T11:55:34.740446+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:34.740446+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4276224,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12264,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12462","reported_seq":12569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.212973+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:35.212973+0000","last_peered":"2024-09-18T11:55:35.212973+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:35.212973+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4535542,"num_objects":554,"num_object_clones":0,"num_object_copies":2216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12460,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1562}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12312","reported_seq":16084,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.381419+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:35.381419+0000","last_peered":"2024-09-18T11:55:35.381419+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:35.381419+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4609270,"num_objects":563,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12310,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1854,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12583","reported_seq":16936,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:34.588713+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:34.588713+0000","last_peered":"2024-09-18T11:55:34.588713+0000","last_clean":"2024-09-18T11:55:34.588713+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:34.588713+0000","last_undegraded":"2024-09-18T11:55:34.588713+0000","last_fullsized":"2024-09-18T11:55:34.588713+0000","mapping_epoch":36,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4852204,"num_objects":593,"num_object_clones":0,"num_object_copies":2372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12053","reported_seq":12167,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.208013+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:34.208013+0000","last_peered":"2024-09-18T11:55:34.208013+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:34.208013+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2000","ondisk_log_start":"31'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4429046,"num_objects":541,"num_object_clones":0,"num_object_copies":2164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12051,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1422}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12417","reported_seq":12538,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.653265+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:34.653265+0000","last_peered":"2024-09-18T11:55:34.653265+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:34.653265+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5042033,"num_objects":616,"num_object_clones":0,"num_object_copies":2464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12414,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12506","reported_seq":12599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.499432+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:34.499432+0000","last_peered":"2024-09-18T11:55:34.499432+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:34.499432+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4299387,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12505,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12390","reported_seq":12488,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.060544+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:34.060544+0000","last_peered":"2024-09-18T11:55:34.060544+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:34.060544+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4725371,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12389,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1469}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12486","reported_seq":12611,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.647312+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:34.647312+0000","last_peered":"2024-09-18T11:55:34.647312+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:34.647312+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4945142,"num_objects":604,"num_object_clones":0,"num_object_copies":2416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12484,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1519}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12364","reported_seq":16653,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:37.484724+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:37.484724+0000","last_peered":"2024-09-18T11:55:37.484724+0000","last_clean":"2024-09-18T11:55:37.484724+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:37.484724+0000","last_undegraded":"2024-09-18T11:55:37.484724+0000","last_fullsized":"2024-09-18T11:55:37.484724+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4412662,"num_objects":539,"num_object_clones":0,"num_object_copies":2156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12362,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12176","reported_seq":12266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.311822+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:35.311822+0000","last_peered":"2024-09-18T11:55:35.311822+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:35.311822+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4347126,"num_objects":531,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12174,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":73844108,"num_objects":9020,"num_object_clones":0,"num_object_copies":36080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9020,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198067,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8471,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":146825216,"data_stored":146825216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160794,"ondisk_log_size":160794,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":960796,"kb_used_data":37268,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93411044,"statfs":{"total":96636764160,"available":95652909056,"internally_reserved":0,"allocated":38162432,"data_stored":37978611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":21.800999999999998},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":21.968}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.9699999999999998},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.23200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951848,"kb_used_data":37152,"kb_used_omap":1,"kb_used_meta":914686,"kb_avail":93419992,"statfs":{"total":96636764160,"available":95662071808,"internally_reserved":0,"allocated":38043648,"data_stored":37859827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936638808},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":1.988},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":2.02}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.603},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.25800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":6.6319999999999997},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":13.493}]}]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":964592,"kb_used_data":34900,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93407248,"statfs":{"total":96636764160,"available":95649021952,"internally_reserved":0,"allocated":35737600,"data_stored":35553779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7,5,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.70099999999999996},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":5.0780000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":3.7210000000000001},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.5129999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":7.5270000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.117}]}]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967952,"kb_used_data":36164,"kb_used_omap":4,"kb_used_meta":931771,"kb_avail":93403888,"statfs":{"total":96636764160,"available":95645581312,"internally_reserved":0,"allocated":37031936,"data_stored":36848115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954134081},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":11.972},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":7.0659999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":13.397},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":7.0330000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":7.0910000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36704256,"data_stored":36704256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35217408,"data_stored":35217408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37609472,"data_stored":37609472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37294080,"data_stored":37294080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:38.237 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:39.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:39.459 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:39.479 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-18T11:55:38.097303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":73844108,"num_objects":9020,"num_object_clones":0,"num_object_copies":36080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9020,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198067,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8471,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160794,"ondisk_log_size":160794,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3845188,"kb_used_data":145484,"kb_used_omap":14,"kb_used_meta":3699633,"kb_avail":373642172,"statfs":{"total":386547056640,"available":382609584128,"internally_reserved":0,"allocated":148975616,"data_stored":148240332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788424219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,6,9,13,6,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17653760,"num_objects":-2155,"num_object_clones":0,"num_object_copies":-8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2155,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010023"},"pg_stats":[{"pgid":"1.f","version":"37'12677","reported_seq":12778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.380246+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:34.380246+0000","last_peered":"2024-09-18T11:55:34.380246+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:34.380246+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4694016,"num_objects":573,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12523","reported_seq":16815,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:37.311187+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:37.311187+0000","last_peered":"2024-09-18T11:55:37.311187+0000","last_clean":"2024-09-18T11:55:37.311187+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:37.311187+0000","last_undegraded":"2024-09-18T11:55:37.311187+0000","last_fullsized":"2024-09-18T11:55:37.311187+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4363510,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12243","reported_seq":12360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.072665+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:34.072665+0000","last_peered":"2024-09-18T11:55:34.072665+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:34.072665+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4627091,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12242,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1474}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12325","reported_seq":12419,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.411850+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:35.411850+0000","last_peered":"2024-09-18T11:55:35.411850+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:35.411850+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4920566,"num_objects":601,"num_object_clones":0,"num_object_copies":2404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12313","reported_seq":12434,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.709696+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:34.709696+0000","last_peered":"2024-09-18T11:55:34.709696+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:34.709696+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4764918,"num_objects":582,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12311,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12264","reported_seq":12383,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.740446+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:34.740446+0000","last_peered":"2024-09-18T11:55:34.740446+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:34.740446+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4276224,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12264,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12462","reported_seq":12569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.212973+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:35.212973+0000","last_peered":"2024-09-18T11:55:35.212973+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:35.212973+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4535542,"num_objects":554,"num_object_clones":0,"num_object_copies":2216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12460,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1562}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12312","reported_seq":16084,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.381419+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:35.381419+0000","last_peered":"2024-09-18T11:55:35.381419+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:35.381419+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4609270,"num_objects":563,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12310,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1854,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12583","reported_seq":16936,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:34.588713+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:34.588713+0000","last_peered":"2024-09-18T11:55:34.588713+0000","last_clean":"2024-09-18T11:55:34.588713+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:34.588713+0000","last_undegraded":"2024-09-18T11:55:34.588713+0000","last_fullsized":"2024-09-18T11:55:34.588713+0000","mapping_epoch":36,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4852204,"num_objects":593,"num_object_clones":0,"num_object_copies":2372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12053","reported_seq":12167,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.208013+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:34.208013+0000","last_peered":"2024-09-18T11:55:34.208013+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:34.208013+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2000","ondisk_log_start":"31'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4429046,"num_objects":541,"num_object_clones":0,"num_object_copies":2164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12051,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1422}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12417","reported_seq":12538,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.653265+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:34.653265+0000","last_peered":"2024-09-18T11:55:34.653265+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:34.653265+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5042033,"num_objects":616,"num_object_clones":0,"num_object_copies":2464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12414,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12506","reported_seq":12599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.499432+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:34.499432+0000","last_peered":"2024-09-18T11:55:34.499432+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:34.499432+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4299387,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12505,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12390","reported_seq":12488,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.060544+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:34.060544+0000","last_peered":"2024-09-18T11:55:34.060544+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:34.060544+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4725371,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12389,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1469}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12486","reported_seq":12611,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:34.647312+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:34.647312+0000","last_peered":"2024-09-18T11:55:34.647312+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:34.647312+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4945142,"num_objects":604,"num_object_clones":0,"num_object_copies":2416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12484,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1519}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12364","reported_seq":16653,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:37.484724+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:37.484724+0000","last_peered":"2024-09-18T11:55:37.484724+0000","last_clean":"2024-09-18T11:55:37.484724+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:37.484724+0000","last_undegraded":"2024-09-18T11:55:37.484724+0000","last_fullsized":"2024-09-18T11:55:37.484724+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4412662,"num_objects":539,"num_object_clones":0,"num_object_copies":2156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12362,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12176","reported_seq":12266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.311822+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:35.311822+0000","last_peered":"2024-09-18T11:55:35.311822+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:35.311822+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4347126,"num_objects":531,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12174,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":73844108,"num_objects":9020,"num_object_clones":0,"num_object_copies":36080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9020,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198067,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8471,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":146825216,"data_stored":146825216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160794,"ondisk_log_size":160794,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":960796,"kb_used_data":37268,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93411044,"statfs":{"total":96636764160,"available":95652909056,"internally_reserved":0,"allocated":38162432,"data_stored":37978611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":21.800999999999998},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":21.968}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.9699999999999998},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.23200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951848,"kb_used_data":37152,"kb_used_omap":1,"kb_used_meta":914686,"kb_avail":93419992,"statfs":{"total":96636764160,"available":95662071808,"internally_reserved":0,"allocated":38043648,"data_stored":37859827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936638808},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":1.988},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":2.02}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.603},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.25800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":6.6319999999999997},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":13.493}]}]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":964592,"kb_used_data":34900,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93407248,"statfs":{"total":96636764160,"available":95649021952,"internally_reserved":0,"allocated":35737600,"data_stored":35553779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7,5,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.70099999999999996},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":5.0780000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":3.7210000000000001},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.5129999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":7.5270000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.117}]}]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967952,"kb_used_data":36164,"kb_used_omap":4,"kb_used_meta":931771,"kb_avail":93403888,"statfs":{"total":96636764160,"available":95645581312,"internally_reserved":0,"allocated":37031936,"data_stored":36848115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954134081},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":11.972},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":7.0659999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":13.397},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":7.0330000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":7.0910000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36704256,"data_stored":36704256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35217408,"data_stored":35217408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37609472,"data_stored":37609472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37294080,"data_stored":37294080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:39.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:39.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12677", 'reported_seq': 12778, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:34.380246+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:55:34.380246+0000', 'last_peered': '2024-09-18T11:55:34.380246+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:55:34.380246+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4694016, 'num_objects': 573, 'num_object_clones': 0, 'num_object_copies': 2292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1530}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:39.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:39.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12243", 'reported_seq': 12360, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:34.072665+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:55:34.072665+0000', 'last_peered': '2024-09-18T11:55:34.072665+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:55:34.072665+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4627091, 'num_objects': 566, 'num_object_clones': 0, 'num_object_copies': 2264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 566, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12242, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1474}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:39.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:39.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12325", 'reported_seq': 12419, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:35.411850+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:35.411850+0000', 'last_peered': '2024-09-18T11:55:35.411850+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:35.411850+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4920566, 'num_objects': 601, 'num_object_clones': 0, 'num_object_copies': 2404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12323, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1500}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:39.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:39.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12313", 'reported_seq': 12434, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:34.709696+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:55:34.709696+0000', 'last_peered': '2024-09-18T11:55:34.709696+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:55:34.709696+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4764918, 'num_objects': 582, 'num_object_clones': 0, 'num_object_copies': 2328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12311, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1470}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:39.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:39.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12264", 'reported_seq': 12383, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:34.740446+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:55:34.740446+0000', 'last_peered': '2024-09-18T11:55:34.740446+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:55:34.740446+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4276224, 'num_objects': 522, 'num_object_clones': 0, 'num_object_copies': 2088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12264, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1597}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:39.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:39.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12462", 'reported_seq': 12569, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:35.212973+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:55:35.212973+0000', 'last_peered': '2024-09-18T11:55:35.212973+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:55:35.212973+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4535542, 'num_objects': 554, 'num_object_clones': 0, 'num_object_copies': 2216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12460, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1562}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:39.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:39.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12312", 'reported_seq': 16084, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:35.381419+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:55:35.381419+0000', 'last_peered': '2024-09-18T11:55:35.381419+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:55:35.381419+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4609270, 'num_objects': 563, 'num_object_clones': 0, 'num_object_copies': 2252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 184, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12310, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1854, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:39.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12053", 'reported_seq': 12167, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:34.208013+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:55:34.208013+0000', 'last_peered': '2024-09-18T11:55:34.208013+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:55:34.208013+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2000", 'ondisk_log_start': "31'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4429046, 'num_objects': 541, 'num_object_clones': 0, 'num_object_copies': 2164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12051, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1422}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12417", 'reported_seq': 12538, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:34.653265+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:55:34.653265+0000', 'last_peered': '2024-09-18T11:55:34.653265+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:55:34.653265+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5042033, 'num_objects': 616, 'num_object_clones': 0, 'num_object_copies': 2464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12414, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1418}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12506", 'reported_seq': 12599, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:34.499432+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:55:34.499432+0000', 'last_peered': '2024-09-18T11:55:34.499432+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:55:34.499432+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4299387, 'num_objects': 525, 'num_object_clones': 0, 'num_object_copies': 2100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12505, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12390", 'reported_seq': 12488, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:34.060544+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:55:34.060544+0000', 'last_peered': '2024-09-18T11:55:34.060544+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:55:34.060544+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4725371, 'num_objects': 577, 'num_object_clones': 0, 'num_object_copies': 2308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12389, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1469}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12486", 'reported_seq': 12611, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:34.647312+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:55:34.647312+0000', 'last_peered': '2024-09-18T11:55:34.647312+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:55:34.647312+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4945142, 'num_objects': 604, 'num_object_clones': 0, 'num_object_copies': 2416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12484, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:39.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12176", 'reported_seq': 12266, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:35.311822+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:55:35.311822+0000', 'last_peered': '2024-09-18T11:55:35.311822+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:55:35.311822+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4347126, 'num_objects': 531, 'num_object_clones': 0, 'num_object_copies': 2124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12174, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:39.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:40.131 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:40.131 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24985/36080 objects degraded (69.249%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":738,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9020,"data_bytes":73844108,"bytes_used":3937472512,"bytes_avail":382609584128,"bytes_total":386547056640,"degraded_objects":24985,"degraded_total":36080,"degraded_ratio":0.69248891352549891,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":179},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:40.165 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:40.165 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:40.786 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:40.787 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:40.809 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-18T11:55:40.097938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":68625804,"num_objects":8383,"num_object_clones":0,"num_object_copies":33532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198704,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8471,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160731,"ondisk_log_size":160731,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3845216,"kb_used_data":137188,"kb_used_omap":14,"kb_used_meta":3707953,"kb_avail":373642144,"statfs":{"total":386547056640,"available":382609555456,"internally_reserved":0,"allocated":140480512,"data_stored":139745228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3796943899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,2,7,8,14,12,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21954560,"num_objects":-2680,"num_object_clones":0,"num_object_copies":-10720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2680,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010026"},"pg_stats":[{"pgid":"1.f","version":"37'12744","reported_seq":12845,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.694977+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:39.694977+0000","last_peered":"2024-09-18T11:55:39.694977+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:39.694977+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4145152,"num_objects":506,"num_object_clones":0,"num_object_copies":2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12744,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12523","reported_seq":16815,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:37.311187+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:37.311187+0000","last_peered":"2024-09-18T11:55:37.311187+0000","last_clean":"2024-09-18T11:55:37.311187+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:37.311187+0000","last_undegraded":"2024-09-18T11:55:37.311187+0000","last_fullsized":"2024-09-18T11:55:37.311187+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4363510,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12305","reported_seq":12420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.384787+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:39.384787+0000","last_peered":"2024-09-18T11:55:39.384787+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:39.384787+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4119187,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12304,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12325","reported_seq":12419,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.411850+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:35.411850+0000","last_peered":"2024-09-18T11:55:35.411850+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:35.411850+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4920566,"num_objects":601,"num_object_clones":0,"num_object_copies":2404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12387","reported_seq":12508,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.539742+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:39.539742+0000","last_peered":"2024-09-18T11:55:39.539742+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:39.539742+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4158710,"num_objects":508,"num_object_clones":0,"num_object_copies":2032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1544}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12319","reported_seq":12438,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.682067+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:39.682067+0000","last_peered":"2024-09-18T11:55:39.682067+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:39.682067+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3825664,"num_objects":467,"num_object_clones":0,"num_object_copies":1868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12319,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12462","reported_seq":12569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.212973+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:35.212973+0000","last_peered":"2024-09-18T11:55:35.212973+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:35.212973+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4535542,"num_objects":554,"num_object_clones":0,"num_object_copies":2216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12460,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1562}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12312","reported_seq":16084,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.381419+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:35.381419+0000","last_peered":"2024-09-18T11:55:35.381419+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:35.381419+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4609270,"num_objects":563,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12310,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1854,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12659","reported_seq":17012,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:39.654122+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:39.654122+0000","last_peered":"2024-09-18T11:55:39.654122+0000","last_clean":"2024-09-18T11:55:39.654122+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:39.654122+0000","last_undegraded":"2024-09-18T11:55:39.654122+0000","last_fullsized":"2024-09-18T11:55:39.654122+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4229612,"num_objects":517,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12656,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12109","reported_seq":12221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.386007+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:39.386007+0000","last_peered":"2024-09-18T11:55:39.386007+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:39.386007+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3970294,"num_objects":485,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12609,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.470148+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:39.470148+0000","last_peered":"2024-09-18T11:55:39.470148+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:39.470148+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4460401,"num_objects":545,"num_object_clones":0,"num_object_copies":2180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12555","reported_seq":12648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.282272+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:39.282272+0000","last_peered":"2024-09-18T11:55:39.282272+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:39.282272+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3897979,"num_objects":476,"num_object_clones":0,"num_object_copies":1904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12457","reported_seq":12555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.021916+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:39.021916+0000","last_peered":"2024-09-18T11:55:39.021916+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:39.021916+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4176507,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12546","reported_seq":12671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.709619+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:39.709619+0000","last_peered":"2024-09-18T11:55:39.709619+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:39.709619+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4453622,"num_objects":544,"num_object_clones":0,"num_object_copies":2176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1579}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12364","reported_seq":16653,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:37.484724+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:37.484724+0000","last_peered":"2024-09-18T11:55:37.484724+0000","last_clean":"2024-09-18T11:55:37.484724+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:37.484724+0000","last_undegraded":"2024-09-18T11:55:37.484724+0000","last_fullsized":"2024-09-18T11:55:37.484724+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4412662,"num_objects":539,"num_object_clones":0,"num_object_copies":2156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12362,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12176","reported_seq":12266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:35.311822+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:35.311822+0000","last_peered":"2024-09-18T11:55:35.311822+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:35.311822+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4347126,"num_objects":531,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12174,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":68625804,"num_objects":8383,"num_object_clones":0,"num_object_copies":33532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198704,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8471,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":138276864,"data_stored":138276864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160731,"ondisk_log_size":160731,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":960780,"kb_used_data":33092,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93411060,"statfs":{"total":96636764160,"available":95652925440,"internally_reserved":0,"allocated":33886208,"data_stored":33702387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,1,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":7.444},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":7.4119999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.77100000000000002},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.49099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.5819999999999999},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.51700000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951892,"kb_used_data":33032,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93419948,"statfs":{"total":96636764160,"available":95662026752,"internally_reserved":0,"allocated":33824768,"data_stored":33640947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.263},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.3290000000000006}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":9.298},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":11.476000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":964592,"kb_used_data":34900,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93407248,"statfs":{"total":96636764160,"available":95649021952,"internally_reserved":0,"allocated":35737600,"data_stored":35553779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7,5,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.70099999999999996},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":5.0780000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":3.7210000000000001},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":4.5129999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":7.5270000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":5.117}]}]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967952,"kb_used_data":36164,"kb_used_omap":4,"kb_used_meta":931771,"kb_avail":93403888,"statfs":{"total":96636764160,"available":95645581312,"internally_reserved":0,"allocated":37031936,"data_stored":36848115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954134081},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":11.972},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":7.0659999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":13.397},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":7.0330000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":7.0910000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36704256,"data_stored":36704256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35217408,"data_stored":35217408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33337344,"data_stored":33337344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33017856,"data_stored":33017856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:43.811 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:44.354 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:44.355 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:44.385 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-18T11:55:44.098857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65283468,"num_objects":7975,"num_object_clones":0,"num_object_copies":31900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199112,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8535,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3828896,"kb_used_data":129196,"kb_used_omap":14,"kb_used_meta":3699633,"kb_avail":373658464,"statfs":{"total":386547056640,"available":382626267136,"internally_reserved":0,"allocated":132296704,"data_stored":131561420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788424219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,6,5,11,17,7,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18014208,"num_objects":-2199,"num_object_clones":0,"num_object_copies":-8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2199,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006699"},"pg_stats":[{"pgid":"1.f","version":"37'12744","reported_seq":12845,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.694977+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:39.694977+0000","last_peered":"2024-09-18T11:55:39.694977+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:39.694977+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4145152,"num_objects":506,"num_object_clones":0,"num_object_copies":2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12744,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12606","reported_seq":16898,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.643811+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:42.643811+0000","last_peered":"2024-09-18T11:55:42.643811+0000","last_clean":"2024-09-18T11:55:42.643811+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:42.643811+0000","last_undegraded":"2024-09-18T11:55:42.643811+0000","last_fullsized":"2024-09-18T11:55:42.643811+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3683574,"num_objects":450,"num_object_clones":0,"num_object_copies":1800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12604,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12305","reported_seq":12420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.384787+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:39.384787+0000","last_peered":"2024-09-18T11:55:39.384787+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:39.384787+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4119187,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12304,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12388","reported_seq":12482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:40.281307+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:40.281307+0000","last_peered":"2024-09-18T11:55:40.281307+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:40.281307+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404470,"num_objects":538,"num_object_clones":0,"num_object_copies":2152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12386,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12387","reported_seq":12508,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.539742+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:39.539742+0000","last_peered":"2024-09-18T11:55:39.539742+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:39.539742+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4158710,"num_objects":508,"num_object_clones":0,"num_object_copies":2032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1544}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12319","reported_seq":12438,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.682067+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:39.682067+0000","last_peered":"2024-09-18T11:55:39.682067+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:39.682067+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3825664,"num_objects":467,"num_object_clones":0,"num_object_copies":1868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12319,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12540","reported_seq":12646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:40.528945+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:40.528945+0000","last_peered":"2024-09-18T11:55:40.528945+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:40.528945+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3896566,"num_objects":476,"num_object_clones":0,"num_object_copies":1904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12538,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1640}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12366","reported_seq":16268,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:40.508863+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:40.508863+0000","last_peered":"2024-09-18T11:55:40.508863+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:40.508863+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4166902,"num_objects":509,"num_object_clones":0,"num_object_copies":2036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12364,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1918,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12659","reported_seq":17012,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:39.654122+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:39.654122+0000","last_peered":"2024-09-18T11:55:39.654122+0000","last_clean":"2024-09-18T11:55:39.654122+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:39.654122+0000","last_undegraded":"2024-09-18T11:55:39.654122+0000","last_fullsized":"2024-09-18T11:55:39.654122+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4229612,"num_objects":517,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12656,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12109","reported_seq":12221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.386007+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:39.386007+0000","last_peered":"2024-09-18T11:55:39.386007+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:39.386007+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3970294,"num_objects":485,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12609,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.470148+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:39.470148+0000","last_peered":"2024-09-18T11:55:39.470148+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:39.470148+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4460401,"num_objects":545,"num_object_clones":0,"num_object_copies":2180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12555","reported_seq":12648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.282272+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:39.282272+0000","last_peered":"2024-09-18T11:55:39.282272+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:39.282272+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3897979,"num_objects":476,"num_object_clones":0,"num_object_copies":1904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12457","reported_seq":12555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.021916+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:39.021916+0000","last_peered":"2024-09-18T11:55:39.021916+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:39.021916+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4176507,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12546","reported_seq":12671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.709619+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:39.709619+0000","last_peered":"2024-09-18T11:55:39.709619+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:39.709619+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4453622,"num_objects":544,"num_object_clones":0,"num_object_copies":2176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1579}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12434","reported_seq":16723,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.639239+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:42.639239+0000","last_peered":"2024-09-18T11:55:42.639239+0000","last_clean":"2024-09-18T11:55:42.639239+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:42.639239+0000","last_undegraded":"2024-09-18T11:55:42.639239+0000","last_fullsized":"2024-09-18T11:55:42.639239+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3839222,"num_objects":469,"num_object_clones":0,"num_object_copies":1876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12432,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12236","reported_seq":12326,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:40.467602+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:40.467602+0000","last_peered":"2024-09-18T11:55:40.467602+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:40.467602+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3855606,"num_objects":471,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1531}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":65283468,"num_objects":7975,"num_object_clones":0,"num_object_copies":31900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199112,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8535,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":129777664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":960780,"kb_used_data":33092,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93411060,"statfs":{"total":96636764160,"available":95652925440,"internally_reserved":0,"allocated":33886208,"data_stored":33702387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,1,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":7.444},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":7.4119999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.77100000000000002},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.49099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.5819999999999999},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.51700000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951892,"kb_used_data":33032,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93419948,"statfs":{"total":96636764160,"available":95662026752,"internally_reserved":0,"allocated":33824768,"data_stored":33640947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.263},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.3290000000000006}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":9.298},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":11.476000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":964728,"kb_used_data":30884,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93407112,"statfs":{"total":96636764160,"available":95648882688,"internally_reserved":0,"allocated":31625216,"data_stored":31441395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":5.125},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":11.971}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":12.097},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.093}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":3.4390000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":6.4939999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951496,"kb_used_data":32188,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93420344,"statfs":{"total":96636764160,"available":95662432256,"internally_reserved":0,"allocated":32960512,"data_stored":32776691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,3,2,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":15.423},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.2609999999999999},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.0490000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":12.630000000000001},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":2.9969999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32399360,"data_stored":32399360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31023104,"data_stored":31023104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33337344,"data_stored":33337344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33017856,"data_stored":33017856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:44.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:44.961 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:44.962 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:44.989 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-18T11:55:44.098857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65283468,"num_objects":7975,"num_object_clones":0,"num_object_copies":31900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199112,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8535,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3828896,"kb_used_data":129196,"kb_used_omap":14,"kb_used_meta":3699633,"kb_avail":373658464,"statfs":{"total":386547056640,"available":382626267136,"internally_reserved":0,"allocated":132296704,"data_stored":131561420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788424219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,6,5,11,17,7,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18014208,"num_objects":-2199,"num_object_clones":0,"num_object_copies":-8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2199,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006699"},"pg_stats":[{"pgid":"1.f","version":"37'12744","reported_seq":12845,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.694977+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:39.694977+0000","last_peered":"2024-09-18T11:55:39.694977+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:39.694977+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4145152,"num_objects":506,"num_object_clones":0,"num_object_copies":2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12744,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12606","reported_seq":16898,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.643811+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:42.643811+0000","last_peered":"2024-09-18T11:55:42.643811+0000","last_clean":"2024-09-18T11:55:42.643811+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:42.643811+0000","last_undegraded":"2024-09-18T11:55:42.643811+0000","last_fullsized":"2024-09-18T11:55:42.643811+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3683574,"num_objects":450,"num_object_clones":0,"num_object_copies":1800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12604,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12305","reported_seq":12420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.384787+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:39.384787+0000","last_peered":"2024-09-18T11:55:39.384787+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:39.384787+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4119187,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12304,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12388","reported_seq":12482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:40.281307+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:40.281307+0000","last_peered":"2024-09-18T11:55:40.281307+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:40.281307+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404470,"num_objects":538,"num_object_clones":0,"num_object_copies":2152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12386,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12387","reported_seq":12508,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.539742+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:39.539742+0000","last_peered":"2024-09-18T11:55:39.539742+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:39.539742+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4158710,"num_objects":508,"num_object_clones":0,"num_object_copies":2032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1544}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12319","reported_seq":12438,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.682067+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:39.682067+0000","last_peered":"2024-09-18T11:55:39.682067+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:39.682067+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3825664,"num_objects":467,"num_object_clones":0,"num_object_copies":1868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12319,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12540","reported_seq":12646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:40.528945+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:40.528945+0000","last_peered":"2024-09-18T11:55:40.528945+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:40.528945+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3896566,"num_objects":476,"num_object_clones":0,"num_object_copies":1904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12538,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1640}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12366","reported_seq":16268,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:40.508863+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:40.508863+0000","last_peered":"2024-09-18T11:55:40.508863+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:40.508863+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4166902,"num_objects":509,"num_object_clones":0,"num_object_copies":2036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12364,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1918,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12659","reported_seq":17012,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:39.654122+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:39.654122+0000","last_peered":"2024-09-18T11:55:39.654122+0000","last_clean":"2024-09-18T11:55:39.654122+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:39.654122+0000","last_undegraded":"2024-09-18T11:55:39.654122+0000","last_fullsized":"2024-09-18T11:55:39.654122+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4229612,"num_objects":517,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12656,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12109","reported_seq":12221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.386007+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:39.386007+0000","last_peered":"2024-09-18T11:55:39.386007+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:39.386007+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3970294,"num_objects":485,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12609,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.470148+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:39.470148+0000","last_peered":"2024-09-18T11:55:39.470148+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:39.470148+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4460401,"num_objects":545,"num_object_clones":0,"num_object_copies":2180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12555","reported_seq":12648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.282272+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:39.282272+0000","last_peered":"2024-09-18T11:55:39.282272+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:39.282272+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3897979,"num_objects":476,"num_object_clones":0,"num_object_copies":1904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12457","reported_seq":12555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.021916+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:39.021916+0000","last_peered":"2024-09-18T11:55:39.021916+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:39.021916+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4176507,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12546","reported_seq":12671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:39.709619+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:39.709619+0000","last_peered":"2024-09-18T11:55:39.709619+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:39.709619+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4453622,"num_objects":544,"num_object_clones":0,"num_object_copies":2176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1579}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12434","reported_seq":16723,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.639239+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:42.639239+0000","last_peered":"2024-09-18T11:55:42.639239+0000","last_clean":"2024-09-18T11:55:42.639239+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:42.639239+0000","last_undegraded":"2024-09-18T11:55:42.639239+0000","last_fullsized":"2024-09-18T11:55:42.639239+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3839222,"num_objects":469,"num_object_clones":0,"num_object_copies":1876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12432,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12236","reported_seq":12326,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:40.467602+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:40.467602+0000","last_peered":"2024-09-18T11:55:40.467602+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:40.467602+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3855606,"num_objects":471,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1531}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":65283468,"num_objects":7975,"num_object_clones":0,"num_object_copies":31900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199112,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8535,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":129777664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":960780,"kb_used_data":33092,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93411060,"statfs":{"total":96636764160,"available":95652925440,"internally_reserved":0,"allocated":33886208,"data_stored":33702387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,1,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":7.444},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":7.4119999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.77100000000000002},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.49099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":7.5819999999999999},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.51700000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951892,"kb_used_data":33032,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93419948,"statfs":{"total":96636764160,"available":95662026752,"internally_reserved":0,"allocated":33824768,"data_stored":33640947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.263},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":9.3290000000000006}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":9.298},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":11.476000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":964728,"kb_used_data":30884,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93407112,"statfs":{"total":96636764160,"available":95648882688,"internally_reserved":0,"allocated":31625216,"data_stored":31441395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":5.125},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":11.971}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":12.097},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.093}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":3.4390000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":6.4939999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951496,"kb_used_data":32188,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93420344,"statfs":{"total":96636764160,"available":95662432256,"internally_reserved":0,"allocated":32960512,"data_stored":32776691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,3,2,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":15.423},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.2609999999999999},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.0490000000000004}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":12.630000000000001},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":2.9969999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32399360,"data_stored":32399360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31023104,"data_stored":31023104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33337344,"data_stored":33337344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33017856,"data_stored":33017856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:44.991 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:44.991 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12744", 'reported_seq': 12845, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:39.694977+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:55:39.694977+0000', 'last_peered': '2024-09-18T11:55:39.694977+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:55:39.694977+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4145152, 'num_objects': 506, 'num_object_clones': 0, 'num_object_copies': 2024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12744, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1597}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:44.991 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:44.991 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12305", 'reported_seq': 12420, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:39.384787+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:55:39.384787+0000', 'last_peered': '2024-09-18T11:55:39.384787+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:55:39.384787+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4119187, 'num_objects': 504, 'num_object_clones': 0, 'num_object_copies': 2016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 504, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12304, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1536}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:44.991 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:44.991 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12388", 'reported_seq': 12482, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:40.281307+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:40.281307+0000', 'last_peered': '2024-09-18T11:55:40.281307+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:40.281307+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4404470, 'num_objects': 538, 'num_object_clones': 0, 'num_object_copies': 2152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12386, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1563}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:44.992 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:44.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12387", 'reported_seq': 12508, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:39.539742+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:55:39.539742+0000', 'last_peered': '2024-09-18T11:55:39.539742+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:55:39.539742+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4158710, 'num_objects': 508, 'num_object_clones': 0, 'num_object_copies': 2032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12385, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1544}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:44.992 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:44.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12319", 'reported_seq': 12438, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:39.682067+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:55:39.682067+0000', 'last_peered': '2024-09-18T11:55:39.682067+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:55:39.682067+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3825664, 'num_objects': 467, 'num_object_clones': 0, 'num_object_copies': 1868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12319, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1652}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:44.992 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:44.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12540", 'reported_seq': 12646, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:40.528945+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:55:40.528945+0000', 'last_peered': '2024-09-18T11:55:40.528945+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:55:40.528945+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3896566, 'num_objects': 476, 'num_object_clones': 0, 'num_object_copies': 1904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12538, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1640}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:44.992 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:44.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12366", 'reported_seq': 16268, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:40.508863+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:55:40.508863+0000', 'last_peered': '2024-09-18T11:55:40.508863+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:55:40.508863+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4166902, 'num_objects': 509, 'num_object_clones': 0, 'num_object_copies': 2036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12364, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1918, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 389}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:44.992 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:44.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12109", 'reported_seq': 12221, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:39.386007+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:55:39.386007+0000', 'last_peered': '2024-09-18T11:55:39.386007+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:55:39.386007+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2100", 'ondisk_log_start': "31'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3970294, 'num_objects': 485, 'num_object_clones': 0, 'num_object_copies': 1940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1478}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:44.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:44.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12609, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:39.470148+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:55:39.470148+0000', 'last_peered': '2024-09-18T11:55:39.470148+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:55:39.470148+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4460401, 'num_objects': 545, 'num_object_clones': 0, 'num_object_copies': 2180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1489}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:44.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:44.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12555", 'reported_seq': 12648, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:39.282272+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:55:39.282272+0000', 'last_peered': '2024-09-18T11:55:39.282272+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:55:39.282272+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3897979, 'num_objects': 476, 'num_object_clones': 0, 'num_object_copies': 1904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12554, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:44.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:44.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12457", 'reported_seq': 12555, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:39.021916+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:55:39.021916+0000', 'last_peered': '2024-09-18T11:55:39.021916+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:55:39.021916+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4176507, 'num_objects': 510, 'num_object_clones': 0, 'num_object_copies': 2040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12456, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:44.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:44.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12546", 'reported_seq': 12671, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:39.709619+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:55:39.709619+0000', 'last_peered': '2024-09-18T11:55:39.709619+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:55:39.709619+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4453622, 'num_objects': 544, 'num_object_clones': 0, 'num_object_copies': 2176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 544, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12544, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1579}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:44.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:44.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12236", 'reported_seq': 12326, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:40.467602+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:55:40.467602+0000', 'last_peered': '2024-09-18T11:55:40.467602+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:55:40.467602+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3855606, 'num_objects': 471, 'num_object_clones': 0, 'num_object_copies': 1884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12234, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:44.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:45.693 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:45.693 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24921/31900 objects degraded (78.122%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":743,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7975,"data_bytes":65283468,"bytes_used":3920789504,"bytes_avail":382626267136,"bytes_total":386547056640,"degraded_objects":24921,"degraded_total":31900,"degraded_ratio":0.78122257053291533,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":183},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:45.711 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:45.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:46.246 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:46.247 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:46.266 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-09-18T11:55:46.099346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":57877900,"num_objects":7071,"num_object_clones":0,"num_object_copies":28284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7071,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200016,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8588,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160643,"ondisk_log_size":160643,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3829068,"kb_used_data":116892,"kb_used_omap":14,"kb_used_meta":3712113,"kb_avail":373658292,"statfs":{"total":386547056640,"available":382626091008,"internally_reserved":0,"allocated":119697408,"data_stored":118962124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3801203739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,4,0,7,6,9,12,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24313856,"num_objects":-2968,"num_object_clones":0,"num_object_copies":-11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006733"},"pg_stats":[{"pgid":"1.f","version":"37'12831","reported_seq":12931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.840846+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:44.840846+0000","last_peered":"2024-09-18T11:55:44.840846+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:44.840846+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3432448,"num_objects":419,"num_object_clones":0,"num_object_copies":1676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12831,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1684}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12606","reported_seq":16898,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.643811+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:42.643811+0000","last_peered":"2024-09-18T11:55:42.643811+0000","last_clean":"2024-09-18T11:55:42.643811+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:42.643811+0000","last_undegraded":"2024-09-18T11:55:42.643811+0000","last_fullsized":"2024-09-18T11:55:42.643811+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3683574,"num_objects":450,"num_object_clones":0,"num_object_copies":1800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12604,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12360","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.500971+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:44.500971+0000","last_peered":"2024-09-18T11:55:44.500971+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:44.500971+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3668627,"num_objects":449,"num_object_clones":0,"num_object_copies":1796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":449,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12359,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1591}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12456","reported_seq":12550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.446053+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:45.446053+0000","last_peered":"2024-09-18T11:55:45.446053+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:45.446053+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3847414,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12449","reported_seq":12568,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.548186+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:44.548186+0000","last_peered":"2024-09-18T11:55:44.548186+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:44.548186+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3650806,"num_objects":446,"num_object_clones":0,"num_object_copies":1784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12447,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12375","reported_seq":12494,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.760489+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:44.760489+0000","last_peered":"2024-09-18T11:55:44.760489+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:44.760489+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3366912,"num_objects":411,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12375,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1708}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12601","reported_seq":12708,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.561473+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:45.561473+0000","last_peered":"2024-09-18T11:55:45.561473+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:45.561473+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3396854,"num_objects":415,"num_object_clones":0,"num_object_copies":1660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12599,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1701}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12450","reported_seq":16457,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.607380+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:45.607380+0000","last_peered":"2024-09-18T11:55:45.607380+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:45.607380+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3478774,"num_objects":425,"num_object_clones":0,"num_object_copies":1700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12721","reported_seq":17074,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:44.656402+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:44.656402+0000","last_peered":"2024-09-18T11:55:44.656402+0000","last_clean":"2024-09-18T11:55:44.656402+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:44.656402+0000","last_undegraded":"2024-09-18T11:55:44.656402+0000","last_fullsized":"2024-09-18T11:55:44.656402+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3721708,"num_objects":455,"num_object_clones":0,"num_object_copies":1820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12718,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12162","reported_seq":12275,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.517079+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:44.517079+0000","last_peered":"2024-09-18T11:55:44.517079+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:44.517079+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3536118,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12160,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1531}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12560","reported_seq":12681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.724676+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:44.724676+0000","last_peered":"2024-09-18T11:55:44.724676+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:44.724676+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3870577,"num_objects":473,"num_object_clones":0,"num_object_copies":1892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12557,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12610","reported_seq":12703,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.797157+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:44.797157+0000","last_peered":"2024-09-18T11:55:44.797157+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:44.797157+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3447419,"num_objects":421,"num_object_clones":0,"num_object_copies":1684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12609,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1663}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12513","reported_seq":12611,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.449454+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:44.449454+0000","last_peered":"2024-09-18T11:55:44.449454+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:44.449454+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3717755,"num_objects":454,"num_object_clones":0,"num_object_copies":1816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12512,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12624","reported_seq":12749,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.795135+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:44.795135+0000","last_peered":"2024-09-18T11:55:44.795135+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:44.795135+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3814646,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12622,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12434","reported_seq":16723,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.639239+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:42.639239+0000","last_peered":"2024-09-18T11:55:42.639239+0000","last_clean":"2024-09-18T11:55:42.639239+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:42.639239+0000","last_undegraded":"2024-09-18T11:55:42.639239+0000","last_fullsized":"2024-09-18T11:55:42.639239+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3839222,"num_objects":469,"num_object_clones":0,"num_object_copies":1876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12432,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12291","reported_seq":12381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.433077+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:45.433077+0000","last_peered":"2024-09-18T11:55:45.433077+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:45.433077+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3405046,"num_objects":416,"num_object_clones":0,"num_object_copies":1664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":57877900,"num_objects":7071,"num_object_clones":0,"num_object_copies":28284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7071,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200016,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8588,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":117055488,"data_stored":117055488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160643,"ondisk_log_size":160643,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":960904,"kb_used_data":29052,"kb_used_omap":4,"kb_used_meta":931835,"kb_avail":93410936,"statfs":{"total":96636764160,"available":95652798464,"internally_reserved":0,"allocated":29749248,"data_stored":29565427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954199617},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,4,0,2,0,5,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":10.182},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.214}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.882},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.58099999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":3.222},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.60599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951948,"kb_used_data":28932,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93419892,"statfs":{"total":96636764160,"available":95661969408,"internally_reserved":0,"allocated":29626368,"data_stored":29442547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":1.7709999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":2.7850000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.161},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":3.927},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":2.9620000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":964728,"kb_used_data":30884,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93407112,"statfs":{"total":96636764160,"available":95648882688,"internally_reserved":0,"allocated":31625216,"data_stored":31441395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":5.125},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":11.971}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":12.097},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.093}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":3.4390000000000001},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":6.4939999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951488,"kb_used_data":28024,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93420352,"statfs":{"total":96636764160,"available":95662440448,"internally_reserved":0,"allocated":28696576,"data_stored":28512755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,0,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.673},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.745}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":4.9610000000000003},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.524}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":12.847},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.561999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28225536,"data_stored":28225536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31023104,"data_stored":31023104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29040640,"data_stored":29040640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28766208,"data_stored":28766208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:49.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:49.866 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:49.866 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:49.901 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-18T11:55:48.099883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":56976780,"num_objects":6961,"num_object_clones":0,"num_object_copies":27844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6961,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200126,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8588,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3812540,"kb_used_data":112856,"kb_used_omap":14,"kb_used_meta":3699633,"kb_avail":373674820,"statfs":{"total":386547056640,"available":382643015680,"internally_reserved":0,"allocated":115564544,"data_stored":114829260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788424219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,4,0,8,6,8,8,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17940480,"num_objects":-2190,"num_object_clones":0,"num_object_copies":-8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2190,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006830"},"pg_stats":[{"pgid":"1.f","version":"37'12831","reported_seq":12931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.840846+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:44.840846+0000","last_peered":"2024-09-18T11:55:44.840846+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:44.840846+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3432448,"num_objects":419,"num_object_clones":0,"num_object_copies":1676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12831,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1684}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12667","reported_seq":16958,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:47.703974+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:47.703974+0000","last_peered":"2024-09-18T11:55:47.703974+0000","last_clean":"2024-09-18T11:55:47.703974+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:47.703974+0000","last_undegraded":"2024-09-18T11:55:47.703974+0000","last_fullsized":"2024-09-18T11:55:47.703974+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3183862,"num_objects":389,"num_object_clones":0,"num_object_copies":1556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12665,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12360","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.500971+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:44.500971+0000","last_peered":"2024-09-18T11:55:44.500971+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:44.500971+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3668627,"num_objects":449,"num_object_clones":0,"num_object_copies":1796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":449,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12359,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1591}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12456","reported_seq":12550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.446053+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:45.446053+0000","last_peered":"2024-09-18T11:55:45.446053+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:45.446053+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3847414,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12449","reported_seq":12568,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.548186+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:44.548186+0000","last_peered":"2024-09-18T11:55:44.548186+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:44.548186+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3650806,"num_objects":446,"num_object_clones":0,"num_object_copies":1784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12447,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12375","reported_seq":12494,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.760489+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:44.760489+0000","last_peered":"2024-09-18T11:55:44.760489+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:44.760489+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3366912,"num_objects":411,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12375,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1708}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12601","reported_seq":12708,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.561473+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:45.561473+0000","last_peered":"2024-09-18T11:55:45.561473+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:45.561473+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3396854,"num_objects":415,"num_object_clones":0,"num_object_copies":1660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12599,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1701}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12450","reported_seq":16457,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.607380+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:45.607380+0000","last_peered":"2024-09-18T11:55:45.607380+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:45.607380+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3478774,"num_objects":425,"num_object_clones":0,"num_object_copies":1700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12721","reported_seq":17074,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:44.656402+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:44.656402+0000","last_peered":"2024-09-18T11:55:44.656402+0000","last_clean":"2024-09-18T11:55:44.656402+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:44.656402+0000","last_undegraded":"2024-09-18T11:55:44.656402+0000","last_fullsized":"2024-09-18T11:55:44.656402+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3721708,"num_objects":455,"num_object_clones":0,"num_object_copies":1820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12718,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12162","reported_seq":12275,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.517079+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:44.517079+0000","last_peered":"2024-09-18T11:55:44.517079+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:44.517079+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3536118,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12160,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1531}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12560","reported_seq":12681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.724676+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:44.724676+0000","last_peered":"2024-09-18T11:55:44.724676+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:44.724676+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3870577,"num_objects":473,"num_object_clones":0,"num_object_copies":1892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12557,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12610","reported_seq":12703,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.797157+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:44.797157+0000","last_peered":"2024-09-18T11:55:44.797157+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:44.797157+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3447419,"num_objects":421,"num_object_clones":0,"num_object_copies":1684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12609,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1663}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12513","reported_seq":12611,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.449454+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:44.449454+0000","last_peered":"2024-09-18T11:55:44.449454+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:44.449454+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3717755,"num_objects":454,"num_object_clones":0,"num_object_copies":1816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12512,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12624","reported_seq":12749,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:44.795135+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:44.795135+0000","last_peered":"2024-09-18T11:55:44.795135+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:44.795135+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3814646,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12622,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12483","reported_seq":16772,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:47.575699+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:47.575699+0000","last_peered":"2024-09-18T11:55:47.575699+0000","last_clean":"2024-09-18T11:55:47.575699+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:47.575699+0000","last_undegraded":"2024-09-18T11:55:47.575699+0000","last_fullsized":"2024-09-18T11:55:47.575699+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3437814,"num_objects":420,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12291","reported_seq":12381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.433077+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:45.433077+0000","last_peered":"2024-09-18T11:55:45.433077+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:45.433077+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3405046,"num_objects":416,"num_object_clones":0,"num_object_copies":1664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":56976780,"num_objects":6961,"num_object_clones":0,"num_object_copies":27844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6961,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200126,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8588,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":112848896,"data_stored":112848896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":960904,"kb_used_data":29052,"kb_used_omap":4,"kb_used_meta":931835,"kb_avail":93410936,"statfs":{"total":96636764160,"available":95652798464,"internally_reserved":0,"allocated":29749248,"data_stored":29565427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954199617},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,4,0,2,0,5,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":10.182},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":10.214}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":11.882},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.58099999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":3.222},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.60599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951948,"kb_used_data":28932,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93419892,"statfs":{"total":96636764160,"available":95661969408,"internally_reserved":0,"allocated":29626368,"data_stored":29442547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":1.7709999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":2.7850000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.161},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":3.927},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":2.9620000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948200,"kb_used_data":26848,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93423640,"statfs":{"total":96636764160,"available":95665807360,"internally_reserved":0,"allocated":27492352,"data_stored":27308531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":3.8690000000000002},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":17.684999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":19.995000000000001},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":2.0720000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":17.654},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.960000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951488,"kb_used_data":28024,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93420352,"statfs":{"total":96636764160,"available":95662440448,"internally_reserved":0,"allocated":28696576,"data_stored":28512755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,0,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.673},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.745}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":4.9610000000000003},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.524}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":12.847},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.561999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28225536,"data_stored":28225536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26816512,"data_stored":26816512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29040640,"data_stored":29040640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28766208,"data_stored":28766208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:49.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:50.456 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:50.480 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-18T11:55:50.100351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51766668,"num_objects":6325,"num_object_clones":0,"num_object_copies":25300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200762,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8588,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3796184,"kb_used_data":104820,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373691176,"statfs":{"total":386547056640,"available":382659764224,"internally_reserved":0,"allocated":107335680,"data_stored":106600396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,6,6,8,2,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22077440,"num_objects":-2695,"num_object_clones":0,"num_object_copies":-10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2695,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003048"},"pg_stats":[{"pgid":"1.f","version":"37'12902","reported_seq":13003,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.904996+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:49.904996+0000","last_peered":"2024-09-18T11:55:49.904996+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:49.904996+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2850816,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12902,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12667","reported_seq":16958,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:47.703974+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:47.703974+0000","last_peered":"2024-09-18T11:55:47.703974+0000","last_clean":"2024-09-18T11:55:47.703974+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:47.703974+0000","last_undegraded":"2024-09-18T11:55:47.703974+0000","last_fullsized":"2024-09-18T11:55:47.703974+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3183862,"num_objects":389,"num_object_clones":0,"num_object_copies":1556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12665,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12433","reported_seq":12550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.507643+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:49.507643+0000","last_peered":"2024-09-18T11:55:49.507643+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:49.507643+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3070611,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12432,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1664}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12456","reported_seq":12550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.446053+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:45.446053+0000","last_peered":"2024-09-18T11:55:45.446053+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:45.446053+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3847414,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12523","reported_seq":12644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.535071+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:49.535071+0000","last_peered":"2024-09-18T11:55:49.535071+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:49.535071+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3044598,"num_objects":372,"num_object_clones":0,"num_object_copies":1488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12432","reported_seq":12550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.937660+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:49.937660+0000","last_peered":"2024-09-18T11:55:49.937660+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:49.937660+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2899968,"num_objects":354,"num_object_clones":0,"num_object_copies":1416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12432,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12601","reported_seq":12708,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.561473+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:45.561473+0000","last_peered":"2024-09-18T11:55:45.561473+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:45.561473+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3396854,"num_objects":415,"num_object_clones":0,"num_object_copies":1660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12599,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1701}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12450","reported_seq":16457,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.607380+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:45.607380+0000","last_peered":"2024-09-18T11:55:45.607380+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:45.607380+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3478774,"num_objects":425,"num_object_clones":0,"num_object_copies":1700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12777","reported_seq":17129,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:49.965010+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:49.965010+0000","last_peered":"2024-09-18T11:55:49.965010+0000","last_clean":"2024-09-18T11:55:49.965010+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:49.965010+0000","last_undegraded":"2024-09-18T11:55:49.965010+0000","last_fullsized":"2024-09-18T11:55:49.965010+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3262956,"num_objects":399,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12774,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12220","reported_seq":12334,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.255523+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:49.255523+0000","last_peered":"2024-09-18T11:55:49.255523+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:49.255523+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3060982,"num_objects":374,"num_object_clones":0,"num_object_copies":1496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12218,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1589}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12631","reported_seq":12752,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.771592+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:49.771592+0000","last_peered":"2024-09-18T11:55:49.771592+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:49.771592+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3288945,"num_objects":402,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12628,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12665","reported_seq":12758,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.620464+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:49.620464+0000","last_peered":"2024-09-18T11:55:49.620464+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:49.620464+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2996859,"num_objects":366,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12664,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1718}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12572","reported_seq":12670,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.417284+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:49.417284+0000","last_peered":"2024-09-18T11:55:49.417284+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:49.417284+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3234427,"num_objects":395,"num_object_clones":0,"num_object_copies":1580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1651}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12686","reported_seq":12811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.877242+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:49.877242+0000","last_peered":"2024-09-18T11:55:49.877242+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:49.877242+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3306742,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12684,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1719}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12483","reported_seq":16772,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:47.575699+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:47.575699+0000","last_peered":"2024-09-18T11:55:47.575699+0000","last_clean":"2024-09-18T11:55:47.575699+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:47.575699+0000","last_undegraded":"2024-09-18T11:55:47.575699+0000","last_fullsized":"2024-09-18T11:55:47.575699+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3437814,"num_objects":420,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12291","reported_seq":12381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.433077+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:45.433077+0000","last_peered":"2024-09-18T11:55:45.433077+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:45.433077+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3405046,"num_objects":416,"num_object_clones":0,"num_object_copies":1664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":51766668,"num_objects":6325,"num_object_clones":0,"num_object_copies":25300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200762,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8588,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104472576,"data_stored":104472576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944416,"kb_used_data":25048,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93427424,"statfs":{"total":96636764160,"available":95669682176,"internally_reserved":0,"allocated":25649152,"data_stored":25465331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,3,3,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":5.258},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":5.2869999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":12.243}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":1.49}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":952080,"kb_used_data":24900,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93419760,"statfs":{"total":96636764160,"available":95661834240,"internally_reserved":0,"allocated":25497600,"data_stored":25313779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":1.7709999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":2.7850000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.161},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":3.927},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":2.9620000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948200,"kb_used_data":26848,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93423640,"statfs":{"total":96636764160,"available":95665807360,"internally_reserved":0,"allocated":27492352,"data_stored":27308531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":3.8690000000000002},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":17.684999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":19.995000000000001},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":2.0720000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":17.654},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.960000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951488,"kb_used_data":28024,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93420352,"statfs":{"total":96636764160,"available":95662440448,"internally_reserved":0,"allocated":28696576,"data_stored":28512755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,0,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.673},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.745}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":4.9610000000000003},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.524}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":12.847},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.561999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28225536,"data_stored":28225536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26816512,"data_stored":26816512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24891392,"data_stored":24891392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24539136,"data_stored":24539136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:50.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:50.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12902", 'reported_seq': 13003, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:49.904996+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:55:49.904996+0000', 'last_peered': '2024-09-18T11:55:49.904996+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:55:49.904996+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2850816, 'num_objects': 348, 'num_object_clones': 0, 'num_object_copies': 1392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12902, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1755}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:50.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:50.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12433", 'reported_seq': 12550, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:49.507643+0000', 'last_change': '2024-09-18T11:44:47.434768+0000', 'last_active': '2024-09-18T11:55:49.507643+0000', 'last_peered': '2024-09-18T11:55:49.507643+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:55:49.507643+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3070611, 'num_objects': 376, 'num_object_clones': 0, 'num_object_copies': 1504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 376, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12432, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1664}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:50.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:50.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12456", 'reported_seq': 12550, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:45.446053+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:45.446053+0000', 'last_peered': '2024-09-18T11:55:45.446053+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:45.446053+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3847414, 'num_objects': 470, 'num_object_clones': 0, 'num_object_copies': 1880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12454, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:50.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:50.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12523", 'reported_seq': 12644, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:49.535071+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:55:49.535071+0000', 'last_peered': '2024-09-18T11:55:49.535071+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:55:49.535071+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3044598, 'num_objects': 372, 'num_object_clones': 0, 'num_object_copies': 1488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12521, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1680}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:50.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:50.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12432", 'reported_seq': 12550, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:49.937660+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:55:49.937660+0000', 'last_peered': '2024-09-18T11:55:49.937660+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:55:49.937660+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2899968, 'num_objects': 354, 'num_object_clones': 0, 'num_object_copies': 1416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 354, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12432, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1765}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:50.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:50.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12601", 'reported_seq': 12708, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:45.561473+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:55:45.561473+0000', 'last_peered': '2024-09-18T11:55:45.561473+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:55:45.561473+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3396854, 'num_objects': 415, 'num_object_clones': 0, 'num_object_copies': 1660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12599, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1701}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:50.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:50.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12450", 'reported_seq': 16457, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:45.607380+0000', 'last_change': '2024-09-18T11:52:35.720789+0000', 'last_active': '2024-09-18T11:55:45.607380+0000', 'last_peered': '2024-09-18T11:55:45.607380+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:55:45.607380+0000', 'last_undegraded': '2024-09-18T11:44:47.185349+0000', 'last_fullsized': '2024-09-18T11:44:47.163766+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3478774, 'num_objects': 425, 'num_object_clones': 0, 'num_object_copies': 1700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 67, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12448, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1971, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:50.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:50.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12220", 'reported_seq': 12334, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:49.255523+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:55:49.255523+0000', 'last_peered': '2024-09-18T11:55:49.255523+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:55:49.255523+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3060982, 'num_objects': 374, 'num_object_clones': 0, 'num_object_copies': 1496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12218, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1589}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:50.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:50.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12631", 'reported_seq': 12752, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:49.771592+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:55:49.771592+0000', 'last_peered': '2024-09-18T11:55:49.771592+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:55:49.771592+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3288945, 'num_objects': 402, 'num_object_clones': 0, 'num_object_copies': 1608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12628, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1632}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:50.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:50.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12665", 'reported_seq': 12758, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:49.620464+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:55:49.620464+0000', 'last_peered': '2024-09-18T11:55:49.620464+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:55:49.620464+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2996859, 'num_objects': 366, 'num_object_clones': 0, 'num_object_copies': 1464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12664, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1718}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:50.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:50.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12572", 'reported_seq': 12670, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:49.417284+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:55:49.417284+0000', 'last_peered': '2024-09-18T11:55:49.417284+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:55:49.417284+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3234427, 'num_objects': 395, 'num_object_clones': 0, 'num_object_copies': 1580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12571, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:50.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:50.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12686", 'reported_seq': 12811, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:49.877242+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:55:49.877242+0000', 'last_peered': '2024-09-18T11:55:49.877242+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:55:49.877242+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3306742, 'num_objects': 404, 'num_object_clones': 0, 'num_object_copies': 1616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12684, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1719}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:50.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:50.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12291", 'reported_seq': 12381, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:45.433077+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:55:45.433077+0000', 'last_peered': '2024-09-18T11:55:45.433077+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:55:45.433077+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3405046, 'num_objects': 416, 'num_object_clones': 0, 'num_object_copies': 1664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12289, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:50.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:51.176 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:51.176 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24868/25300 objects degraded (98.292%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":749,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6325,"data_bytes":51766668,"bytes_used":3887292416,"bytes_avail":382659764224,"bytes_total":386547056640,"degraded_objects":24868,"degraded_total":25300,"degraded_ratio":0.98292490118577081,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":224},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:51.199 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:51.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:51.818 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:51.819 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:51.852 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-18T11:55:50.100351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51766668,"num_objects":6325,"num_object_clones":0,"num_object_copies":25300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200762,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8588,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3796184,"kb_used_data":104820,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373691176,"statfs":{"total":386547056640,"available":382659764224,"internally_reserved":0,"allocated":107335680,"data_stored":106600396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,6,6,8,2,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22077440,"num_objects":-2695,"num_object_clones":0,"num_object_copies":-10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2695,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003048"},"pg_stats":[{"pgid":"1.f","version":"37'12902","reported_seq":13003,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.904996+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:49.904996+0000","last_peered":"2024-09-18T11:55:49.904996+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:49.904996+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2850816,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12902,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12667","reported_seq":16958,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:47.703974+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:47.703974+0000","last_peered":"2024-09-18T11:55:47.703974+0000","last_clean":"2024-09-18T11:55:47.703974+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:47.703974+0000","last_undegraded":"2024-09-18T11:55:47.703974+0000","last_fullsized":"2024-09-18T11:55:47.703974+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3183862,"num_objects":389,"num_object_clones":0,"num_object_copies":1556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12665,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12433","reported_seq":12550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.507643+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:49.507643+0000","last_peered":"2024-09-18T11:55:49.507643+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:49.507643+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3070611,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12432,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1664}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12456","reported_seq":12550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.446053+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:45.446053+0000","last_peered":"2024-09-18T11:55:45.446053+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:45.446053+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3847414,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12523","reported_seq":12644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.535071+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:49.535071+0000","last_peered":"2024-09-18T11:55:49.535071+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:49.535071+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3044598,"num_objects":372,"num_object_clones":0,"num_object_copies":1488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12432","reported_seq":12550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.937660+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:49.937660+0000","last_peered":"2024-09-18T11:55:49.937660+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:49.937660+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2899968,"num_objects":354,"num_object_clones":0,"num_object_copies":1416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12432,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12601","reported_seq":12708,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.561473+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:45.561473+0000","last_peered":"2024-09-18T11:55:45.561473+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:45.561473+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3396854,"num_objects":415,"num_object_clones":0,"num_object_copies":1660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12599,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1701}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12450","reported_seq":16457,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.607380+0000","last_change":"2024-09-18T11:52:35.720789+0000","last_active":"2024-09-18T11:55:45.607380+0000","last_peered":"2024-09-18T11:55:45.607380+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:45.607380+0000","last_undegraded":"2024-09-18T11:44:47.185349+0000","last_fullsized":"2024-09-18T11:44:47.163766+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3478774,"num_objects":425,"num_object_clones":0,"num_object_copies":1700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12777","reported_seq":17129,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:49.965010+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:49.965010+0000","last_peered":"2024-09-18T11:55:49.965010+0000","last_clean":"2024-09-18T11:55:49.965010+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:49.965010+0000","last_undegraded":"2024-09-18T11:55:49.965010+0000","last_fullsized":"2024-09-18T11:55:49.965010+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3262956,"num_objects":399,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12774,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12220","reported_seq":12334,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.255523+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:49.255523+0000","last_peered":"2024-09-18T11:55:49.255523+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:49.255523+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3060982,"num_objects":374,"num_object_clones":0,"num_object_copies":1496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12218,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1589}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12631","reported_seq":12752,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.771592+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:49.771592+0000","last_peered":"2024-09-18T11:55:49.771592+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:49.771592+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3288945,"num_objects":402,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12628,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12665","reported_seq":12758,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.620464+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:49.620464+0000","last_peered":"2024-09-18T11:55:49.620464+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:49.620464+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2996859,"num_objects":366,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12664,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1718}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12572","reported_seq":12670,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.417284+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:49.417284+0000","last_peered":"2024-09-18T11:55:49.417284+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:49.417284+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3234427,"num_objects":395,"num_object_clones":0,"num_object_copies":1580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1651}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12686","reported_seq":12811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.877242+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:49.877242+0000","last_peered":"2024-09-18T11:55:49.877242+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:49.877242+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3306742,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12684,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1719}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12483","reported_seq":16772,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:47.575699+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:47.575699+0000","last_peered":"2024-09-18T11:55:47.575699+0000","last_clean":"2024-09-18T11:55:47.575699+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:47.575699+0000","last_undegraded":"2024-09-18T11:55:47.575699+0000","last_fullsized":"2024-09-18T11:55:47.575699+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3437814,"num_objects":420,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12291","reported_seq":12381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:45.433077+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:45.433077+0000","last_peered":"2024-09-18T11:55:45.433077+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:45.433077+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3405046,"num_objects":416,"num_object_clones":0,"num_object_copies":1664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":51766668,"num_objects":6325,"num_object_clones":0,"num_object_copies":25300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200762,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8588,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104472576,"data_stored":104472576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944416,"kb_used_data":25048,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93427424,"statfs":{"total":96636764160,"available":95669682176,"internally_reserved":0,"allocated":25649152,"data_stored":25465331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,3,3,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":5.258},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":5.2869999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":12.243}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":1.49}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":952080,"kb_used_data":24900,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93419760,"statfs":{"total":96636764160,"available":95661834240,"internally_reserved":0,"allocated":25497600,"data_stored":25313779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":1.7709999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":2.7850000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.161},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":3.927},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":2.9620000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948200,"kb_used_data":26848,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93423640,"statfs":{"total":96636764160,"available":95665807360,"internally_reserved":0,"allocated":27492352,"data_stored":27308531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":3.8690000000000002},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":17.684999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":19.995000000000001},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":2.0720000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":17.654},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.960000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951488,"kb_used_data":28024,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93420352,"statfs":{"total":96636764160,"available":95662440448,"internally_reserved":0,"allocated":28696576,"data_stored":28512755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,0,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":1.673},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.745}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":4.9610000000000003},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.524}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":12.847},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":10.561999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28225536,"data_stored":28225536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26816512,"data_stored":26816512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24891392,"data_stored":24891392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24539136,"data_stored":24539136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:54.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:55.446 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:55.447 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:55.472 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-09-18T11:55:54.101269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":48358796,"num_objects":5909,"num_object_clones":0,"num_object_copies":23636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201178,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8655,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3792360,"kb_used_data":96812,"kb_used_omap":14,"kb_used_meta":3695473,"kb_avail":373695000,"statfs":{"total":386547056640,"available":382663680000,"internally_reserved":0,"allocated":99135488,"data_stored":98396000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3784164379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,11,1,8,19,3,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9519104,"num_objects":-1162,"num_object_clones":0,"num_object_copies":-4648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1162,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001922"},"pg_stats":[{"pgid":"1.f","version":"37'12902","reported_seq":13003,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.904996+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:49.904996+0000","last_peered":"2024-09-18T11:55:49.904996+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:49.904996+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2850816,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12902,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'12729","reported_seq":17022,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.474995+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:52.474995+0000","last_peered":"2024-09-18T11:55:52.474995+0000","last_clean":"2024-09-18T11:55:52.474995+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:52.474995+0000","last_undegraded":"2024-09-18T11:55:52.474995+0000","last_fullsized":"2024-09-18T11:55:52.474995+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2675958,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12727,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12433","reported_seq":12550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.507643+0000","last_change":"2024-09-18T11:44:47.434768+0000","last_active":"2024-09-18T11:55:49.507643+0000","last_peered":"2024-09-18T11:55:49.507643+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:49.507643+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3070611,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12432,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1664}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12530","reported_seq":12624,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:50.619568+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:50.619568+0000","last_peered":"2024-09-18T11:55:50.619568+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:50.619568+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3241206,"num_objects":396,"num_object_clones":0,"num_object_copies":1584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12528,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12523","reported_seq":12644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.535071+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:49.535071+0000","last_peered":"2024-09-18T11:55:49.535071+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:49.535071+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3044598,"num_objects":372,"num_object_clones":0,"num_object_copies":1488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12432","reported_seq":12550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.937660+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:49.937660+0000","last_peered":"2024-09-18T11:55:49.937660+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:49.937660+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2899968,"num_objects":354,"num_object_clones":0,"num_object_copies":1416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12432,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12668","reported_seq":12775,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:50.538156+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:50.538156+0000","last_peered":"2024-09-18T11:55:50.538156+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:50.538156+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2847990,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1768}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12517","reported_seq":16663,"reported_epoch":38,"state":"peering","last_fresh":"2024-09-18T11:55:52.006340+0000","last_change":"2024-09-18T11:55:52.006009+0000","last_active":"2024-09-18T11:55:52.005789+0000","last_peered":"2024-09-18T11:55:51.935071+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:52.006340+0000","last_undegraded":"2024-09-18T11:55:52.006340+0000","last_fullsized":"2024-09-18T11:55:52.006340+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2929910,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12515,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12777","reported_seq":17129,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:49.965010+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:49.965010+0000","last_peered":"2024-09-18T11:55:49.965010+0000","last_clean":"2024-09-18T11:55:49.965010+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:49.965010+0000","last_undegraded":"2024-09-18T11:55:49.965010+0000","last_fullsized":"2024-09-18T11:55:49.965010+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3262956,"num_objects":399,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12774,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12220","reported_seq":12334,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.255523+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:49.255523+0000","last_peered":"2024-09-18T11:55:49.255523+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:49.255523+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3060982,"num_objects":374,"num_object_clones":0,"num_object_copies":1496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12218,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1589}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12631","reported_seq":12752,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.771592+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:49.771592+0000","last_peered":"2024-09-18T11:55:49.771592+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:49.771592+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3288945,"num_objects":402,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12628,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12665","reported_seq":12758,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.620464+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:49.620464+0000","last_peered":"2024-09-18T11:55:49.620464+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:49.620464+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2996859,"num_objects":366,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12664,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1718}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12572","reported_seq":12670,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.417284+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:49.417284+0000","last_peered":"2024-09-18T11:55:49.417284+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:49.417284+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3234427,"num_objects":395,"num_object_clones":0,"num_object_copies":1580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1651}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12686","reported_seq":12811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:49.877242+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:49.877242+0000","last_peered":"2024-09-18T11:55:49.877242+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:49.877242+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3306742,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12684,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1719}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'12553","reported_seq":16843,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.504252+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:52.504252+0000","last_peered":"2024-09-18T11:55:52.504252+0000","last_clean":"2024-09-18T11:55:52.504252+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:52.504252+0000","last_undegraded":"2024-09-18T11:55:52.504252+0000","last_fullsized":"2024-09-18T11:55:52.504252+0000","mapping_epoch":32,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2864374,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12551,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12367","reported_seq":12457,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:50.520969+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:50.520969+0000","last_peered":"2024-09-18T11:55:50.520969+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:50.520969+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782454,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12365,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1662}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":48358796,"num_objects":5909,"num_object_clones":0,"num_object_copies":23636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201178,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8655,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96174080,"data_stored":96174080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944416,"kb_used_data":25048,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93427424,"statfs":{"total":96636764160,"available":95669682176,"internally_reserved":0,"allocated":25649152,"data_stored":25465331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,3,3,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":5.258},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":5.2869999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":12.243}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":1.49}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":952080,"kb_used_data":24900,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93419760,"statfs":{"total":96636764160,"available":95661834240,"internally_reserved":0,"allocated":25497600,"data_stored":25313779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":1.7709999999999999},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":2.7850000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":2.161},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":3.927},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":2.9620000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948328,"kb_used_data":22796,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93423512,"statfs":{"total":96636764160,"available":95665676288,"internally_reserved":0,"allocated":23343104,"data_stored":23155079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,1,12,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.1309999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":13.065},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":3.3450000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":1.1120000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":947536,"kb_used_data":24068,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93424304,"statfs":{"total":96636764160,"available":95666487296,"internally_reserved":0,"allocated":24645632,"data_stored":24461811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.764},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.1859999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28799999999999998},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":5.2599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23986176,"data_stored":23986176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22757376,"data_stored":22757376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24891392,"data_stored":24891392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24539136,"data_stored":24539136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:55.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:56.115 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:56.115 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:56.138 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-18T11:55:56.101685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41821580,"num_objects":5111,"num_object_clones":0,"num_object_copies":20444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201976,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8703,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160603,"ondisk_log_size":160603,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3775776,"kb_used_data":84360,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373711584,"statfs":{"total":386547056640,"available":382680662016,"internally_reserved":0,"allocated":86384640,"data_stored":85619865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,8,8,13,23,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15155200,"num_objects":-1850,"num_object_clones":0,"num_object_copies":-7400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1850,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001802"},"pg_stats":[{"pgid":"1.f","version":"39'12958","reported_seq":13061,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.794676+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:54.794676+0000","last_peered":"2024-09-18T11:55:54.794676+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:54.794676+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2392064,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12958,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'12729","reported_seq":17022,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.474995+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:52.474995+0000","last_peered":"2024-09-18T11:55:52.474995+0000","last_clean":"2024-09-18T11:55:52.474995+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:52.474995+0000","last_undegraded":"2024-09-18T11:55:52.474995+0000","last_fullsized":"2024-09-18T11:55:52.474995+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2675958,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12727,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12477","reported_seq":12693,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.630720+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:55:54.630720+0000","last_peered":"2024-09-18T11:55:54.630720+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:54.630720+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2710163,"num_objects":332,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12476,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1661}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12610","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.677887+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:55.677887+0000","last_peered":"2024-09-18T11:55:55.677887+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:55.677887+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585846,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12608,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12573","reported_seq":12696,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.875413+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:54.875413+0000","last_peered":"2024-09-18T11:55:54.875413+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:54.875413+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2634998,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12504","reported_seq":12625,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.777491+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:54.777491+0000","last_peered":"2024-09-18T11:55:54.777491+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:54.777491+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2310144,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12504,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12727","reported_seq":12836,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.659547+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:55.659547+0000","last_peered":"2024-09-18T11:55:55.659547+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:55.659547+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2364662,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12517","reported_seq":16663,"reported_epoch":38,"state":"peering","last_fresh":"2024-09-18T11:55:52.006340+0000","last_change":"2024-09-18T11:55:52.006009+0000","last_active":"2024-09-18T11:55:52.005789+0000","last_peered":"2024-09-18T11:55:51.935071+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:52.006340+0000","last_undegraded":"2024-09-18T11:55:52.006340+0000","last_fullsized":"2024-09-18T11:55:52.006340+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2929910,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12515,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12840","reported_seq":17194,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:54.978411+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:54.978411+0000","last_peered":"2024-09-18T11:55:54.978411+0000","last_clean":"2024-09-18T11:55:54.978411+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:54.978411+0000","last_undegraded":"2024-09-18T11:55:54.978411+0000","last_fullsized":"2024-09-18T11:55:54.978411+0000","mapping_epoch":36,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2746860,"num_objects":336,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12837,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12289","reported_seq":12404,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.590759+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:54.590759+0000","last_peered":"2024-09-18T11:55:54.590759+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:54.590759+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2495734,"num_objects":305,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12287,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12705","reported_seq":12828,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.883282+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:54.883282+0000","last_peered":"2024-09-18T11:55:54.883282+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:54.883282+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2682737,"num_objects":328,"num_object_clones":0,"num_object_copies":1312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12702,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12734","reported_seq":12829,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.943052+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:54.943052+0000","last_peered":"2024-09-18T11:55:54.943052+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:54.943052+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2431611,"num_objects":297,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12733,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1787}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12628","reported_seq":12728,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.478951+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:54.478951+0000","last_peered":"2024-09-18T11:55:54.478951+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:54.478951+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2775675,"num_objects":339,"num_object_clones":0,"num_object_copies":1356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12627,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1707}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12744","reported_seq":12871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.908055+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:54.908055+0000","last_peered":"2024-09-18T11:55:54.908055+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:54.908055+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2831606,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12742,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1777}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'12553","reported_seq":16843,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.504252+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:52.504252+0000","last_peered":"2024-09-18T11:55:52.504252+0000","last_clean":"2024-09-18T11:55:52.504252+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:52.504252+0000","last_undegraded":"2024-09-18T11:55:52.504252+0000","last_fullsized":"2024-09-18T11:55:52.504252+0000","mapping_epoch":32,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2864374,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12551,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12415","reported_seq":12507,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.484698+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:55.484698+0000","last_peered":"2024-09-18T11:55:55.484698+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:55.484698+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2389238,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12413,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":41821580,"num_objects":5111,"num_object_clones":0,"num_object_copies":20444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201976,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8703,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":84127744,"data_stored":84127744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160603,"ondisk_log_size":160603,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":943984,"kb_used_data":20456,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93427856,"statfs":{"total":96636764160,"available":95670124544,"internally_reserved":0,"allocated":20946944,"data_stored":20754694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,7,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":16.733000000000001},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.9159999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":12.217000000000001},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":11.275}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":11.34},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":4.4569999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935680,"kb_used_data":20976,"kb_used_omap":1,"kb_used_meta":914686,"kb_avail":93436160,"statfs":{"total":96636764160,"available":95678627840,"internally_reserved":0,"allocated":21479424,"data_stored":21287174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936638808},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":2.5670000000000002},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":2.6000000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":6.5170000000000003},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":13.279}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.46800000000000003},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.432}]}]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948328,"kb_used_data":22796,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93423512,"statfs":{"total":96636764160,"available":95665676288,"internally_reserved":0,"allocated":23343104,"data_stored":23155079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,1,12,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.1309999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":13.065},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":3.3450000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":1.1120000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":947784,"kb_used_data":20132,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93424056,"statfs":{"total":96636764160,"available":95666233344,"internally_reserved":0,"allocated":20615168,"data_stored":20422918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":5.4489999999999998},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":9.4960000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":12.164},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.8490000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":11.576000000000001},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19976192,"data_stored":19976192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22757376,"data_stored":22757376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20856832,"data_stored":20856832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20537344,"data_stored":20537344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:56.140 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:55:56.140 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12958", 'reported_seq': 13061, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:54.794676+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:55:54.794676+0000', 'last_peered': '2024-09-18T11:55:54.794676+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:55:54.794676+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2392064, 'num_objects': 292, 'num_object_clones': 0, 'num_object_copies': 1168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12958, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:56.140 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:55:56.140 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12477", 'reported_seq': 12693, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:54.630720+0000', 'last_change': '2024-09-18T11:55:51.694654+0000', 'last_active': '2024-09-18T11:55:54.630720+0000', 'last_peered': '2024-09-18T11:55:54.630720+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:55:54.630720+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2710163, 'num_objects': 332, 'num_object_clones': 0, 'num_object_copies': 1328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 332, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12476, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1661}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:56.140 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:55:56.140 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12610", 'reported_seq': 12706, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:55.677887+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:55.677887+0000', 'last_peered': '2024-09-18T11:55:55.677887+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:55.677887+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2585846, 'num_objects': 316, 'num_object_clones': 0, 'num_object_copies': 1264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12608, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1785}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:56.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:55:56.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12573", 'reported_seq': 12696, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:54.875413+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:55:54.875413+0000', 'last_peered': '2024-09-18T11:55:54.875413+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:55:54.875413+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2634998, 'num_objects': 322, 'num_object_clones': 0, 'num_object_copies': 1288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12571, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1730}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:56.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:55:56.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12504", 'reported_seq': 12625, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:54.777491+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:55:54.777491+0000', 'last_peered': '2024-09-18T11:55:54.777491+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:55:54.777491+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2310144, 'num_objects': 282, 'num_object_clones': 0, 'num_object_copies': 1128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 282, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12504, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1837}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:56.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:55:56.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12727", 'reported_seq': 12836, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:55.659547+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:55:55.659547+0000', 'last_peered': '2024-09-18T11:55:55.659547+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:55:55.659547+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2364662, 'num_objects': 289, 'num_object_clones': 0, 'num_object_copies': 1156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12725, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1827}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:56.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-18T11:55:56.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12517", 'reported_seq': 16663, 'reported_epoch': 38, 'state': 'peering', 'last_fresh': '2024-09-18T11:55:52.006340+0000', 'last_change': '2024-09-18T11:55:52.006009+0000', 'last_active': '2024-09-18T11:55:52.005789+0000', 'last_peered': '2024-09-18T11:55:51.935071+0000', 'last_clean': '2024-09-18T11:43:44.640228+0000', 'last_became_active': '2024-09-18T11:44:47.407533+0000', 'last_became_peered': '2024-09-18T11:44:47.407533+0000', 'last_unstale': '2024-09-18T11:55:52.006340+0000', 'last_undegraded': '2024-09-18T11:55:52.006340+0000', 'last_fullsized': '2024-09-18T11:55:52.006340+0000', 'mapping_epoch': 38, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2929910, 'num_objects': 358, 'num_object_clones': 0, 'num_object_copies': 1432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12515, 'num_write_kb': 51492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2038, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-18T11:55:56.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:55:56.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12289", 'reported_seq': 12404, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:54.590759+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:55:54.590759+0000', 'last_peered': '2024-09-18T11:55:54.590759+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:55:54.590759+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2495734, 'num_objects': 305, 'num_object_clones': 0, 'num_object_copies': 1220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12287, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1658}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:56.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:55:56.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12705", 'reported_seq': 12828, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:54.883282+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:55:54.883282+0000', 'last_peered': '2024-09-18T11:55:54.883282+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:55:54.883282+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2682737, 'num_objects': 328, 'num_object_clones': 0, 'num_object_copies': 1312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12702, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1706}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:56.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:55:56.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12734", 'reported_seq': 12829, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:54.943052+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:55:54.943052+0000', 'last_peered': '2024-09-18T11:55:54.943052+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:55:54.943052+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2431611, 'num_objects': 297, 'num_object_clones': 0, 'num_object_copies': 1188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 297, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12733, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1787}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:56.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:55:56.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12628", 'reported_seq': 12728, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:54.478951+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:55:54.478951+0000', 'last_peered': '2024-09-18T11:55:54.478951+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:55:54.478951+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2775675, 'num_objects': 339, 'num_object_clones': 0, 'num_object_copies': 1356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12627, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1707}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:56.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:55:56.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12744", 'reported_seq': 12871, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:54.908055+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:55:54.908055+0000', 'last_peered': '2024-09-18T11:55:54.908055+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:55:54.908055+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2831606, 'num_objects': 346, 'num_object_clones': 0, 'num_object_copies': 1384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12742, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1777}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:55:56.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:55:56.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12415", 'reported_seq': 12507, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:55.484698+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:55:55.484698+0000', 'last_peered': '2024-09-18T11:55:55.484698+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:55:55.484698+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2389238, 'num_objects': 292, 'num_object_clones': 0, 'num_object_copies': 1168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12413, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1710}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:56.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:56.843 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:56.843 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24753/20444 objects degraded (121.077%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":754,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":3},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":5111,"data_bytes":41821580,"bytes_used":3866394624,"bytes_avail":382680662016,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":24753,"degraded_total":20444,"degraded_ratio":1.2107708863236157,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":231},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:55:56.872 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:56.873 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:57.449 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:55:57.450 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:55:57.474 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-18T11:55:56.101685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41821580,"num_objects":5111,"num_object_clones":0,"num_object_copies":20444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201976,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8703,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160603,"ondisk_log_size":160603,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3775776,"kb_used_data":84360,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373711584,"statfs":{"total":386547056640,"available":382680662016,"internally_reserved":0,"allocated":86384640,"data_stored":85619865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,8,8,13,23,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15155200,"num_objects":-1850,"num_object_clones":0,"num_object_copies":-7400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1850,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001802"},"pg_stats":[{"pgid":"1.f","version":"39'12958","reported_seq":13061,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.794676+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:54.794676+0000","last_peered":"2024-09-18T11:55:54.794676+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:54.794676+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2392064,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12958,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'12729","reported_seq":17022,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.474995+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:52.474995+0000","last_peered":"2024-09-18T11:55:52.474995+0000","last_clean":"2024-09-18T11:55:52.474995+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:52.474995+0000","last_undegraded":"2024-09-18T11:55:52.474995+0000","last_fullsized":"2024-09-18T11:55:52.474995+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2675958,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12727,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12477","reported_seq":12693,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.630720+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:55:54.630720+0000","last_peered":"2024-09-18T11:55:54.630720+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:54.630720+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2710163,"num_objects":332,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12476,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1661}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12610","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.677887+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:55.677887+0000","last_peered":"2024-09-18T11:55:55.677887+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:55.677887+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585846,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12608,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12573","reported_seq":12696,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.875413+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:54.875413+0000","last_peered":"2024-09-18T11:55:54.875413+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:54.875413+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2634998,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12504","reported_seq":12625,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.777491+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:54.777491+0000","last_peered":"2024-09-18T11:55:54.777491+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:54.777491+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2310144,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12504,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12727","reported_seq":12836,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.659547+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:55.659547+0000","last_peered":"2024-09-18T11:55:55.659547+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:55.659547+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2364662,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12517","reported_seq":16663,"reported_epoch":38,"state":"peering","last_fresh":"2024-09-18T11:55:52.006340+0000","last_change":"2024-09-18T11:55:52.006009+0000","last_active":"2024-09-18T11:55:52.005789+0000","last_peered":"2024-09-18T11:55:51.935071+0000","last_clean":"2024-09-18T11:43:44.640228+0000","last_became_active":"2024-09-18T11:44:47.407533+0000","last_became_peered":"2024-09-18T11:44:47.407533+0000","last_unstale":"2024-09-18T11:55:52.006340+0000","last_undegraded":"2024-09-18T11:55:52.006340+0000","last_fullsized":"2024-09-18T11:55:52.006340+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2929910,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12515,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12840","reported_seq":17194,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:54.978411+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:55:54.978411+0000","last_peered":"2024-09-18T11:55:54.978411+0000","last_clean":"2024-09-18T11:55:54.978411+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:55:54.978411+0000","last_undegraded":"2024-09-18T11:55:54.978411+0000","last_fullsized":"2024-09-18T11:55:54.978411+0000","mapping_epoch":36,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2746860,"num_objects":336,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12837,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12289","reported_seq":12404,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.590759+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:54.590759+0000","last_peered":"2024-09-18T11:55:54.590759+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:54.590759+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2495734,"num_objects":305,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12287,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12705","reported_seq":12828,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.883282+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:54.883282+0000","last_peered":"2024-09-18T11:55:54.883282+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:54.883282+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2682737,"num_objects":328,"num_object_clones":0,"num_object_copies":1312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12702,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12734","reported_seq":12829,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.943052+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:54.943052+0000","last_peered":"2024-09-18T11:55:54.943052+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:54.943052+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2431611,"num_objects":297,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12733,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1787}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12628","reported_seq":12728,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.478951+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:54.478951+0000","last_peered":"2024-09-18T11:55:54.478951+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:54.478951+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2775675,"num_objects":339,"num_object_clones":0,"num_object_copies":1356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12627,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1707}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12744","reported_seq":12871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:54.908055+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:54.908055+0000","last_peered":"2024-09-18T11:55:54.908055+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:54.908055+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2831606,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12742,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1777}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'12553","reported_seq":16843,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.504252+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:52.504252+0000","last_peered":"2024-09-18T11:55:52.504252+0000","last_clean":"2024-09-18T11:55:52.504252+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:52.504252+0000","last_undegraded":"2024-09-18T11:55:52.504252+0000","last_fullsized":"2024-09-18T11:55:52.504252+0000","mapping_epoch":32,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2864374,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12551,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12415","reported_seq":12507,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.484698+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:55.484698+0000","last_peered":"2024-09-18T11:55:55.484698+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:55.484698+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2389238,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12413,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":41821580,"num_objects":5111,"num_object_clones":0,"num_object_copies":20444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201976,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8703,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":84127744,"data_stored":84127744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160603,"ondisk_log_size":160603,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":943984,"kb_used_data":20456,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93427856,"statfs":{"total":96636764160,"available":95670124544,"internally_reserved":0,"allocated":20946944,"data_stored":20754694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,7,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":16.733000000000001},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":1.9159999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":12.217000000000001},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":11.275}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":11.34},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":4.4569999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935680,"kb_used_data":20976,"kb_used_omap":1,"kb_used_meta":914686,"kb_avail":93436160,"statfs":{"total":96636764160,"available":95678627840,"internally_reserved":0,"allocated":21479424,"data_stored":21287174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936638808},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":2.5670000000000002},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":2.6000000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":6.5170000000000003},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":13.279}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.46800000000000003},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.432}]}]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948328,"kb_used_data":22796,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93423512,"statfs":{"total":96636764160,"available":95665676288,"internally_reserved":0,"allocated":23343104,"data_stored":23155079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,1,12,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":2.1309999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":13.065},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":3.3450000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":1.1120000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":947784,"kb_used_data":20132,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93424056,"statfs":{"total":96636764160,"available":95666233344,"internally_reserved":0,"allocated":20615168,"data_stored":20422918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":5.4489999999999998},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":9.4960000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":12.164},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.8490000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":11.576000000000001},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19976192,"data_stored":19976192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22757376,"data_stored":22757376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20856832,"data_stored":20856832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20537344,"data_stored":20537344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:00.477 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:01.071 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:01.072 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:01.100 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-09-18T11:56:00.102610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":35161484,"num_objects":4298,"num_object_clones":0,"num_object_copies":17192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202789,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8755,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3776400,"kb_used_data":72528,"kb_used_omap":14,"kb_used_meta":3703793,"kb_avail":373710960,"statfs":{"total":386547056640,"available":382680023040,"internally_reserved":0,"allocated":74268672,"data_stored":73499672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792684059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,13,12,11,14,14,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14409728,"num_objects":-1759,"num_object_clones":0,"num_object_copies":-7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1759,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.115477"},"pg_stats":[{"pgid":"1.f","version":"39'13014","reported_seq":13117,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.928783+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:59.928783+0000","last_peered":"2024-09-18T11:55:59.928783+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:59.928783+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933312,"num_objects":236,"num_object_clones":0,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13014,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1867}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12786","reported_seq":17080,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.765196+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:57.765196+0000","last_peered":"2024-09-18T11:55:57.765196+0000","last_clean":"2024-09-18T11:55:57.765196+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:57.765196+0000","last_undegraded":"2024-09-18T11:55:57.765196+0000","last_fullsized":"2024-09-18T11:55:57.765196+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2209014,"num_objects":270,"num_object_clones":0,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12784,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12533","reported_seq":12853,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.557011+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:55:59.557011+0000","last_peered":"2024-09-18T11:55:59.557011+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:59.557011+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2251411,"num_objects":276,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12532,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1665}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12610","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.677887+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:55.677887+0000","last_peered":"2024-09-18T11:55:55.677887+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:55.677887+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585846,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12608,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12643","reported_seq":12765,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.946341+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:59.946341+0000","last_peered":"2024-09-18T11:55:59.946341+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:59.946341+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2061558,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12641,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12569","reported_seq":12690,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.929894+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:59.929894+0000","last_peered":"2024-09-18T11:55:59.929894+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:59.929894+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1777664,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12569,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12727","reported_seq":12836,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.659547+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:55.659547+0000","last_peered":"2024-09-18T11:55:55.659547+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:55.659547+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2364662,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12590","reported_seq":16746,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.769163+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:55:57.769163+0000","last_peered":"2024-09-18T11:55:57.769163+0000","last_clean":"2024-09-18T11:55:57.769163+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:55:57.769163+0000","last_undegraded":"2024-09-18T11:55:57.769163+0000","last_fullsized":"2024-09-18T11:55:57.769163+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2331894,"num_objects":285,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12913","reported_seq":17264,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:00.023299+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:00.023299+0000","last_peered":"2024-09-18T11:56:00.023299+0000","last_clean":"2024-09-18T11:56:00.023299+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:00.023299+0000","last_undegraded":"2024-09-18T11:56:00.023299+0000","last_fullsized":"2024-09-18T11:56:00.023299+0000","mapping_epoch":36,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2148844,"num_objects":263,"num_object_clones":0,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12910,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12342","reported_seq":12458,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.556922+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:59.556922+0000","last_peered":"2024-09-18T11:55:59.556922+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:59.556922+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2061558,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12340,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12769","reported_seq":12892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.895439+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:59.895439+0000","last_peered":"2024-09-18T11:55:59.895439+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:59.895439+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2158449,"num_objects":264,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12766,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12787","reported_seq":12882,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.657429+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:59.657429+0000","last_peered":"2024-09-18T11:55:59.657429+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:59.657429+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997435,"num_objects":244,"num_object_clones":0,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1840}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12695","reported_seq":12795,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.421239+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:59.421239+0000","last_peered":"2024-09-18T11:55:59.421239+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:59.421239+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2226811,"num_objects":272,"num_object_clones":0,"num_object_copies":1088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12694,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1774}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12795","reported_seq":12922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.818221+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:59.818221+0000","last_peered":"2024-09-18T11:55:59.818221+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:59.818221+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2413814,"num_objects":295,"num_object_clones":0,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12793,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1828}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12628","reported_seq":16919,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.803077+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:57.803077+0000","last_peered":"2024-09-18T11:55:57.803077+0000","last_clean":"2024-09-18T11:55:57.803077+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:57.803077+0000","last_undegraded":"2024-09-18T11:55:57.803077+0000","last_fullsized":"2024-09-18T11:55:57.803077+0000","mapping_epoch":32,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2249974,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12626,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12415","reported_seq":12507,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.484698+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:55.484698+0000","last_peered":"2024-09-18T11:55:55.484698+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:55.484698+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2389238,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12413,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":35161484,"num_objects":4298,"num_object_clones":0,"num_object_copies":17192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202789,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8755,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":71839744,"data_stored":71839744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944240,"kb_used_data":16548,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93427600,"statfs":{"total":96636764160,"available":95669862400,"internally_reserved":0,"allocated":16945152,"data_stored":16752902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,2,1,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.383},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":17.18}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.1579999999999999},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":11.119999999999999},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.23599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935776,"kb_used_data":16920,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93436064,"statfs":{"total":96636764160,"available":95678529536,"internally_reserved":0,"allocated":17326080,"data_stored":17133830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.29799999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.57399999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.39800000000000002},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.61299999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948600,"kb_used_data":18928,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93423240,"statfs":{"total":96636764160,"available":95665397760,"internally_reserved":0,"allocated":19382272,"data_stored":19190022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":16.074000000000002},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":11.750999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":2.6549999999999998},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":16.036999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":4.7000000000000002},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":11.704000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":947784,"kb_used_data":20132,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93424056,"statfs":{"total":96636764160,"available":95666233344,"internally_reserved":0,"allocated":20615168,"data_stored":20422918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":5.4489999999999998},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":9.4960000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":12.164},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.8490000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":11.576000000000001},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19976192,"data_stored":19976192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18677760,"data_stored":18677760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16764928,"data_stored":16764928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16420864,"data_stored":16420864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:01.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:01.645 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:01.646 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:01.666 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-09-18T11:56:00.102610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":35161484,"num_objects":4298,"num_object_clones":0,"num_object_copies":17192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202789,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8755,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3776400,"kb_used_data":72528,"kb_used_omap":14,"kb_used_meta":3703793,"kb_avail":373710960,"statfs":{"total":386547056640,"available":382680023040,"internally_reserved":0,"allocated":74268672,"data_stored":73499672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792684059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,13,12,11,14,14,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14409728,"num_objects":-1759,"num_object_clones":0,"num_object_copies":-7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1759,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.115477"},"pg_stats":[{"pgid":"1.f","version":"39'13014","reported_seq":13117,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.928783+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:59.928783+0000","last_peered":"2024-09-18T11:55:59.928783+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:59.928783+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933312,"num_objects":236,"num_object_clones":0,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13014,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1867}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12786","reported_seq":17080,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.765196+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:57.765196+0000","last_peered":"2024-09-18T11:55:57.765196+0000","last_clean":"2024-09-18T11:55:57.765196+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:57.765196+0000","last_undegraded":"2024-09-18T11:55:57.765196+0000","last_fullsized":"2024-09-18T11:55:57.765196+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2209014,"num_objects":270,"num_object_clones":0,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12784,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12533","reported_seq":12853,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.557011+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:55:59.557011+0000","last_peered":"2024-09-18T11:55:59.557011+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:59.557011+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2251411,"num_objects":276,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12532,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1665}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12610","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.677887+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:55:55.677887+0000","last_peered":"2024-09-18T11:55:55.677887+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:55:55.677887+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585846,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12608,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12643","reported_seq":12765,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.946341+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:59.946341+0000","last_peered":"2024-09-18T11:55:59.946341+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:59.946341+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2061558,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12641,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12569","reported_seq":12690,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.929894+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:59.929894+0000","last_peered":"2024-09-18T11:55:59.929894+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:59.929894+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1777664,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12569,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12727","reported_seq":12836,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.659547+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:55:55.659547+0000","last_peered":"2024-09-18T11:55:55.659547+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:55:55.659547+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2364662,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12590","reported_seq":16746,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.769163+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:55:57.769163+0000","last_peered":"2024-09-18T11:55:57.769163+0000","last_clean":"2024-09-18T11:55:57.769163+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:55:57.769163+0000","last_undegraded":"2024-09-18T11:55:57.769163+0000","last_fullsized":"2024-09-18T11:55:57.769163+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2331894,"num_objects":285,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12913","reported_seq":17264,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:00.023299+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:00.023299+0000","last_peered":"2024-09-18T11:56:00.023299+0000","last_clean":"2024-09-18T11:56:00.023299+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:00.023299+0000","last_undegraded":"2024-09-18T11:56:00.023299+0000","last_fullsized":"2024-09-18T11:56:00.023299+0000","mapping_epoch":36,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2148844,"num_objects":263,"num_object_clones":0,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12910,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12342","reported_seq":12458,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.556922+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:59.556922+0000","last_peered":"2024-09-18T11:55:59.556922+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:59.556922+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2061558,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12340,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12769","reported_seq":12892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.895439+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:59.895439+0000","last_peered":"2024-09-18T11:55:59.895439+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:59.895439+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2158449,"num_objects":264,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12766,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12787","reported_seq":12882,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.657429+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:59.657429+0000","last_peered":"2024-09-18T11:55:59.657429+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:59.657429+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997435,"num_objects":244,"num_object_clones":0,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1840}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12695","reported_seq":12795,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.421239+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:59.421239+0000","last_peered":"2024-09-18T11:55:59.421239+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:59.421239+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2226811,"num_objects":272,"num_object_clones":0,"num_object_copies":1088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12694,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1774}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12795","reported_seq":12922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.818221+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:59.818221+0000","last_peered":"2024-09-18T11:55:59.818221+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:59.818221+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2413814,"num_objects":295,"num_object_clones":0,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12793,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1828}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12628","reported_seq":16919,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.803077+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:57.803077+0000","last_peered":"2024-09-18T11:55:57.803077+0000","last_clean":"2024-09-18T11:55:57.803077+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:57.803077+0000","last_undegraded":"2024-09-18T11:55:57.803077+0000","last_fullsized":"2024-09-18T11:55:57.803077+0000","mapping_epoch":32,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2249974,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12626,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12415","reported_seq":12507,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:55.484698+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:55:55.484698+0000","last_peered":"2024-09-18T11:55:55.484698+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:55:55.484698+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2389238,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12413,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":35161484,"num_objects":4298,"num_object_clones":0,"num_object_copies":17192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202789,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8755,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":71839744,"data_stored":71839744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944240,"kb_used_data":16548,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93427600,"statfs":{"total":96636764160,"available":95669862400,"internally_reserved":0,"allocated":16945152,"data_stored":16752902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,2,1,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.383},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":17.18}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.1579999999999999},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":11.119999999999999},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.23599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935776,"kb_used_data":16920,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93436064,"statfs":{"total":96636764160,"available":95678529536,"internally_reserved":0,"allocated":17326080,"data_stored":17133830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.29799999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.57399999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.39800000000000002},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.61299999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948600,"kb_used_data":18928,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93423240,"statfs":{"total":96636764160,"available":95665397760,"internally_reserved":0,"allocated":19382272,"data_stored":19190022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":16.074000000000002},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":11.750999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":2.6549999999999998},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":16.036999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":4.7000000000000002},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":11.704000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":947784,"kb_used_data":20132,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93424056,"statfs":{"total":96636764160,"available":95666233344,"internally_reserved":0,"allocated":20615168,"data_stored":20422918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":5.4489999999999998},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":9.4960000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":12.164},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":5.8490000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":11.576000000000001},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19976192,"data_stored":19976192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18677760,"data_stored":18677760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16764928,"data_stored":16764928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16420864,"data_stored":16420864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:01.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:01.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13014", 'reported_seq': 13117, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:59.928783+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:55:59.928783+0000', 'last_peered': '2024-09-18T11:55:59.928783+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:55:59.928783+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1933312, 'num_objects': 236, 'num_object_clones': 0, 'num_object_copies': 944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13014, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1867}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:01.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:56:01.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12533", 'reported_seq': 12853, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:59.557011+0000', 'last_change': '2024-09-18T11:55:51.694654+0000', 'last_active': '2024-09-18T11:55:59.557011+0000', 'last_peered': '2024-09-18T11:55:59.557011+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:55:59.557011+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2251411, 'num_objects': 276, 'num_object_clones': 0, 'num_object_copies': 1104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1941, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 276, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12532, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 115, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1665}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:01.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:01.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12610", 'reported_seq': 12706, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:55.677887+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:55:55.677887+0000', 'last_peered': '2024-09-18T11:55:55.677887+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:55:55.677887+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2585846, 'num_objects': 316, 'num_object_clones': 0, 'num_object_copies': 1264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12608, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1785}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:01.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12643", 'reported_seq': 12765, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:59.946341+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:55:59.946341+0000', 'last_peered': '2024-09-18T11:55:59.946341+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:55:59.946341+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2061558, 'num_objects': 252, 'num_object_clones': 0, 'num_object_copies': 1008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12641, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1800}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:01.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:01.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12569", 'reported_seq': 12690, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:59.929894+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:55:59.929894+0000', 'last_peered': '2024-09-18T11:55:59.929894+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:55:59.929894+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1777664, 'num_objects': 217, 'num_object_clones': 0, 'num_object_copies': 868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12569, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1902}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:01.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:01.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12727", 'reported_seq': 12836, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:55.659547+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:55:55.659547+0000', 'last_peered': '2024-09-18T11:55:55.659547+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:55:55.659547+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2364662, 'num_objects': 289, 'num_object_clones': 0, 'num_object_copies': 1156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12725, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1827}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:01.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12342", 'reported_seq': 12458, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:59.556922+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:55:59.556922+0000', 'last_peered': '2024-09-18T11:55:59.556922+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:55:59.556922+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2061558, 'num_objects': 252, 'num_object_clones': 0, 'num_object_copies': 1008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12340, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1711}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:01.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:01.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12769", 'reported_seq': 12892, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:59.895439+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:55:59.895439+0000', 'last_peered': '2024-09-18T11:55:59.895439+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:55:59.895439+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2158449, 'num_objects': 264, 'num_object_clones': 0, 'num_object_copies': 1056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12766, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1770}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:01.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:01.670 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12787", 'reported_seq': 12882, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:59.657429+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:55:59.657429+0000', 'last_peered': '2024-09-18T11:55:59.657429+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:55:59.657429+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1997435, 'num_objects': 244, 'num_object_clones': 0, 'num_object_copies': 976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1840}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:01.670 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:01.670 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12695", 'reported_seq': 12795, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:59.421239+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:55:59.421239+0000', 'last_peered': '2024-09-18T11:55:59.421239+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:55:59.421239+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2226811, 'num_objects': 272, 'num_object_clones': 0, 'num_object_copies': 1088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12694, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1774}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:01.670 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:01.670 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12795", 'reported_seq': 12922, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:59.818221+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:55:59.818221+0000', 'last_peered': '2024-09-18T11:55:59.818221+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:55:59.818221+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2413814, 'num_objects': 295, 'num_object_clones': 0, 'num_object_copies': 1180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12793, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:01.670 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:56:01.670 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12415", 'reported_seq': 12507, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:55.484698+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:55:55.484698+0000', 'last_peered': '2024-09-18T11:55:55.484698+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:55:55.484698+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2389238, 'num_objects': 292, 'num_object_clones': 0, 'num_object_copies': 1168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12413, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1710}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:02.365 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:02.366 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24701/16352 objects degraded (151.058%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":760,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4088,"data_bytes":33441164,"bytes_used":3850100736,"bytes_avail":382696955904,"bytes_total":386547056640,"degraded_objects":24701,"degraded_total":16352,"degraded_ratio":1.5105797455968686,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:02.401 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:02.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:03.014 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:03.014 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:03.033 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-18T11:56:02.103078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33441164,"num_objects":4088,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4088,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202999,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8755,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161026,"ondisk_log_size":161026,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3759864,"kb_used_data":68500,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373727496,"statfs":{"total":386547056640,"available":382696955904,"internally_reserved":0,"allocated":70144000,"data_stored":69375000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,12,8,8,13,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16130048,"num_objects":-1969,"num_object_clones":0,"num_object_copies":-7876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1969,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002235"},"pg_stats":[{"pgid":"1.f","version":"39'13014","reported_seq":13117,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.928783+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:55:59.928783+0000","last_peered":"2024-09-18T11:55:59.928783+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:55:59.928783+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933312,"num_objects":236,"num_object_clones":0,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13014,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1867}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12786","reported_seq":17080,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.765196+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:55:57.765196+0000","last_peered":"2024-09-18T11:55:57.765196+0000","last_clean":"2024-09-18T11:55:57.765196+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:55:57.765196+0000","last_undegraded":"2024-09-18T11:55:57.765196+0000","last_fullsized":"2024-09-18T11:55:57.765196+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2209014,"num_objects":270,"num_object_clones":0,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12784,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12533","reported_seq":12853,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.557011+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:55:59.557011+0000","last_peered":"2024-09-18T11:55:59.557011+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:55:59.557011+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2251411,"num_objects":276,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12532,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1665}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12677","reported_seq":12773,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:00.649350+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:00.649350+0000","last_peered":"2024-09-18T11:56:00.649350+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:00.649350+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2036982,"num_objects":249,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12675,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12643","reported_seq":12765,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.946341+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:55:59.946341+0000","last_peered":"2024-09-18T11:55:59.946341+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:55:59.946341+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2061558,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12641,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12569","reported_seq":12690,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.929894+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:55:59.929894+0000","last_peered":"2024-09-18T11:55:59.929894+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:55:59.929894+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1777664,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12569,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12800","reported_seq":12909,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:00.664303+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:00.664303+0000","last_peered":"2024-09-18T11:56:00.664303+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:00.664303+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1766646,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12798,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1900}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12590","reported_seq":16746,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.769163+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:55:57.769163+0000","last_peered":"2024-09-18T11:55:57.769163+0000","last_clean":"2024-09-18T11:55:57.769163+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:55:57.769163+0000","last_undegraded":"2024-09-18T11:55:57.769163+0000","last_fullsized":"2024-09-18T11:55:57.769163+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2331894,"num_objects":285,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12913","reported_seq":17264,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:00.023299+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:00.023299+0000","last_peered":"2024-09-18T11:56:00.023299+0000","last_clean":"2024-09-18T11:56:00.023299+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:00.023299+0000","last_undegraded":"2024-09-18T11:56:00.023299+0000","last_fullsized":"2024-09-18T11:56:00.023299+0000","mapping_epoch":36,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2148844,"num_objects":263,"num_object_clones":0,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12910,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12342","reported_seq":12458,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.556922+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:55:59.556922+0000","last_peered":"2024-09-18T11:55:59.556922+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:55:59.556922+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2061558,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12340,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12769","reported_seq":12892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.895439+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:55:59.895439+0000","last_peered":"2024-09-18T11:55:59.895439+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:55:59.895439+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2158449,"num_objects":264,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12766,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12787","reported_seq":12882,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.657429+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:55:59.657429+0000","last_peered":"2024-09-18T11:55:59.657429+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:55:59.657429+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997435,"num_objects":244,"num_object_clones":0,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1840}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12695","reported_seq":12795,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.421239+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:55:59.421239+0000","last_peered":"2024-09-18T11:55:59.421239+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:55:59.421239+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2226811,"num_objects":272,"num_object_clones":0,"num_object_copies":1088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12694,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1774}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12795","reported_seq":12922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:59.818221+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:55:59.818221+0000","last_peered":"2024-09-18T11:55:59.818221+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:55:59.818221+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2413814,"num_objects":295,"num_object_clones":0,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12793,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1828}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12628","reported_seq":16919,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.803077+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:55:57.803077+0000","last_peered":"2024-09-18T11:55:57.803077+0000","last_clean":"2024-09-18T11:55:57.803077+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:55:57.803077+0000","last_undegraded":"2024-09-18T11:55:57.803077+0000","last_fullsized":"2024-09-18T11:55:57.803077+0000","mapping_epoch":32,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2249974,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12626,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12485","reported_seq":12577,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:00.705812+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:00.705812+0000","last_peered":"2024-09-18T11:56:00.705812+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:00.705812+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1815798,"num_objects":222,"num_object_clones":0,"num_object_copies":888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12483,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1780}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33441164,"num_objects":4088,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4088,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202999,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8755,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":67690496,"data_stored":67690496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161026,"ondisk_log_size":161026,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944240,"kb_used_data":16548,"kb_used_omap":4,"kb_used_meta":927675,"kb_avail":93427600,"statfs":{"total":96636764160,"available":95669862400,"internally_reserved":0,"allocated":16945152,"data_stored":16752902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949939777},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,2,1,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":1.383},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":17.18}]},{"osd":1,"last update":"Wed Sep 18 11:55:02 2024","interfaces":[{"interface":"back","average":{"1min":7.2720000000000002,"5min":8.0549999999999997,"15min":6.4080000000000004},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":18.143000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":1.1579999999999999},{"interface":"front","average":{"1min":4.0640000000000001,"5min":6.5780000000000003,"15min":5.1580000000000004},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.257,"5min":57.372,"15min":57.372},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":11.119999999999999},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.23599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935776,"kb_used_data":16920,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93436064,"statfs":{"total":96636764160,"available":95678529536,"internally_reserved":0,"allocated":17326080,"data_stored":17133830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.29799999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.57399999999999995}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.39800000000000002},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.61299999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948600,"kb_used_data":18928,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93423240,"statfs":{"total":96636764160,"available":95665397760,"internally_reserved":0,"allocated":19382272,"data_stored":19190022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":16.074000000000002},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":11.750999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":2.6549999999999998},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":16.036999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":4.7000000000000002},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":11.704000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931248,"kb_used_data":16104,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93440592,"statfs":{"total":96636764160,"available":95683166208,"internally_reserved":0,"allocated":16490496,"data_stored":16298246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":11.356},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":11.441000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":11.164999999999999},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":11.403}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":7.1630000000000003},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15826944,"data_stored":15826944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18677760,"data_stored":18677760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16764928,"data_stored":16764928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16420864,"data_stored":16420864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:06.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:06.592 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:06.619 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-18T11:56:06.103996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25380236,"num_objects":3104,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203983,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8795,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3760464,"kb_used_data":52460,"kb_used_omap":14,"kb_used_meta":3707953,"kb_avail":373726896,"statfs":{"total":386547056640,"available":382696341504,"internally_reserved":0,"allocated":53719040,"data_stored":52950040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3796943899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,6,4,15,11,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22978560,"num_objects":-2805,"num_object_clones":0,"num_object_copies":-11220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2805,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002727"},"pg_stats":[{"pgid":"1.f","version":"39'13082","reported_seq":13185,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.049584+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:05.049584+0000","last_peered":"2024-09-18T11:56:05.049584+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:05.049584+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":168,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13082,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12840","reported_seq":17134,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.831159+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:02.831159+0000","last_peered":"2024-09-18T11:56:02.831159+0000","last_clean":"2024-09-18T11:56:02.831159+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:02.831159+0000","last_undegraded":"2024-09-18T11:56:02.831159+0000","last_fullsized":"2024-09-18T11:56:02.831159+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1766646,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12838,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12598","reported_seq":12997,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.635344+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:04.635344+0000","last_peered":"2024-09-18T11:56:04.635344+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:04.635344+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1718931,"num_objects":211,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12597,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1690}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12747","reported_seq":12841,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.626381+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:05.626381+0000","last_peered":"2024-09-18T11:56:05.626381+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:05.626381+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1463542,"num_objects":179,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12745,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1922}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12709","reported_seq":12832,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.883383+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:04.883383+0000","last_peered":"2024-09-18T11:56:04.883383+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:04.883383+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1520886,"num_objects":186,"num_object_clones":0,"num_object_copies":744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12707,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12607","reported_seq":12728,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.977538+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:04.977538+0000","last_peered":"2024-09-18T11:56:04.977538+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:04.977538+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1466368,"num_objects":179,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12607,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12864","reported_seq":12973,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.659716+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:05.659716+0000","last_peered":"2024-09-18T11:56:05.659716+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:05.659716+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1242358,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12862,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1964}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12654","reported_seq":16810,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.918442+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:02.918442+0000","last_peered":"2024-09-18T11:56:02.918442+0000","last_clean":"2024-09-18T11:56:02.918442+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:02.918442+0000","last_undegraded":"2024-09-18T11:56:02.918442+0000","last_fullsized":"2024-09-18T11:56:02.918442+0000","mapping_epoch":38,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1807606,"num_objects":221,"num_object_clones":0,"num_object_copies":884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12652,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12970","reported_seq":17325,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:04.979393+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:04.979393+0000","last_peered":"2024-09-18T11:56:04.979393+0000","last_clean":"2024-09-18T11:56:04.979393+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:04.979393+0000","last_undegraded":"2024-09-18T11:56:04.979393+0000","last_fullsized":"2024-09-18T11:56:04.979393+0000","mapping_epoch":36,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1681900,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12967,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12416","reported_seq":12532,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.531357+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:04.531357+0000","last_peered":"2024-09-18T11:56:04.531357+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:04.531357+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1455350,"num_objects":178,"num_object_clones":0,"num_object_copies":712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12414,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12840","reported_seq":12963,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.966527+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:04.966527+0000","last_peered":"2024-09-18T11:56:04.966527+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:04.966527+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1576817,"num_objects":193,"num_object_clones":0,"num_object_copies":772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12837,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12848","reported_seq":12942,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.018094+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:05.018094+0000","last_peered":"2024-09-18T11:56:05.018094+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:05.018094+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1497723,"num_objects":183,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12847,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1901}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12751","reported_seq":12851,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.464128+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:04.464128+0000","last_peered":"2024-09-18T11:56:04.464128+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:04.464128+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1768059,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12750,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12870","reported_seq":12997,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.984392+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:04.984392+0000","last_peered":"2024-09-18T11:56:04.984392+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:04.984392+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1799414,"num_objects":220,"num_object_clones":0,"num_object_copies":880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12868,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1903}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12676","reported_seq":16967,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.712305+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:02.712305+0000","last_peered":"2024-09-18T11:56:02.712305+0000","last_clean":"2024-09-18T11:56:02.712305+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:02.712305+0000","last_undegraded":"2024-09-18T11:56:02.712305+0000","last_fullsized":"2024-09-18T11:56:02.712305+0000","mapping_epoch":32,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1856758,"num_objects":227,"num_object_clones":0,"num_object_copies":908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12674,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12538","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.434585+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:05.434585+0000","last_peered":"2024-09-18T11:56:05.434585+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:05.434585+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1381622,"num_objects":169,"num_object_clones":0,"num_object_copies":676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1833}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":25380236,"num_objects":3104,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203983,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8795,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":51228672,"data_stored":51228672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944348,"kb_used_data":12500,"kb_used_omap":4,"kb_used_meta":931835,"kb_avail":93427492,"statfs":{"total":96636764160,"available":95669751808,"internally_reserved":0,"allocated":12800000,"data_stored":12607750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954199617},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":6.4210000000000003},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.4000000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":6.3639999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":2.532}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.28399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936004,"kb_used_data":12988,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93435836,"statfs":{"total":96636764160,"available":95678296064,"internally_reserved":0,"allocated":13299712,"data_stored":13107462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":6.2119999999999997},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":6.2439999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.233000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.311}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.83699999999999997},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.89500000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948644,"kb_used_data":14812,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93423196,"statfs":{"total":96636764160,"available":95665352704,"internally_reserved":0,"allocated":15167488,"data_stored":14975238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":2.9199999999999999},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.46500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":10.381}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.91800000000000004},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.515999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931468,"kb_used_data":12160,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93440372,"statfs":{"total":96636764160,"available":95682940928,"internally_reserved":0,"allocated":12451840,"data_stored":12259590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.8239999999999998},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":4.6180000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":4.5069999999999997},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":4.5869999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":5.359},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":18.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11739136,"data_stored":11739136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14557184,"data_stored":14557184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12627968,"data_stored":12627968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12304384,"data_stored":12304384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:06.620 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:07.186 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:07.186 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:07.221 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-18T11:56:06.103996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25380236,"num_objects":3104,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203983,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8795,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3760464,"kb_used_data":52460,"kb_used_omap":14,"kb_used_meta":3707953,"kb_avail":373726896,"statfs":{"total":386547056640,"available":382696341504,"internally_reserved":0,"allocated":53719040,"data_stored":52950040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3796943899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,6,4,15,11,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22978560,"num_objects":-2805,"num_object_clones":0,"num_object_copies":-11220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2805,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002727"},"pg_stats":[{"pgid":"1.f","version":"39'13082","reported_seq":13185,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.049584+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:05.049584+0000","last_peered":"2024-09-18T11:56:05.049584+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:05.049584+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":168,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13082,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12840","reported_seq":17134,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.831159+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:02.831159+0000","last_peered":"2024-09-18T11:56:02.831159+0000","last_clean":"2024-09-18T11:56:02.831159+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:02.831159+0000","last_undegraded":"2024-09-18T11:56:02.831159+0000","last_fullsized":"2024-09-18T11:56:02.831159+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1766646,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12838,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12598","reported_seq":12997,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.635344+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:04.635344+0000","last_peered":"2024-09-18T11:56:04.635344+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:04.635344+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1718931,"num_objects":211,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12597,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1690}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12747","reported_seq":12841,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.626381+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:05.626381+0000","last_peered":"2024-09-18T11:56:05.626381+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:05.626381+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1463542,"num_objects":179,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12745,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1922}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12709","reported_seq":12832,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.883383+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:04.883383+0000","last_peered":"2024-09-18T11:56:04.883383+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:04.883383+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1520886,"num_objects":186,"num_object_clones":0,"num_object_copies":744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12707,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12607","reported_seq":12728,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.977538+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:04.977538+0000","last_peered":"2024-09-18T11:56:04.977538+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:04.977538+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1466368,"num_objects":179,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12607,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12864","reported_seq":12973,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.659716+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:05.659716+0000","last_peered":"2024-09-18T11:56:05.659716+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:05.659716+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1242358,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12862,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1964}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12654","reported_seq":16810,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.918442+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:02.918442+0000","last_peered":"2024-09-18T11:56:02.918442+0000","last_clean":"2024-09-18T11:56:02.918442+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:02.918442+0000","last_undegraded":"2024-09-18T11:56:02.918442+0000","last_fullsized":"2024-09-18T11:56:02.918442+0000","mapping_epoch":38,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1807606,"num_objects":221,"num_object_clones":0,"num_object_copies":884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12652,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12970","reported_seq":17325,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:04.979393+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:04.979393+0000","last_peered":"2024-09-18T11:56:04.979393+0000","last_clean":"2024-09-18T11:56:04.979393+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:04.979393+0000","last_undegraded":"2024-09-18T11:56:04.979393+0000","last_fullsized":"2024-09-18T11:56:04.979393+0000","mapping_epoch":36,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1681900,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12967,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12416","reported_seq":12532,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.531357+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:04.531357+0000","last_peered":"2024-09-18T11:56:04.531357+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:04.531357+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1455350,"num_objects":178,"num_object_clones":0,"num_object_copies":712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12414,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12840","reported_seq":12963,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.966527+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:04.966527+0000","last_peered":"2024-09-18T11:56:04.966527+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:04.966527+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1576817,"num_objects":193,"num_object_clones":0,"num_object_copies":772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12837,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12848","reported_seq":12942,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.018094+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:05.018094+0000","last_peered":"2024-09-18T11:56:05.018094+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:05.018094+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1497723,"num_objects":183,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12847,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1901}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12751","reported_seq":12851,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.464128+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:04.464128+0000","last_peered":"2024-09-18T11:56:04.464128+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:04.464128+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1768059,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12750,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12870","reported_seq":12997,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.984392+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:04.984392+0000","last_peered":"2024-09-18T11:56:04.984392+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:04.984392+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1799414,"num_objects":220,"num_object_clones":0,"num_object_copies":880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12868,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1903}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12676","reported_seq":16967,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.712305+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:02.712305+0000","last_peered":"2024-09-18T11:56:02.712305+0000","last_clean":"2024-09-18T11:56:02.712305+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:02.712305+0000","last_undegraded":"2024-09-18T11:56:02.712305+0000","last_fullsized":"2024-09-18T11:56:02.712305+0000","mapping_epoch":32,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1856758,"num_objects":227,"num_object_clones":0,"num_object_copies":908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12674,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12538","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.434585+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:05.434585+0000","last_peered":"2024-09-18T11:56:05.434585+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:05.434585+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1381622,"num_objects":169,"num_object_clones":0,"num_object_copies":676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1833}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":25380236,"num_objects":3104,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203983,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8795,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":51228672,"data_stored":51228672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944348,"kb_used_data":12500,"kb_used_omap":4,"kb_used_meta":931835,"kb_avail":93427492,"statfs":{"total":96636764160,"available":95669751808,"internally_reserved":0,"allocated":12800000,"data_stored":12607750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954199617},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":6.4210000000000003},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.4000000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":6.3639999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":2.532}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.28399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936004,"kb_used_data":12988,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93435836,"statfs":{"total":96636764160,"available":95678296064,"internally_reserved":0,"allocated":13299712,"data_stored":13107462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":6.2119999999999997},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":6.2439999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.233000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.311}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.83699999999999997},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.89500000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948644,"kb_used_data":14812,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93423196,"statfs":{"total":96636764160,"available":95665352704,"internally_reserved":0,"allocated":15167488,"data_stored":14975238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":2.9199999999999999},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.46500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":10.381}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.91800000000000004},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":19.515999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931468,"kb_used_data":12160,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93440372,"statfs":{"total":96636764160,"available":95682940928,"internally_reserved":0,"allocated":12451840,"data_stored":12259590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.8239999999999998},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":4.6180000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":4.5069999999999997},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":4.5869999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":5.359},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":18.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11739136,"data_stored":11739136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14557184,"data_stored":14557184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12627968,"data_stored":12627968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12304384,"data_stored":12304384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:07.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:07.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13082", 'reported_seq': 13185, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:05.049584+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:05.049584+0000', 'last_peered': '2024-09-18T11:56:05.049584+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:05.049584+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1376256, 'num_objects': 168, 'num_object_clones': 0, 'num_object_copies': 672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13082, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1935}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:07.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:56:07.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12598", 'reported_seq': 12997, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:04.635344+0000', 'last_change': '2024-09-18T11:55:51.694654+0000', 'last_active': '2024-09-18T11:56:04.635344+0000', 'last_peered': '2024-09-18T11:56:04.635344+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:56:04.635344+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1718931, 'num_objects': 211, 'num_object_clones': 0, 'num_object_copies': 844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1901, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 211, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12597, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 155, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1690}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:07.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:07.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12747", 'reported_seq': 12841, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:05.626381+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:05.626381+0000', 'last_peered': '2024-09-18T11:56:05.626381+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:05.626381+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1463542, 'num_objects': 179, 'num_object_clones': 0, 'num_object_copies': 716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12745, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1922}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:07.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12709", 'reported_seq': 12832, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:04.883383+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:04.883383+0000', 'last_peered': '2024-09-18T11:56:04.883383+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:04.883383+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1520886, 'num_objects': 186, 'num_object_clones': 0, 'num_object_copies': 744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12707, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1866}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:07.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:07.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12607", 'reported_seq': 12728, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:04.977538+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:04.977538+0000', 'last_peered': '2024-09-18T11:56:04.977538+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:04.977538+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1466368, 'num_objects': 179, 'num_object_clones': 0, 'num_object_copies': 716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12607, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1940}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:07.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:07.225 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12864", 'reported_seq': 12973, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:05.659716+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:56:05.659716+0000', 'last_peered': '2024-09-18T11:56:05.659716+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:05.659716+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1242358, 'num_objects': 152, 'num_object_clones': 0, 'num_object_copies': 608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12862, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1964}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.225 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:07.225 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12416", 'reported_seq': 12532, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:04.531357+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:04.531357+0000', 'last_peered': '2024-09-18T11:56:04.531357+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:04.531357+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1455350, 'num_objects': 178, 'num_object_clones': 0, 'num_object_copies': 712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12414, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1785}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:07.225 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:07.225 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12840", 'reported_seq': 12963, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:04.966527+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:56:04.966527+0000', 'last_peered': '2024-09-18T11:56:04.966527+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:56:04.966527+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1576817, 'num_objects': 193, 'num_object_clones': 0, 'num_object_copies': 772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12837, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1841}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:07.225 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:07.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12848", 'reported_seq': 12942, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:05.018094+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:05.018094+0000', 'last_peered': '2024-09-18T11:56:05.018094+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:05.018094+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1497723, 'num_objects': 183, 'num_object_clones': 0, 'num_object_copies': 732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12847, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:07.226 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:07.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12751", 'reported_seq': 12851, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:04.464128+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:04.464128+0000', 'last_peered': '2024-09-18T11:56:04.464128+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:04.464128+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1768059, 'num_objects': 216, 'num_object_clones': 0, 'num_object_copies': 864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12750, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1830}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:07.226 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:07.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12870", 'reported_seq': 12997, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:04.984392+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:04.984392+0000', 'last_peered': '2024-09-18T11:56:04.984392+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:04.984392+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1799414, 'num_objects': 220, 'num_object_clones': 0, 'num_object_copies': 880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12868, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1903}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:07.226 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:56:07.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12538", 'reported_seq': 12630, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:05.434585+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:56:05.434585+0000', 'last_peered': '2024-09-18T11:56:05.434585+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:56:05.434585+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1381622, 'num_objects': 169, 'num_object_clones': 0, 'num_object_copies': 676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12536, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1833}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:07.883 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:07.883 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24661/12416 objects degraded (198.623%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":765,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":3104,"data_bytes":25380236,"bytes_used":3850715136,"bytes_avail":382696341504,"bytes_total":386547056640,"degraded_objects":24661,"degraded_total":12416,"degraded_ratio":1.9862274484536082,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":233},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:07.913 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:07.914 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:08.520 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:08.536 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-18T11:56:08.104511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23831948,"num_objects":2915,"num_object_clones":0,"num_object_copies":11660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204172,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8795,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160699,"ondisk_log_size":160699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3743940,"kb_used_data":48416,"kb_used_omap":14,"kb_used_meta":3695473,"kb_avail":373743420,"statfs":{"total":386547056640,"available":382713262080,"internally_reserved":0,"allocated":49577984,"data_stored":48808984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3784164379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,9,3,7,16,12,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17989632,"num_objects":-2196,"num_object_clones":0,"num_object_copies":-8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2196,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002825"},"pg_stats":[{"pgid":"1.f","version":"39'13082","reported_seq":13185,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.049584+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:05.049584+0000","last_peered":"2024-09-18T11:56:05.049584+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:05.049584+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":168,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13082,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12907","reported_seq":17201,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.785124+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:07.785124+0000","last_peered":"2024-09-18T11:56:07.785124+0000","last_clean":"2024-09-18T11:56:07.785124+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:07.785124+0000","last_undegraded":"2024-09-18T11:56:07.785124+0000","last_fullsized":"2024-09-18T11:56:07.785124+0000","mapping_epoch":34,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1217782,"num_objects":149,"num_object_clones":0,"num_object_copies":596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12905,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12598","reported_seq":12997,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.635344+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:04.635344+0000","last_peered":"2024-09-18T11:56:04.635344+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:04.635344+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1718931,"num_objects":211,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12597,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1690}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12747","reported_seq":12841,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.626381+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:05.626381+0000","last_peered":"2024-09-18T11:56:05.626381+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:05.626381+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1463542,"num_objects":179,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12745,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1922}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12709","reported_seq":12832,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.883383+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:04.883383+0000","last_peered":"2024-09-18T11:56:04.883383+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:04.883383+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1520886,"num_objects":186,"num_object_clones":0,"num_object_copies":744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12707,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12607","reported_seq":12728,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.977538+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:04.977538+0000","last_peered":"2024-09-18T11:56:04.977538+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:04.977538+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1466368,"num_objects":179,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12607,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12864","reported_seq":12973,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.659716+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:05.659716+0000","last_peered":"2024-09-18T11:56:05.659716+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:05.659716+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1242358,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12862,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1964}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12707","reported_seq":16862,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.806369+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:07.806369+0000","last_peered":"2024-09-18T11:56:07.806369+0000","last_clean":"2024-09-18T11:56:07.806369+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:07.806369+0000","last_undegraded":"2024-09-18T11:56:07.806369+0000","last_fullsized":"2024-09-18T11:56:07.806369+0000","mapping_epoch":38,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1373430,"num_objects":168,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12705,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12970","reported_seq":17325,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:04.979393+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:04.979393+0000","last_peered":"2024-09-18T11:56:04.979393+0000","last_clean":"2024-09-18T11:56:04.979393+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:04.979393+0000","last_undegraded":"2024-09-18T11:56:04.979393+0000","last_fullsized":"2024-09-18T11:56:04.979393+0000","mapping_epoch":36,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1681900,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12967,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12416","reported_seq":12532,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.531357+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:04.531357+0000","last_peered":"2024-09-18T11:56:04.531357+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:04.531357+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1455350,"num_objects":178,"num_object_clones":0,"num_object_copies":712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12414,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12840","reported_seq":12963,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.966527+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:04.966527+0000","last_peered":"2024-09-18T11:56:04.966527+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:04.966527+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1576817,"num_objects":193,"num_object_clones":0,"num_object_copies":772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12837,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12848","reported_seq":12942,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.018094+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:05.018094+0000","last_peered":"2024-09-18T11:56:05.018094+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:05.018094+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1497723,"num_objects":183,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12847,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1901}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12751","reported_seq":12851,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.464128+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:04.464128+0000","last_peered":"2024-09-18T11:56:04.464128+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:04.464128+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1768059,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12750,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12870","reported_seq":12997,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:04.984392+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:04.984392+0000","last_peered":"2024-09-18T11:56:04.984392+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:04.984392+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1799414,"num_objects":220,"num_object_clones":0,"num_object_copies":880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12868,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1903}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12745","reported_seq":17036,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.830038+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:07.830038+0000","last_peered":"2024-09-18T11:56:07.830038+0000","last_clean":"2024-09-18T11:56:07.830038+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:07.830038+0000","last_undegraded":"2024-09-18T11:56:07.830038+0000","last_fullsized":"2024-09-18T11:56:07.830038+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1291510,"num_objects":158,"num_object_clones":0,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12743,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12538","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:05.434585+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:05.434585+0000","last_peered":"2024-09-18T11:56:05.434585+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:05.434585+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1381622,"num_objects":169,"num_object_clones":0,"num_object_copies":676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1833}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":23831948,"num_objects":2915,"num_object_clones":0,"num_object_copies":11660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204172,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8795,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":47063040,"data_stored":47063040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160699,"ondisk_log_size":160699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944348,"kb_used_data":12500,"kb_used_omap":4,"kb_used_meta":931835,"kb_avail":93427492,"statfs":{"total":96636764160,"available":95669751808,"internally_reserved":0,"allocated":12800000,"data_stored":12607750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954199617},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":6.4210000000000003},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":6.4000000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":6.3639999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":2.532}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.28399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936004,"kb_used_data":12988,"kb_used_omap":1,"kb_used_meta":923006,"kb_avail":93435836,"statfs":{"total":96636764160,"available":95678296064,"internally_reserved":0,"allocated":13299712,"data_stored":13107462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945158488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":6.2119999999999997},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":6.2439999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":8.0850000000000009,"5min":8.4090000000000007,"15min":6.7759999999999998},"min":{"1min":0.38100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":21.117999999999999,"5min":46.793999999999997,"15min":46.793999999999997},"last":11.233000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":5.1440000000000001,"15min":4.4240000000000004},"min":{"1min":0.35499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":15.802,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.311}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.83699999999999997},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.89500000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932120,"kb_used_data":10768,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93439720,"statfs":{"total":96636764160,"available":95682273280,"internally_reserved":0,"allocated":11026432,"data_stored":10834182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,0,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":13.15},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":6.165}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":6.9610000000000003},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":6.3920000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":6.2569999999999997},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":7.101}]}]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931468,"kb_used_data":12160,"kb_used_omap":4,"kb_used_meta":919291,"kb_avail":93440372,"statfs":{"total":96636764160,"available":95682940928,"internally_reserved":0,"allocated":12451840,"data_stored":12259590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941354561},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:55:05 2024","interfaces":[{"interface":"back","average":{"1min":7.4619999999999997,"5min":7.8479999999999999,"15min":5.3659999999999997},"min":{"1min":0.23000000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":18.478999999999999,"5min":28.059000000000001,"15min":28.059000000000001},"last":7.8239999999999998},{"interface":"front","average":{"1min":8.9589999999999996,"5min":6.5890000000000004,"15min":4.9939999999999998},"min":{"1min":0.89500000000000002,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":27.445,"5min":34.786999999999999,"15min":34.786999999999999},"last":4.6180000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":4.5069999999999997},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":4.5869999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":5.359},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":18.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11739136,"data_stored":11739136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10391552,"data_stored":10391552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12627968,"data_stored":12627968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12304384,"data_stored":12304384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:11.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:12.128 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:12.128 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:12.144 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-18T11:56:12.105417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17139084,"num_objects":2098,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204989,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8828,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3727684,"kb_used_data":36288,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373759676,"statfs":{"total":386547056640,"available":382729908224,"internally_reserved":0,"allocated":37158912,"data_stored":36389912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,2,8,14,15,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18022400,"num_objects":-2200,"num_object_clones":0,"num_object_copies":-8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-73,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2200,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002807"},"pg_stats":[{"pgid":"1.f","version":"39'13135","reported_seq":13238,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.908939+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:09.908939+0000","last_peered":"2024-09-18T11:56:09.908939+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:09.908939+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":942080,"num_objects":115,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13135,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12907","reported_seq":17201,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.785124+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:07.785124+0000","last_peered":"2024-09-18T11:56:07.785124+0000","last_clean":"2024-09-18T11:56:07.785124+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:07.785124+0000","last_undegraded":"2024-09-18T11:56:07.785124+0000","last_fullsized":"2024-09-18T11:56:07.785124+0000","mapping_epoch":34,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1217782,"num_objects":149,"num_object_clones":0,"num_object_copies":596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12905,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12683","reported_seq":13149,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.755225+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:09.755225+0000","last_peered":"2024-09-18T11:56:09.755225+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:09.755225+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1022611,"num_objects":126,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12682,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1742}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12800","reported_seq":12896,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.632792+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:10.632792+0000","last_peered":"2024-09-18T11:56:10.632792+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:10.632792+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029366,"num_objects":126,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12798,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12768","reported_seq":12891,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.062796+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:10.062796+0000","last_peered":"2024-09-18T11:56:10.062796+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:10.062796+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1037558,"num_objects":127,"num_object_clones":0,"num_object_copies":508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12766,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12670","reported_seq":12791,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.151522+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:10.151522+0000","last_peered":"2024-09-18T11:56:10.151522+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:10.151522+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":950272,"num_objects":116,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12902","reported_seq":13011,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.603444+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:10.603444+0000","last_peered":"2024-09-18T11:56:10.603444+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:10.603444+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":931062,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12900,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2002}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12707","reported_seq":16862,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.806369+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:07.806369+0000","last_peered":"2024-09-18T11:56:07.806369+0000","last_clean":"2024-09-18T11:56:07.806369+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:07.806369+0000","last_undegraded":"2024-09-18T11:56:07.806369+0000","last_fullsized":"2024-09-18T11:56:07.806369+0000","mapping_epoch":38,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1373430,"num_objects":168,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12705,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13020","reported_seq":17375,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:10.125443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:10.125443+0000","last_peered":"2024-09-18T11:56:10.125443+0000","last_clean":"2024-09-18T11:56:10.125443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:10.125443+0000","last_undegraded":"2024-09-18T11:56:10.125443+0000","last_fullsized":"2024-09-18T11:56:10.125443+0000","mapping_epoch":36,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1272300,"num_objects":156,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13017,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12477","reported_seq":12593,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.733348+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:09.733348+0000","last_peered":"2024-09-18T11:56:09.733348+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:09.733348+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":955638,"num_objects":117,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12475,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1846}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12905","reported_seq":13028,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.053117+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:10.053117+0000","last_peered":"2024-09-18T11:56:10.053117+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:10.053117+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1044337,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12902,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12907","reported_seq":13000,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.150267+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:10.150267+0000","last_peered":"2024-09-18T11:56:10.150267+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:10.150267+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1014395,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12906,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12836","reported_seq":12936,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.774627+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:09.774627+0000","last_peered":"2024-09-18T11:56:09.774627+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:09.774627+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1071739,"num_objects":131,"num_object_clones":0,"num_object_copies":524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12835,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1915}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12949","reported_seq":13074,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.962752+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:09.962752+0000","last_peered":"2024-09-18T11:56:09.962752+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:09.962752+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152246,"num_objects":141,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12947,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12745","reported_seq":17036,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.830038+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:07.830038+0000","last_peered":"2024-09-18T11:56:07.830038+0000","last_clean":"2024-09-18T11:56:07.830038+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:07.830038+0000","last_undegraded":"2024-09-18T11:56:07.830038+0000","last_fullsized":"2024-09-18T11:56:07.830038+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1291510,"num_objects":158,"num_object_clones":0,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12743,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12605","reported_seq":12697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.598592+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:10.598592+0000","last_peered":"2024-09-18T11:56:10.598592+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:10.598592+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":832758,"num_objects":102,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12603,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":17139084,"num_objects":2098,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204989,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8828,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34660352,"data_stored":34660352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927828,"kb_used_data":8436,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93444012,"statfs":{"total":96636764160,"available":95686668288,"internally_reserved":0,"allocated":8638464,"data_stored":8446214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,6,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":15.975},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":16.006}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":16.033999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":3.8330000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":16.292000000000002},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":4.9139999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936172,"kb_used_data":8988,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93435668,"statfs":{"total":96636764160,"available":95678124032,"internally_reserved":0,"allocated":9203712,"data_stored":9011462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,5,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":3.5640000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":3.589}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":3.6139999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":3.5230000000000001},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":1.9650000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932120,"kb_used_data":10768,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93439720,"statfs":{"total":96636764160,"available":95682273280,"internally_reserved":0,"allocated":11026432,"data_stored":10834182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,0,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":13.15},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":6.165}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":6.9610000000000003},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":6.3920000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":6.2569999999999997},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":7.101}]}]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931564,"kb_used_data":8096,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93440276,"statfs":{"total":96636764160,"available":95682842624,"internally_reserved":0,"allocated":8290304,"data_stored":8098054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":25.125},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":7.4470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":4.2779999999999996},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":7.508}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":25.166},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":7.4829999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7634944,"data_stored":7634944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10391552,"data_stored":10391552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8462336,"data_stored":8462336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8171520,"data_stored":8171520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:12.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:12.721 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:12.721 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:12.741 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-18T11:56:12.105417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17139084,"num_objects":2098,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204989,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8828,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3727684,"kb_used_data":36288,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373759676,"statfs":{"total":386547056640,"available":382729908224,"internally_reserved":0,"allocated":37158912,"data_stored":36389912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,2,8,14,15,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18022400,"num_objects":-2200,"num_object_clones":0,"num_object_copies":-8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-73,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2200,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002807"},"pg_stats":[{"pgid":"1.f","version":"39'13135","reported_seq":13238,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.908939+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:09.908939+0000","last_peered":"2024-09-18T11:56:09.908939+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:09.908939+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":942080,"num_objects":115,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13135,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12907","reported_seq":17201,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.785124+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:07.785124+0000","last_peered":"2024-09-18T11:56:07.785124+0000","last_clean":"2024-09-18T11:56:07.785124+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:07.785124+0000","last_undegraded":"2024-09-18T11:56:07.785124+0000","last_fullsized":"2024-09-18T11:56:07.785124+0000","mapping_epoch":34,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1217782,"num_objects":149,"num_object_clones":0,"num_object_copies":596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12905,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12683","reported_seq":13149,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.755225+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:09.755225+0000","last_peered":"2024-09-18T11:56:09.755225+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:09.755225+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1022611,"num_objects":126,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12682,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1742}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12800","reported_seq":12896,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.632792+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:10.632792+0000","last_peered":"2024-09-18T11:56:10.632792+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:10.632792+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029366,"num_objects":126,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12798,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12768","reported_seq":12891,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.062796+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:10.062796+0000","last_peered":"2024-09-18T11:56:10.062796+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:10.062796+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1037558,"num_objects":127,"num_object_clones":0,"num_object_copies":508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12766,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12670","reported_seq":12791,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.151522+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:10.151522+0000","last_peered":"2024-09-18T11:56:10.151522+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:10.151522+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":950272,"num_objects":116,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12902","reported_seq":13011,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.603444+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:10.603444+0000","last_peered":"2024-09-18T11:56:10.603444+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:10.603444+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":931062,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12900,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2002}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12707","reported_seq":16862,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.806369+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:07.806369+0000","last_peered":"2024-09-18T11:56:07.806369+0000","last_clean":"2024-09-18T11:56:07.806369+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:07.806369+0000","last_undegraded":"2024-09-18T11:56:07.806369+0000","last_fullsized":"2024-09-18T11:56:07.806369+0000","mapping_epoch":38,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1373430,"num_objects":168,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12705,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13020","reported_seq":17375,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:10.125443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:10.125443+0000","last_peered":"2024-09-18T11:56:10.125443+0000","last_clean":"2024-09-18T11:56:10.125443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:10.125443+0000","last_undegraded":"2024-09-18T11:56:10.125443+0000","last_fullsized":"2024-09-18T11:56:10.125443+0000","mapping_epoch":36,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1272300,"num_objects":156,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13017,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12477","reported_seq":12593,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.733348+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:09.733348+0000","last_peered":"2024-09-18T11:56:09.733348+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:09.733348+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":955638,"num_objects":117,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12475,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1846}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12905","reported_seq":13028,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.053117+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:10.053117+0000","last_peered":"2024-09-18T11:56:10.053117+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:10.053117+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1044337,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12902,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12907","reported_seq":13000,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.150267+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:10.150267+0000","last_peered":"2024-09-18T11:56:10.150267+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:10.150267+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1014395,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12906,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12836","reported_seq":12936,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.774627+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:09.774627+0000","last_peered":"2024-09-18T11:56:09.774627+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:09.774627+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1071739,"num_objects":131,"num_object_clones":0,"num_object_copies":524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12835,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1915}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12949","reported_seq":13074,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.962752+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:09.962752+0000","last_peered":"2024-09-18T11:56:09.962752+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:09.962752+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152246,"num_objects":141,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12947,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12745","reported_seq":17036,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.830038+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:07.830038+0000","last_peered":"2024-09-18T11:56:07.830038+0000","last_clean":"2024-09-18T11:56:07.830038+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:07.830038+0000","last_undegraded":"2024-09-18T11:56:07.830038+0000","last_fullsized":"2024-09-18T11:56:07.830038+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1291510,"num_objects":158,"num_object_clones":0,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12743,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12605","reported_seq":12697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.598592+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:10.598592+0000","last_peered":"2024-09-18T11:56:10.598592+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:10.598592+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":832758,"num_objects":102,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12603,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":17139084,"num_objects":2098,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204989,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8828,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34660352,"data_stored":34660352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927828,"kb_used_data":8436,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93444012,"statfs":{"total":96636764160,"available":95686668288,"internally_reserved":0,"allocated":8638464,"data_stored":8446214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,6,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":15.975},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":16.006}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":16.033999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":3.8330000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":16.292000000000002},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":4.9139999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936172,"kb_used_data":8988,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93435668,"statfs":{"total":96636764160,"available":95678124032,"internally_reserved":0,"allocated":9203712,"data_stored":9011462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,5,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":3.5640000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":3.589}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":3.6139999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":3.5230000000000001},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":1.9650000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932120,"kb_used_data":10768,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93439720,"statfs":{"total":96636764160,"available":95682273280,"internally_reserved":0,"allocated":11026432,"data_stored":10834182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,0,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":13.15},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":6.165}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":6.9610000000000003},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":6.3920000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":6.2569999999999997},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":7.101}]}]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931564,"kb_used_data":8096,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93440276,"statfs":{"total":96636764160,"available":95682842624,"internally_reserved":0,"allocated":8290304,"data_stored":8098054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":25.125},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":7.4470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":4.2779999999999996},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":7.508}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":25.166},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":7.4829999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7634944,"data_stored":7634944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10391552,"data_stored":10391552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8462336,"data_stored":8462336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8171520,"data_stored":8171520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:12.742 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:12.742 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13135", 'reported_seq': 13238, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:09.908939+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:09.908939+0000', 'last_peered': '2024-09-18T11:56:09.908939+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:09.908939+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3100", 'ondisk_log_start': "31'3100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 942080, 'num_objects': 115, 'num_object_clones': 0, 'num_object_copies': 460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 115, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13135, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1988}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:12.742 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:56:12.742 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12683", 'reported_seq': 13149, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:09.755225+0000', 'last_change': '2024-09-18T11:55:51.694654+0000', 'last_active': '2024-09-18T11:56:09.755225+0000', 'last_peered': '2024-09-18T11:56:09.755225+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:56:09.755225+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1022611, 'num_objects': 126, 'num_object_clones': 0, 'num_object_copies': 504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1868, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 126, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12682, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 188, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1742}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:12.743 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:12.743 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12800", 'reported_seq': 12896, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:10.632792+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:10.632792+0000', 'last_peered': '2024-09-18T11:56:10.632792+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:10.632792+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1029366, 'num_objects': 126, 'num_object_clones': 0, 'num_object_copies': 504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12798, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.743 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:12.743 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12768", 'reported_seq': 12891, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:10.062796+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:10.062796+0000', 'last_peered': '2024-09-18T11:56:10.062796+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:10.062796+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1037558, 'num_objects': 127, 'num_object_clones': 0, 'num_object_copies': 508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12766, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1925}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:12.743 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:12.743 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12670", 'reported_seq': 12791, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:10.151522+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:10.151522+0000', 'last_peered': '2024-09-18T11:56:10.151522+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:10.151522+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 950272, 'num_objects': 116, 'num_object_clones': 0, 'num_object_copies': 464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2003}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:12.743 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:12.743 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12902", 'reported_seq': 13011, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:10.603444+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:56:10.603444+0000', 'last_peered': '2024-09-18T11:56:10.603444+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:10.603444+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 931062, 'num_objects': 114, 'num_object_clones': 0, 'num_object_copies': 456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12900, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2002}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.743 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:12.743 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12477", 'reported_seq': 12593, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:09.733348+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:09.733348+0000', 'last_peered': '2024-09-18T11:56:09.733348+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:09.733348+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 955638, 'num_objects': 117, 'num_object_clones': 0, 'num_object_copies': 468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12475, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1846}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:12.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:12.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12905", 'reported_seq': 13028, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:10.053117+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:56:10.053117+0000', 'last_peered': '2024-09-18T11:56:10.053117+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:56:10.053117+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1044337, 'num_objects': 128, 'num_object_clones': 0, 'num_object_copies': 512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12902, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1906}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:12.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:12.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12907", 'reported_seq': 13000, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:10.150267+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:10.150267+0000', 'last_peered': '2024-09-18T11:56:10.150267+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:10.150267+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1014395, 'num_objects': 124, 'num_object_clones': 0, 'num_object_copies': 496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12906, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1960}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:12.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:12.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12836", 'reported_seq': 12936, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:09.774627+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:09.774627+0000', 'last_peered': '2024-09-18T11:56:09.774627+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:09.774627+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1071739, 'num_objects': 131, 'num_object_clones': 0, 'num_object_copies': 524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12835, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1915}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:12.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:12.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12949", 'reported_seq': 13074, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:09.962752+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:09.962752+0000', 'last_peered': '2024-09-18T11:56:09.962752+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:09.962752+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1152246, 'num_objects': 141, 'num_object_clones': 0, 'num_object_copies': 564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12947, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1982}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:12.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:56:12.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12605", 'reported_seq': 12697, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:10.598592+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:56:10.598592+0000', 'last_peered': '2024-09-18T11:56:10.598592+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:56:10.598592+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 832758, 'num_objects': 102, 'num_object_clones': 0, 'num_object_copies': 408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12603, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:13.472 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:13.472 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24628/8392 objects degraded (293.470%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":771,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":2098,"data_bytes":17139084,"bytes_used":3817148416,"bytes_avail":382729908224,"bytes_total":386547056640,"degraded_objects":24628,"degraded_total":8392,"degraded_ratio":2.934699714013346,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":183},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:13.490 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:13.490 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:14.073 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:14.074 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:14.093 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-18T11:56:12.105417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17139084,"num_objects":2098,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204989,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8828,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3727684,"kb_used_data":36288,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373759676,"statfs":{"total":386547056640,"available":382729908224,"internally_reserved":0,"allocated":37158912,"data_stored":36389912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,2,8,14,15,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18022400,"num_objects":-2200,"num_object_clones":0,"num_object_copies":-8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-73,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2200,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002807"},"pg_stats":[{"pgid":"1.f","version":"39'13135","reported_seq":13238,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.908939+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:09.908939+0000","last_peered":"2024-09-18T11:56:09.908939+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:09.908939+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":942080,"num_objects":115,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13135,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12907","reported_seq":17201,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.785124+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:07.785124+0000","last_peered":"2024-09-18T11:56:07.785124+0000","last_clean":"2024-09-18T11:56:07.785124+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:07.785124+0000","last_undegraded":"2024-09-18T11:56:07.785124+0000","last_fullsized":"2024-09-18T11:56:07.785124+0000","mapping_epoch":34,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1217782,"num_objects":149,"num_object_clones":0,"num_object_copies":596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12905,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12683","reported_seq":13149,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.755225+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:09.755225+0000","last_peered":"2024-09-18T11:56:09.755225+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:09.755225+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1022611,"num_objects":126,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12682,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1742}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12800","reported_seq":12896,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.632792+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:10.632792+0000","last_peered":"2024-09-18T11:56:10.632792+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:10.632792+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029366,"num_objects":126,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12798,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12768","reported_seq":12891,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.062796+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:10.062796+0000","last_peered":"2024-09-18T11:56:10.062796+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:10.062796+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1037558,"num_objects":127,"num_object_clones":0,"num_object_copies":508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12766,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12670","reported_seq":12791,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.151522+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:10.151522+0000","last_peered":"2024-09-18T11:56:10.151522+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:10.151522+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":950272,"num_objects":116,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12902","reported_seq":13011,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.603444+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:10.603444+0000","last_peered":"2024-09-18T11:56:10.603444+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:10.603444+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":931062,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12900,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2002}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12707","reported_seq":16862,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.806369+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:07.806369+0000","last_peered":"2024-09-18T11:56:07.806369+0000","last_clean":"2024-09-18T11:56:07.806369+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:07.806369+0000","last_undegraded":"2024-09-18T11:56:07.806369+0000","last_fullsized":"2024-09-18T11:56:07.806369+0000","mapping_epoch":38,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1373430,"num_objects":168,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12705,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13020","reported_seq":17375,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:10.125443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:10.125443+0000","last_peered":"2024-09-18T11:56:10.125443+0000","last_clean":"2024-09-18T11:56:10.125443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:10.125443+0000","last_undegraded":"2024-09-18T11:56:10.125443+0000","last_fullsized":"2024-09-18T11:56:10.125443+0000","mapping_epoch":36,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1272300,"num_objects":156,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13017,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12477","reported_seq":12593,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.733348+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:09.733348+0000","last_peered":"2024-09-18T11:56:09.733348+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:09.733348+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":955638,"num_objects":117,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12475,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1846}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12905","reported_seq":13028,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.053117+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:10.053117+0000","last_peered":"2024-09-18T11:56:10.053117+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:10.053117+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1044337,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12902,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12907","reported_seq":13000,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.150267+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:10.150267+0000","last_peered":"2024-09-18T11:56:10.150267+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:10.150267+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1014395,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12906,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12836","reported_seq":12936,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.774627+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:09.774627+0000","last_peered":"2024-09-18T11:56:09.774627+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:09.774627+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1071739,"num_objects":131,"num_object_clones":0,"num_object_copies":524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12835,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1915}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'12949","reported_seq":13074,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:09.962752+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:09.962752+0000","last_peered":"2024-09-18T11:56:09.962752+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:09.962752+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152246,"num_objects":141,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12947,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12745","reported_seq":17036,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.830038+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:07.830038+0000","last_peered":"2024-09-18T11:56:07.830038+0000","last_clean":"2024-09-18T11:56:07.830038+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:07.830038+0000","last_undegraded":"2024-09-18T11:56:07.830038+0000","last_fullsized":"2024-09-18T11:56:07.830038+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1291510,"num_objects":158,"num_object_clones":0,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12743,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12605","reported_seq":12697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:10.598592+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:10.598592+0000","last_peered":"2024-09-18T11:56:10.598592+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:10.598592+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":832758,"num_objects":102,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12603,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":17139084,"num_objects":2098,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204989,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8828,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34660352,"data_stored":34660352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927828,"kb_used_data":8436,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93444012,"statfs":{"total":96636764160,"available":95686668288,"internally_reserved":0,"allocated":8638464,"data_stored":8446214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,6,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":15.975},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":16.006}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":16.033999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":3.8330000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":16.292000000000002},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":4.9139999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936172,"kb_used_data":8988,"kb_used_omap":1,"kb_used_meta":927166,"kb_avail":93435668,"statfs":{"total":96636764160,"available":95678124032,"internally_reserved":0,"allocated":9203712,"data_stored":9011462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949418328},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,5,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":3.5640000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":3.589}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":3.6139999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":3.5230000000000001},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":1.9650000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932120,"kb_used_data":10768,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93439720,"statfs":{"total":96636764160,"available":95682273280,"internally_reserved":0,"allocated":11026432,"data_stored":10834182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,0,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":10.305,"5min":7.4329999999999998,"15min":5.1070000000000002},"min":{"1min":0.442,"5min":0.247,"15min":0.159},"max":{"1min":36.302999999999997,"5min":36.302999999999997,"15min":45.462000000000003},"last":13.15},{"interface":"front","average":{"1min":5.2389999999999999,"5min":6.8520000000000003,"15min":5.641},"min":{"1min":0.498,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":21.343,"5min":54.710999999999999,"15min":54.710999999999999},"last":6.165}]},{"osd":2,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":7.3710000000000004,"5min":7.7110000000000003,"15min":5.7409999999999997},"min":{"1min":0.48799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":19.382000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":6.9610000000000003},{"interface":"front","average":{"1min":6.8680000000000003,"5min":6.0149999999999997,"15min":4.7519999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.19},"max":{"1min":21.065999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":6.3920000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:55:08 2024","interfaces":[{"interface":"back","average":{"1min":6.8339999999999996,"5min":6.5049999999999999,"15min":5.0700000000000003},"min":{"1min":0.35499999999999998,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":43.334000000000003,"5min":43.334000000000003,"15min":43.334000000000003},"last":6.2569999999999997},{"interface":"front","average":{"1min":8.2230000000000008,"5min":7.7489999999999997,"15min":6.0640000000000001},"min":{"1min":0.37,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.484000000000002,"5min":45.786999999999999,"15min":45.786999999999999},"last":7.101}]}]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931564,"kb_used_data":8096,"kb_used_omap":4,"kb_used_meta":923451,"kb_avail":93440276,"statfs":{"total":96636764160,"available":95682842624,"internally_reserved":0,"allocated":8290304,"data_stored":8098054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945614401},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":25.125},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":7.4470000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":4.2779999999999996},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":7.508}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":25.166},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":7.4829999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7634944,"data_stored":7634944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10391552,"data_stored":10391552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8462336,"data_stored":8462336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8171520,"data_stored":8171520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:17.095 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:17.655 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:17.655 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:17.673 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-09-18T11:56:16.106323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8627596,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206028,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8872,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3724112,"kb_used_data":20272,"kb_used_omap":14,"kb_used_meta":3703793,"kb_avail":373763248,"statfs":{"total":386547056640,"available":382733565952,"internally_reserved":0,"allocated":20758528,"data_stored":19989528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792684059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,8,18,11,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23453696,"num_objects":-2863,"num_object_clones":0,"num_object_copies":-11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2863,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002772"},"pg_stats":[{"pgid":"1.f","version":"39'13191","reported_seq":13293,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.090218+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:15.090218+0000","last_peered":"2024-09-18T11:56:15.090218+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:15.090218+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483328,"num_objects":59,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13191,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12969","reported_seq":17263,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:13.079360+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:13.079360+0000","last_peered":"2024-09-18T11:56:13.079360+0000","last_clean":"2024-09-18T11:56:13.079360+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:13.079360+0000","last_undegraded":"2024-09-18T11:56:13.079360+0000","last_fullsized":"2024-09-18T11:56:13.079360+0000","mapping_epoch":34,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709878,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12967,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12742","reported_seq":13295,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:14.789133+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:14.789133+0000","last_peered":"2024-09-18T11:56:14.789133+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:14.789133+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539283,"num_objects":67,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12741,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12864","reported_seq":12959,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.870074+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:15.870074+0000","last_peered":"2024-09-18T11:56:15.870074+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:15.870074+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505078,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12862,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12833","reported_seq":12956,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.017661+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:15.017661+0000","last_peered":"2024-09-18T11:56:15.017661+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:15.017661+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505078,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12831,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1990}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12734","reported_seq":12855,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.017120+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:15.017120+0000","last_peered":"2024-09-18T11:56:15.017120+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:15.017120+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425984,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12965","reported_seq":13074,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.805518+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:15.805518+0000","last_peered":"2024-09-18T11:56:15.805518+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:15.805518+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414966,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12963,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2065}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12788","reported_seq":16942,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:13.061101+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:13.061101+0000","last_peered":"2024-09-18T11:56:13.061101+0000","last_clean":"2024-09-18T11:56:13.061101+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:13.061101+0000","last_undegraded":"2024-09-18T11:56:13.061101+0000","last_fullsized":"2024-09-18T11:56:13.061101+0000","mapping_epoch":38,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709878,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13098","reported_seq":17453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:15.134319+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:15.134319+0000","last_peered":"2024-09-18T11:56:15.134319+0000","last_clean":"2024-09-18T11:56:15.134319+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:15.134319+0000","last_undegraded":"2024-09-18T11:56:15.134319+0000","last_fullsized":"2024-09-18T11:56:15.134319+0000","mapping_epoch":36,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":633324,"num_objects":78,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13095,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12540","reported_seq":12656,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:14.701124+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:14.701124+0000","last_peered":"2024-09-18T11:56:14.701124+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:14.701124+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439542,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12538,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1909}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12975","reported_seq":13098,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.050304+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:15.050304+0000","last_peered":"2024-09-18T11:56:15.050304+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:15.050304+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470897,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12972,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12973","reported_seq":13068,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.133166+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:15.133166+0000","last_peered":"2024-09-18T11:56:15.133166+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:15.133166+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473723,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12972,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2026}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12901","reported_seq":13001,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:14.182176+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:14.182176+0000","last_peered":"2024-09-18T11:56:14.182176+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:14.182176+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539259,"num_objects":66,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12900,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1980}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13015","reported_seq":13142,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.122595+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:15.122595+0000","last_peered":"2024-09-18T11:56:15.122595+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:15.122595+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611574,"num_objects":75,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13013,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2048}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12810","reported_seq":17101,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:12.743824+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:12.743824+0000","last_peered":"2024-09-18T11:56:12.743824+0000","last_clean":"2024-09-18T11:56:12.743824+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:12.743824+0000","last_undegraded":"2024-09-18T11:56:12.743824+0000","last_fullsized":"2024-09-18T11:56:12.743824+0000","mapping_epoch":32,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":759030,"num_objects":93,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12808,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12657","reported_seq":12749,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.615031+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:15.615031+0000","last_peered":"2024-09-18T11:56:15.615031+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:15.615031+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406774,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":8627596,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206028,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8872,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17858560,"data_stored":17858560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923804,"kb_used_data":4436,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93448036,"statfs":{"total":96636764160,"available":95690788864,"internally_reserved":0,"allocated":4542464,"data_stored":4350214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,5,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":9.1129999999999995},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":9.0719999999999992}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":9.3170000000000002},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":9.0310000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936336,"kb_used_data":5000,"kb_used_omap":1,"kb_used_meta":931326,"kb_avail":93435504,"statfs":{"total":96636764160,"available":95677956096,"internally_reserved":0,"allocated":5120000,"data_stored":4927750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":953678168},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.46899999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":19.864999999999998},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.41799999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.71799999999999997},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.55700000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932300,"kb_used_data":6788,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93439540,"statfs":{"total":96636764160,"available":95682088960,"internally_reserved":0,"allocated":6950912,"data_stored":6758662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":6.1150000000000002},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":9.2279999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":15.996},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":1.4019999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":1.2589999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":3.1309999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931672,"kb_used_data":4048,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93440168,"statfs":{"total":96636764160,"available":95682732032,"internally_reserved":0,"allocated":4145152,"data_stored":3952902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":12.958},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.404999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":3.9580000000000002},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":10.433999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":12.999000000000001},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":13.250999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3420160,"data_stored":3420160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6135808,"data_stored":6135808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4337664,"data_stored":4337664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3964928,"data_stored":3964928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:17.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:18.205 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:18.206 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:18.231 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-18T11:56:18.106801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8627596,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206028,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8872,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3724112,"kb_used_data":20272,"kb_used_omap":14,"kb_used_meta":3703793,"kb_avail":373763248,"statfs":{"total":386547056640,"available":382733565952,"internally_reserved":0,"allocated":20758528,"data_stored":19989528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792684059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,8,18,11,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16752640,"num_objects":-2045,"num_object_clones":0,"num_object_copies":-8180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002805"},"pg_stats":[{"pgid":"1.f","version":"39'13191","reported_seq":13293,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.090218+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:15.090218+0000","last_peered":"2024-09-18T11:56:15.090218+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:15.090218+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483328,"num_objects":59,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13191,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12969","reported_seq":17263,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:13.079360+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:13.079360+0000","last_peered":"2024-09-18T11:56:13.079360+0000","last_clean":"2024-09-18T11:56:13.079360+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:13.079360+0000","last_undegraded":"2024-09-18T11:56:13.079360+0000","last_fullsized":"2024-09-18T11:56:13.079360+0000","mapping_epoch":34,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709878,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12967,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12742","reported_seq":13295,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:14.789133+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:14.789133+0000","last_peered":"2024-09-18T11:56:14.789133+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:14.789133+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539283,"num_objects":67,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12741,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12864","reported_seq":12959,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.870074+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:15.870074+0000","last_peered":"2024-09-18T11:56:15.870074+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:15.870074+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505078,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12862,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12833","reported_seq":12956,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.017661+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:15.017661+0000","last_peered":"2024-09-18T11:56:15.017661+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:15.017661+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505078,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12831,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1990}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12734","reported_seq":12855,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.017120+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:15.017120+0000","last_peered":"2024-09-18T11:56:15.017120+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:15.017120+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425984,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12965","reported_seq":13074,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.805518+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:15.805518+0000","last_peered":"2024-09-18T11:56:15.805518+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:15.805518+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414966,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12963,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2065}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12788","reported_seq":16942,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:13.061101+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:13.061101+0000","last_peered":"2024-09-18T11:56:13.061101+0000","last_clean":"2024-09-18T11:56:13.061101+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:13.061101+0000","last_undegraded":"2024-09-18T11:56:13.061101+0000","last_fullsized":"2024-09-18T11:56:13.061101+0000","mapping_epoch":38,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709878,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13098","reported_seq":17453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:15.134319+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:15.134319+0000","last_peered":"2024-09-18T11:56:15.134319+0000","last_clean":"2024-09-18T11:56:15.134319+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:15.134319+0000","last_undegraded":"2024-09-18T11:56:15.134319+0000","last_fullsized":"2024-09-18T11:56:15.134319+0000","mapping_epoch":36,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":633324,"num_objects":78,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13095,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12540","reported_seq":12656,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:14.701124+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:14.701124+0000","last_peered":"2024-09-18T11:56:14.701124+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:14.701124+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439542,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12538,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1909}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12975","reported_seq":13098,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.050304+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:15.050304+0000","last_peered":"2024-09-18T11:56:15.050304+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:15.050304+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470897,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12972,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12973","reported_seq":13068,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.133166+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:15.133166+0000","last_peered":"2024-09-18T11:56:15.133166+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:15.133166+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473723,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12972,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2026}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12901","reported_seq":13001,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:14.182176+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:14.182176+0000","last_peered":"2024-09-18T11:56:14.182176+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:14.182176+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539259,"num_objects":66,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12900,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1980}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13015","reported_seq":13142,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.122595+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:15.122595+0000","last_peered":"2024-09-18T11:56:15.122595+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:15.122595+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611574,"num_objects":75,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13013,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2048}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12810","reported_seq":17101,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:12.743824+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:12.743824+0000","last_peered":"2024-09-18T11:56:12.743824+0000","last_clean":"2024-09-18T11:56:12.743824+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:12.743824+0000","last_undegraded":"2024-09-18T11:56:12.743824+0000","last_fullsized":"2024-09-18T11:56:12.743824+0000","mapping_epoch":32,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":759030,"num_objects":93,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12808,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12657","reported_seq":12749,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.615031+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:15.615031+0000","last_peered":"2024-09-18T11:56:15.615031+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:15.615031+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406774,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":8627596,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206028,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8872,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17858560,"data_stored":17858560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923804,"kb_used_data":4436,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93448036,"statfs":{"total":96636764160,"available":95690788864,"internally_reserved":0,"allocated":4542464,"data_stored":4350214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,5,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":9.1129999999999995},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":9.0719999999999992}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":9.3170000000000002},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":9.0310000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936336,"kb_used_data":5000,"kb_used_omap":1,"kb_used_meta":931326,"kb_avail":93435504,"statfs":{"total":96636764160,"available":95677956096,"internally_reserved":0,"allocated":5120000,"data_stored":4927750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":953678168},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.46899999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":19.864999999999998},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.41799999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.71799999999999997},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.55700000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932300,"kb_used_data":6788,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93439540,"statfs":{"total":96636764160,"available":95682088960,"internally_reserved":0,"allocated":6950912,"data_stored":6758662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":6.1150000000000002},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":9.2279999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":15.996},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":1.4019999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":1.2589999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":3.1309999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931672,"kb_used_data":4048,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93440168,"statfs":{"total":96636764160,"available":95682732032,"internally_reserved":0,"allocated":4145152,"data_stored":3952902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":12.958},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.404999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":3.9580000000000002},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":10.433999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":12.999000000000001},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":13.250999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3420160,"data_stored":3420160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6135808,"data_stored":6135808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4337664,"data_stored":4337664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3964928,"data_stored":3964928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:18.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:18.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13191", 'reported_seq': 13293, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:15.090218+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:15.090218+0000', 'last_peered': '2024-09-18T11:56:15.090218+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:15.090218+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3100", 'ondisk_log_start': "31'3100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483328, 'num_objects': 59, 'num_object_clones': 0, 'num_object_copies': 236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 59, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13191, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:18.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:56:18.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12742", 'reported_seq': 13295, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:14.789133+0000', 'last_change': '2024-09-18T11:55:51.694654+0000', 'last_active': '2024-09-18T11:56:14.789133+0000', 'last_peered': '2024-09-18T11:56:14.789133+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:56:14.789133+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 539283, 'num_objects': 67, 'num_object_clones': 0, 'num_object_copies': 268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1824, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 67, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12741, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 232, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1757}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:18.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:18.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12864", 'reported_seq': 12959, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:15.870074+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:15.870074+0000', 'last_peered': '2024-09-18T11:56:15.870074+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:15.870074+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 505078, 'num_objects': 62, 'num_object_clones': 0, 'num_object_copies': 248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 62, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12862, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2039}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12833", 'reported_seq': 12956, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:15.017661+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:15.017661+0000', 'last_peered': '2024-09-18T11:56:15.017661+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:15.017661+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 505078, 'num_objects': 62, 'num_object_clones': 0, 'num_object_copies': 248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 62, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12831, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1990}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12734", 'reported_seq': 12855, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:15.017120+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:15.017120+0000', 'last_peered': '2024-09-18T11:56:15.017120+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:15.017120+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 425984, 'num_objects': 52, 'num_object_clones': 0, 'num_object_copies': 208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 52, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12734, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2067}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12965", 'reported_seq': 13074, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:15.805518+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:56:15.805518+0000', 'last_peered': '2024-09-18T11:56:15.805518+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:15.805518+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 414966, 'num_objects': 51, 'num_object_clones': 0, 'num_object_copies': 204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 51, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12963, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2065}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12540", 'reported_seq': 12656, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:14.701124+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:14.701124+0000', 'last_peered': '2024-09-18T11:56:14.701124+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:14.701124+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 439542, 'num_objects': 54, 'num_object_clones': 0, 'num_object_copies': 216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 54, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12538, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1909}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12975", 'reported_seq': 13098, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:15.050304+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:56:15.050304+0000', 'last_peered': '2024-09-18T11:56:15.050304+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:56:15.050304+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 470897, 'num_objects': 58, 'num_object_clones': 0, 'num_object_copies': 232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 58, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12972, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1976}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:18.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:18.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12973", 'reported_seq': 13068, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:15.133166+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:15.133166+0000', 'last_peered': '2024-09-18T11:56:15.133166+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:15.133166+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 473723, 'num_objects': 58, 'num_object_clones': 0, 'num_object_copies': 232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 58, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12972, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2026}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:18.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:18.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12901", 'reported_seq': 13001, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:14.182176+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:14.182176+0000', 'last_peered': '2024-09-18T11:56:14.182176+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:14.182176+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 539259, 'num_objects': 66, 'num_object_clones': 0, 'num_object_copies': 264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 66, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12900, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:18.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:18.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13015", 'reported_seq': 13142, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:15.122595+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:15.122595+0000', 'last_peered': '2024-09-18T11:56:15.122595+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:15.122595+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 611574, 'num_objects': 75, 'num_object_clones': 0, 'num_object_copies': 300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 75, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13013, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2048}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:18.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:56:18.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12657", 'reported_seq': 12749, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:15.615031+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:56:15.615031+0000', 'last_peered': '2024-09-18T11:56:15.615031+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:56:15.615031+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 406774, 'num_objects': 50, 'num_object_clones': 0, 'num_object_copies': 200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 50, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:18.941 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:18.941 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24584/4236 objects degraded (580.359%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":777,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":1059,"data_bytes":8627596,"bytes_used":3813490688,"bytes_avail":382733565952,"bytes_total":386547056640,"degraded_objects":24584,"degraded_total":4236,"degraded_ratio":5.8035882908404153,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":170},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:18.980 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:18.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:19.538 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:19.538 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:19.569 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-18T11:56:18.106801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8627596,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206028,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8872,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3724112,"kb_used_data":20272,"kb_used_omap":14,"kb_used_meta":3703793,"kb_avail":373763248,"statfs":{"total":386547056640,"available":382733565952,"internally_reserved":0,"allocated":20758528,"data_stored":19989528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792684059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,8,18,11,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16752640,"num_objects":-2045,"num_object_clones":0,"num_object_copies":-8180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002805"},"pg_stats":[{"pgid":"1.f","version":"39'13191","reported_seq":13293,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.090218+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:15.090218+0000","last_peered":"2024-09-18T11:56:15.090218+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:15.090218+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483328,"num_objects":59,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13191,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12969","reported_seq":17263,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:13.079360+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:13.079360+0000","last_peered":"2024-09-18T11:56:13.079360+0000","last_clean":"2024-09-18T11:56:13.079360+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:13.079360+0000","last_undegraded":"2024-09-18T11:56:13.079360+0000","last_fullsized":"2024-09-18T11:56:13.079360+0000","mapping_epoch":34,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709878,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12967,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12742","reported_seq":13295,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:14.789133+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:14.789133+0000","last_peered":"2024-09-18T11:56:14.789133+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:14.789133+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539283,"num_objects":67,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12741,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12864","reported_seq":12959,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.870074+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:15.870074+0000","last_peered":"2024-09-18T11:56:15.870074+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:15.870074+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505078,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12862,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12833","reported_seq":12956,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.017661+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:15.017661+0000","last_peered":"2024-09-18T11:56:15.017661+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:15.017661+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505078,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12831,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1990}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12734","reported_seq":12855,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.017120+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:15.017120+0000","last_peered":"2024-09-18T11:56:15.017120+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:15.017120+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425984,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12965","reported_seq":13074,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.805518+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:15.805518+0000","last_peered":"2024-09-18T11:56:15.805518+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:15.805518+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414966,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12963,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2065}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12788","reported_seq":16942,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:13.061101+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:13.061101+0000","last_peered":"2024-09-18T11:56:13.061101+0000","last_clean":"2024-09-18T11:56:13.061101+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:13.061101+0000","last_undegraded":"2024-09-18T11:56:13.061101+0000","last_fullsized":"2024-09-18T11:56:13.061101+0000","mapping_epoch":38,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709878,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13098","reported_seq":17453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:15.134319+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:15.134319+0000","last_peered":"2024-09-18T11:56:15.134319+0000","last_clean":"2024-09-18T11:56:15.134319+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:15.134319+0000","last_undegraded":"2024-09-18T11:56:15.134319+0000","last_fullsized":"2024-09-18T11:56:15.134319+0000","mapping_epoch":36,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":633324,"num_objects":78,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13095,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12540","reported_seq":12656,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:14.701124+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:14.701124+0000","last_peered":"2024-09-18T11:56:14.701124+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:14.701124+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439542,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12538,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1909}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12975","reported_seq":13098,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.050304+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:15.050304+0000","last_peered":"2024-09-18T11:56:15.050304+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:15.050304+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470897,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12972,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12973","reported_seq":13068,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.133166+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:15.133166+0000","last_peered":"2024-09-18T11:56:15.133166+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:15.133166+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473723,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12972,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2026}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12901","reported_seq":13001,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:14.182176+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:14.182176+0000","last_peered":"2024-09-18T11:56:14.182176+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:14.182176+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539259,"num_objects":66,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12900,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1980}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13015","reported_seq":13142,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.122595+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:15.122595+0000","last_peered":"2024-09-18T11:56:15.122595+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:15.122595+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611574,"num_objects":75,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13013,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2048}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12810","reported_seq":17101,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:12.743824+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:12.743824+0000","last_peered":"2024-09-18T11:56:12.743824+0000","last_clean":"2024-09-18T11:56:12.743824+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:12.743824+0000","last_undegraded":"2024-09-18T11:56:12.743824+0000","last_fullsized":"2024-09-18T11:56:12.743824+0000","mapping_epoch":32,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":759030,"num_objects":93,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12808,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12657","reported_seq":12749,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:15.615031+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:15.615031+0000","last_peered":"2024-09-18T11:56:15.615031+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:15.615031+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406774,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":8627596,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206028,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8872,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17858560,"data_stored":17858560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923804,"kb_used_data":4436,"kb_used_omap":4,"kb_used_meta":919355,"kb_avail":93448036,"statfs":{"total":96636764160,"available":95690788864,"internally_reserved":0,"allocated":4542464,"data_stored":4350214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941420097},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,5,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":9.1129999999999995},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":9.0719999999999992}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":9.3170000000000002},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":9.0310000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936336,"kb_used_data":5000,"kb_used_omap":1,"kb_used_meta":931326,"kb_avail":93435504,"statfs":{"total":96636764160,"available":95677956096,"internally_reserved":0,"allocated":5120000,"data_stored":4927750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":953678168},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:16 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":5.5119999999999996,"15min":4.0780000000000003},"min":{"1min":0.40400000000000003,"5min":0.153,"15min":0.153},"max":{"1min":30.859000000000002,"5min":30.859000000000002,"15min":30.859000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":7.0590000000000002,"5min":5.6740000000000004,"15min":3.7229999999999999},"min":{"1min":0.435,"5min":0.20300000000000001,"15min":0.17699999999999999},"max":{"1min":30.893999999999998,"5min":30.893999999999998,"15min":30.893999999999998},"last":0.46899999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":19.864999999999998},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.41799999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.71799999999999997},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.55700000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932300,"kb_used_data":6788,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93439540,"statfs":{"total":96636764160,"available":95682088960,"internally_reserved":0,"allocated":6950912,"data_stored":6758662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":6.1150000000000002},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":9.2279999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":15.996},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":1.4019999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":1.2589999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":3.1309999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931672,"kb_used_data":4048,"kb_used_omap":4,"kb_used_meta":927611,"kb_avail":93440168,"statfs":{"total":96636764160,"available":95682732032,"internally_reserved":0,"allocated":4145152,"data_stored":3952902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949874241},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":12.958},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":10.404999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":3.9580000000000002},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":10.433999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":12.999000000000001},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":13.250999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3420160,"data_stored":3420160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6135808,"data_stored":6135808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4337664,"data_stored":4337664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3964928,"data_stored":3964928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:20.034 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Removed 103501 objects 2024-09-18T11:56:20.034 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi071.stdout:Clean up completed and total clean up time :482.754 2024-09-18T11:56:22.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:22.911 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:22.911 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:22.923 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-18T11:56:22.107561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":738700,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206991,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3691528,"kb_used_data":4328,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373795832,"statfs":{"total":386547056640,"available":382766931968,"internally_reserved":0,"allocated":4431872,"data_stored":3662872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,5,6,9,11,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21200896,"num_objects":-2588,"num_object_clones":0,"num_object_copies":-10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002604"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13353,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.018250+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:20.018250+0000","last_peered":"2024-09-18T11:56:20.018250+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:20.018250+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13033","reported_seq":17326,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:18.159799+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:18.159799+0000","last_peered":"2024-09-18T11:56:18.159799+0000","last_clean":"2024-09-18T11:56:18.159799+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:18.159799+0000","last_undegraded":"2024-09-18T11:56:18.159799+0000","last_fullsized":"2024-09-18T11:56:18.159799+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185590,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13031,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12806","reported_seq":13462,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.766422+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:19.766422+0000","last_peered":"2024-09-18T11:56:19.766422+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:19.766422+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14995,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12805,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.980113+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:19.980113+0000","last_peered":"2024-09-18T11:56:19.980113+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:19.980113+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.004935+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:20.004935+0000","last_peered":"2024-09-18T11:56:20.004935+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:20.004935+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12907,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.721283+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:19.721283+0000","last_peered":"2024-09-18T11:56:19.721283+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:19.721283+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13123,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.942305+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:19.942305+0000","last_peered":"2024-09-18T11:56:19.942305+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:19.942305+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12856","reported_seq":17012,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:18.066164+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:18.066164+0000","last_peered":"2024-09-18T11:56:18.066164+0000","last_clean":"2024-09-18T11:56:18.066164+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:18.066164+0000","last_undegraded":"2024-09-18T11:56:18.066164+0000","last_fullsized":"2024-09-18T11:56:18.066164+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152822,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12854,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17527,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.936443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:19.936443+0000","last_peered":"2024-09-18T11:56:19.936443+0000","last_clean":"2024-09-18T11:56:19.936443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:19.936443+0000","last_undegraded":"2024-09-18T11:56:19.936443+0000","last_fullsized":"2024-09-18T11:56:19.936443+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12590","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.692053+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:19.692053+0000","last_peered":"2024-09-18T11:56:19.692053+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:19.692053+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29942,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1959}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.743801+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:19.743801+0000","last_peered":"2024-09-18T11:56:19.743801+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:19.743801+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13125,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.946992+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:19.946992+0000","last_peered":"2024-09-18T11:56:19.946992+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:19.946992+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12963","reported_seq":13063,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.501428+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:19.501428+0000","last_peered":"2024-09-18T11:56:19.501428+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:19.501428+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31355,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12962,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.886947+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:19.886947+0000","last_peered":"2024-09-18T11:56:19.886947+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:19.886947+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12878","reported_seq":17169,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:18.000852+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:18.000852+0000","last_peered":"2024-09-18T11:56:18.000852+0000","last_clean":"2024-09-18T11:56:18.000852+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:18.000852+0000","last_undegraded":"2024-09-18T11:56:18.000852+0000","last_fullsized":"2024-09-18T11:56:18.000852+0000","mapping_epoch":32,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201974,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12876,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.019447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:20.019447+0000","last_peered":"2024-09-18T11:56:20.019447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:20.019447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":738700,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206991,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2076672,"data_stored":2076672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923984,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447856,"statfs":{"total":96636764160,"available":95690604544,"internally_reserved":0,"allocated":466944,"data_stored":274694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,2,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":2.081},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":9.7400000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":9.6969999999999992},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":9.6660000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":28.202999999999999},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":11.784000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919756,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452084,"statfs":{"total":96636764160,"available":95694934016,"internally_reserved":0,"allocated":921600,"data_stored":729350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":5.734},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":5.7560000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":6.4210000000000003},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":10.99}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":5.6980000000000004},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":4.3369999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932320,"kb_used_data":2648,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93439520,"statfs":{"total":96636764160,"available":95682068480,"internally_reserved":0,"allocated":2711552,"data_stored":2519302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":5.5359999999999996},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":18.920000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":6.9889999999999999},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.5700000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":1.4419999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":6.952}]}]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915468,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456372,"statfs":{"total":96636764160,"available":95699324928,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":13.087999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":3.1000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.0429999999999999},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.95799999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":1.014},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.98599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1847296,"data_stored":1847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:22.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:23.226 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:56:23.225+0000 7fa593fff640 0 --2- 172.21.15.71:0/961264967 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa594068270 0x7fa594134a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T11:56:23.402 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:23.402 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:23.415 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-18T11:56:22.107561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":738700,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206991,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3691528,"kb_used_data":4328,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373795832,"statfs":{"total":386547056640,"available":382766931968,"internally_reserved":0,"allocated":4431872,"data_stored":3662872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,5,6,9,11,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21200896,"num_objects":-2588,"num_object_clones":0,"num_object_copies":-10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002604"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13353,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.018250+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:20.018250+0000","last_peered":"2024-09-18T11:56:20.018250+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:20.018250+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13033","reported_seq":17326,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:18.159799+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:18.159799+0000","last_peered":"2024-09-18T11:56:18.159799+0000","last_clean":"2024-09-18T11:56:18.159799+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:18.159799+0000","last_undegraded":"2024-09-18T11:56:18.159799+0000","last_fullsized":"2024-09-18T11:56:18.159799+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185590,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13031,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12806","reported_seq":13462,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.766422+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:19.766422+0000","last_peered":"2024-09-18T11:56:19.766422+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:19.766422+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14995,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12805,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.980113+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:19.980113+0000","last_peered":"2024-09-18T11:56:19.980113+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:19.980113+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.004935+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:20.004935+0000","last_peered":"2024-09-18T11:56:20.004935+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:20.004935+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12907,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.721283+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:19.721283+0000","last_peered":"2024-09-18T11:56:19.721283+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:19.721283+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13123,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.942305+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:19.942305+0000","last_peered":"2024-09-18T11:56:19.942305+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:19.942305+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12856","reported_seq":17012,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:18.066164+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:18.066164+0000","last_peered":"2024-09-18T11:56:18.066164+0000","last_clean":"2024-09-18T11:56:18.066164+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:18.066164+0000","last_undegraded":"2024-09-18T11:56:18.066164+0000","last_fullsized":"2024-09-18T11:56:18.066164+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152822,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12854,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17527,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.936443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:19.936443+0000","last_peered":"2024-09-18T11:56:19.936443+0000","last_clean":"2024-09-18T11:56:19.936443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:19.936443+0000","last_undegraded":"2024-09-18T11:56:19.936443+0000","last_fullsized":"2024-09-18T11:56:19.936443+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12590","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.692053+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:19.692053+0000","last_peered":"2024-09-18T11:56:19.692053+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:19.692053+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29942,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1959}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.743801+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:19.743801+0000","last_peered":"2024-09-18T11:56:19.743801+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:19.743801+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13125,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.946992+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:19.946992+0000","last_peered":"2024-09-18T11:56:19.946992+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:19.946992+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12963","reported_seq":13063,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.501428+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:19.501428+0000","last_peered":"2024-09-18T11:56:19.501428+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:19.501428+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31355,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12962,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.886947+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:19.886947+0000","last_peered":"2024-09-18T11:56:19.886947+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:19.886947+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12878","reported_seq":17169,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:18.000852+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:18.000852+0000","last_peered":"2024-09-18T11:56:18.000852+0000","last_clean":"2024-09-18T11:56:18.000852+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:18.000852+0000","last_undegraded":"2024-09-18T11:56:18.000852+0000","last_fullsized":"2024-09-18T11:56:18.000852+0000","mapping_epoch":32,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201974,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12876,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.019447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:20.019447+0000","last_peered":"2024-09-18T11:56:20.019447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:20.019447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":738700,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206991,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2076672,"data_stored":2076672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923984,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447856,"statfs":{"total":96636764160,"available":95690604544,"internally_reserved":0,"allocated":466944,"data_stored":274694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,2,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":2.081},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":9.7400000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":9.6969999999999992},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":9.6660000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":28.202999999999999},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":11.784000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919756,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452084,"statfs":{"total":96636764160,"available":95694934016,"internally_reserved":0,"allocated":921600,"data_stored":729350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":5.734},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":5.7560000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":6.4210000000000003},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":10.99}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":5.6980000000000004},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":4.3369999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932320,"kb_used_data":2648,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93439520,"statfs":{"total":96636764160,"available":95682068480,"internally_reserved":0,"allocated":2711552,"data_stored":2519302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":5.5359999999999996},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":18.920000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":6.9889999999999999},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.5700000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":1.4419999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":6.952}]}]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915468,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456372,"statfs":{"total":96636764160,"available":95699324928,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":13.087999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":3.1000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.0429999999999999},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.95799999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":1.014},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.98599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1847296,"data_stored":1847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:23.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:23.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13353, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:20.018250+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:20.018250+0000', 'last_peered': '2024-09-18T11:56:20.018250+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:20.018250+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:23.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:56:23.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12806", 'reported_seq': 13462, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.766422+0000', 'last_change': '2024-09-18T11:55:51.694654+0000', 'last_active': '2024-09-18T11:56:19.766422+0000', 'last_peered': '2024-09-18T11:56:19.766422+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:56:19.766422+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14995, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12805, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 283, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1770}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:23.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:23.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13020, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.980113+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:19.980113+0000', 'last_peered': '2024-09-18T11:56:19.980113+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:19.980113+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:23.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:23.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13016, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:20.004935+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:20.004935+0000', 'last_peered': '2024-09-18T11:56:20.004935+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:20.004935+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:23.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:23.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12907, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.721283+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:19.721283+0000', 'last_peered': '2024-09-18T11:56:19.721283+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:19.721283+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:23.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:23.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'13014", 'reported_seq': 13123, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.942305+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:56:19.942305+0000', 'last_peered': '2024-09-18T11:56:19.942305+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:19.942305+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13012, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2114}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:23.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12590", 'reported_seq': 12706, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.692053+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:19.692053+0000', 'last_peered': '2024-09-18T11:56:19.692053+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:19.692053+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29942, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1959}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13030", 'reported_seq': 13153, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.743801+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:56:19.743801+0000', 'last_peered': '2024-09-18T11:56:19.743801+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:56:19.743801+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20337, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13027, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13125, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.946992+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:19.946992+0000', 'last_peered': '2024-09-18T11:56:19.946992+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:19.946992+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12963", 'reported_seq': 13063, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.501428+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:19.501428+0000', 'last_peered': '2024-09-18T11:56:19.501428+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:19.501428+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31355, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12962, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13215, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.886947+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:19.886947+0000', 'last_peered': '2024-09-18T11:56:19.886947+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:19.886947+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:56:23.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12705", 'reported_seq': 12797, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:20.019447+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:56:20.019447+0000', 'last_peered': '2024-09-18T11:56:20.019447+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:56:20.019447+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12703, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2000}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:23.419 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24533/384 objects degraded (6388.802%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":781,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":96,"data_bytes":738700,"bytes_used":3780124672,"bytes_avail":382766931968,"bytes_total":386547056640,"degraded_objects":24533,"degraded_total":384,"degraded_ratio":63.888020833333343,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":215},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:23.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:24.086 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:24.087 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:24.099 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-18T11:56:22.107561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":738700,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206991,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3691528,"kb_used_data":4328,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373795832,"statfs":{"total":386547056640,"available":382766931968,"internally_reserved":0,"allocated":4431872,"data_stored":3662872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,5,6,9,11,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21200896,"num_objects":-2588,"num_object_clones":0,"num_object_copies":-10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002604"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13353,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.018250+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:20.018250+0000","last_peered":"2024-09-18T11:56:20.018250+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:20.018250+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13033","reported_seq":17326,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:18.159799+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:18.159799+0000","last_peered":"2024-09-18T11:56:18.159799+0000","last_clean":"2024-09-18T11:56:18.159799+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:18.159799+0000","last_undegraded":"2024-09-18T11:56:18.159799+0000","last_fullsized":"2024-09-18T11:56:18.159799+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185590,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13031,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12806","reported_seq":13462,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.766422+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:19.766422+0000","last_peered":"2024-09-18T11:56:19.766422+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:19.766422+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14995,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12805,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.980113+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:19.980113+0000","last_peered":"2024-09-18T11:56:19.980113+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:19.980113+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.004935+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:20.004935+0000","last_peered":"2024-09-18T11:56:20.004935+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:20.004935+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12907,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.721283+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:19.721283+0000","last_peered":"2024-09-18T11:56:19.721283+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:19.721283+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13123,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.942305+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:19.942305+0000","last_peered":"2024-09-18T11:56:19.942305+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:19.942305+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12856","reported_seq":17012,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:18.066164+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:18.066164+0000","last_peered":"2024-09-18T11:56:18.066164+0000","last_clean":"2024-09-18T11:56:18.066164+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:18.066164+0000","last_undegraded":"2024-09-18T11:56:18.066164+0000","last_fullsized":"2024-09-18T11:56:18.066164+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152822,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12854,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17527,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.936443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:19.936443+0000","last_peered":"2024-09-18T11:56:19.936443+0000","last_clean":"2024-09-18T11:56:19.936443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:19.936443+0000","last_undegraded":"2024-09-18T11:56:19.936443+0000","last_fullsized":"2024-09-18T11:56:19.936443+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12590","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.692053+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:19.692053+0000","last_peered":"2024-09-18T11:56:19.692053+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:19.692053+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29942,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1959}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.743801+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:19.743801+0000","last_peered":"2024-09-18T11:56:19.743801+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:19.743801+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13125,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.946992+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:19.946992+0000","last_peered":"2024-09-18T11:56:19.946992+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:19.946992+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12963","reported_seq":13063,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.501428+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:19.501428+0000","last_peered":"2024-09-18T11:56:19.501428+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:19.501428+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31355,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12962,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.886947+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:19.886947+0000","last_peered":"2024-09-18T11:56:19.886947+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:19.886947+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12878","reported_seq":17169,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:18.000852+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:18.000852+0000","last_peered":"2024-09-18T11:56:18.000852+0000","last_clean":"2024-09-18T11:56:18.000852+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:18.000852+0000","last_undegraded":"2024-09-18T11:56:18.000852+0000","last_fullsized":"2024-09-18T11:56:18.000852+0000","mapping_epoch":32,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201974,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12876,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.019447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:20.019447+0000","last_peered":"2024-09-18T11:56:20.019447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:20.019447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":738700,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206991,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8923,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2076672,"data_stored":2076672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923984,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447856,"statfs":{"total":96636764160,"available":95690604544,"internally_reserved":0,"allocated":466944,"data_stored":274694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,2,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":2.081},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":9.7400000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":9.6969999999999992},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":9.6660000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":28.202999999999999},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":11.784000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919756,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452084,"statfs":{"total":96636764160,"available":95694934016,"internally_reserved":0,"allocated":921600,"data_stored":729350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":5.734},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":5.7560000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":6.4210000000000003},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":10.99}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":5.6980000000000004},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":4.3369999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932320,"kb_used_data":2648,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93439520,"statfs":{"total":96636764160,"available":95682068480,"internally_reserved":0,"allocated":2711552,"data_stored":2519302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":5.5359999999999996},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":18.920000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":6.9889999999999999},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":5.5700000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":1.4419999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":6.952}]}]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915468,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456372,"statfs":{"total":96636764160,"available":95699324928,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":13.087999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":3.1000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:20 2024","interfaces":[{"interface":"back","average":{"1min":5.4770000000000003,"5min":6.1909999999999998,"15min":4.327},"min":{"1min":0.34000000000000002,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.487,"5min":28.969000000000001,"15min":32.795000000000002},"last":1.0429999999999999},{"interface":"front","average":{"1min":5.6559999999999997,"5min":5.7869999999999999,"15min":3.9020000000000001},"min":{"1min":0.48299999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.507999999999999,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.95799999999999996}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":1.014},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.98599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1847296,"data_stored":1847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:27.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:27.396 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:27.397 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:27.408 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-18T11:56:26.108249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23694,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9762,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697196,"kb_used_data":1676,"kb_used_omap":14,"kb_used_meta":3695473,"kb_avail":373790164,"statfs":{"total":386547056640,"available":382761127936,"internally_reserved":0,"allocated":1716224,"data_stored":947224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3784164379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15245336,"num_objects":-1862,"num_object_clones":0,"num_object_copies":-7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1862,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002367"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13353,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.018250+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:20.018250+0000","last_peered":"2024-09-18T11:56:20.018250+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:20.018250+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17348,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.932196+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:19.932196+0000","last_peered":"2024-09-18T11:56:19.932196+0000","last_clean":"2024-09-18T11:56:19.932196+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:19.932196+0000","last_undegraded":"2024-09-18T11:56:19.932196+0000","last_fullsized":"2024-09-18T11:56:19.932196+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":15142,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:24.838465+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:24.838465+0000","last_peered":"2024-09-18T11:56:24.838465+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:24.838465+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-933}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.980113+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:19.980113+0000","last_peered":"2024-09-18T11:56:19.980113+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:19.980113+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.004935+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:20.004935+0000","last_peered":"2024-09-18T11:56:20.004935+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:20.004935+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12907,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.721283+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:19.721283+0000","last_peered":"2024-09-18T11:56:19.721283+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:19.721283+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13123,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.942305+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:19.942305+0000","last_peered":"2024-09-18T11:56:19.942305+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:19.942305+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.905625+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:19.905625+0000","last_peered":"2024-09-18T11:56:19.905625+0000","last_clean":"2024-09-18T11:56:19.905625+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:19.905625+0000","last_undegraded":"2024-09-18T11:56:19.905625+0000","last_fullsized":"2024-09-18T11:56:19.905625+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17527,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.936443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:19.936443+0000","last_peered":"2024-09-18T11:56:19.936443+0000","last_clean":"2024-09-18T11:56:19.936443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:19.936443+0000","last_undegraded":"2024-09-18T11:56:19.936443+0000","last_fullsized":"2024-09-18T11:56:19.936443+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12708,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.918134+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:19.918134+0000","last_peered":"2024-09-18T11:56:19.918134+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:19.918134+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.743801+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:19.743801+0000","last_peered":"2024-09-18T11:56:19.743801+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:19.743801+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13125,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.946992+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:19.946992+0000","last_peered":"2024-09-18T11:56:19.946992+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:19.946992+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13066,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.967353+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:19.967353+0000","last_peered":"2024-09-18T11:56:19.967353+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:19.967353+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.886947+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:19.886947+0000","last_peered":"2024-09-18T11:56:19.886947+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:19.886947+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17192,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.974257+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:19.974257+0000","last_peered":"2024-09-18T11:56:19.974257+0000","last_clean":"2024-09-18T11:56:19.974257+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:19.974257+0000","last_undegraded":"2024-09-18T11:56:19.974257+0000","last_fullsized":"2024-09-18T11:56:19.974257+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.019447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:20.019447+0000","last_peered":"2024-09-18T11:56:20.019447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:20.019447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23694,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9762,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":405504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923852,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447988,"statfs":{"total":96636764160,"available":95690739712,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.246},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.25800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919180,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452660,"statfs":{"total":96636764160,"available":95695523840,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.251},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.27400000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.191}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.222},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.25800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":938696,"kb_used_data":704,"kb_used_omap":4,"kb_used_meta":937979,"kb_avail":93433144,"statfs":{"total":96636764160,"available":95675539456,"internally_reserved":0,"allocated":720896,"data_stored":528646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":960491073},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":1.431},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":8.2509999999999994}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":7.1100000000000003},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":7.0529999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":16.347999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":8.4410000000000007}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915468,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456372,"statfs":{"total":96636764160,"available":95699324928,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.17599999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.218}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:27.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:27.688 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:27.689 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:27.701 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-18T11:56:26.108249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23694,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9762,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697196,"kb_used_data":1676,"kb_used_omap":14,"kb_used_meta":3695473,"kb_avail":373790164,"statfs":{"total":386547056640,"available":382761127936,"internally_reserved":0,"allocated":1716224,"data_stored":947224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3784164379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15245336,"num_objects":-1862,"num_object_clones":0,"num_object_copies":-7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1862,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002367"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13353,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.018250+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:20.018250+0000","last_peered":"2024-09-18T11:56:20.018250+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:20.018250+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17348,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.932196+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:19.932196+0000","last_peered":"2024-09-18T11:56:19.932196+0000","last_clean":"2024-09-18T11:56:19.932196+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:19.932196+0000","last_undegraded":"2024-09-18T11:56:19.932196+0000","last_fullsized":"2024-09-18T11:56:19.932196+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":15142,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:24.838465+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:24.838465+0000","last_peered":"2024-09-18T11:56:24.838465+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:24.838465+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-933}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.980113+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:19.980113+0000","last_peered":"2024-09-18T11:56:19.980113+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:19.980113+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.004935+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:20.004935+0000","last_peered":"2024-09-18T11:56:20.004935+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:20.004935+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12907,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.721283+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:19.721283+0000","last_peered":"2024-09-18T11:56:19.721283+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:19.721283+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13123,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.942305+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:19.942305+0000","last_peered":"2024-09-18T11:56:19.942305+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:19.942305+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.905625+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:19.905625+0000","last_peered":"2024-09-18T11:56:19.905625+0000","last_clean":"2024-09-18T11:56:19.905625+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:19.905625+0000","last_undegraded":"2024-09-18T11:56:19.905625+0000","last_fullsized":"2024-09-18T11:56:19.905625+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17527,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.936443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:19.936443+0000","last_peered":"2024-09-18T11:56:19.936443+0000","last_clean":"2024-09-18T11:56:19.936443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:19.936443+0000","last_undegraded":"2024-09-18T11:56:19.936443+0000","last_fullsized":"2024-09-18T11:56:19.936443+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12708,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.918134+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:19.918134+0000","last_peered":"2024-09-18T11:56:19.918134+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:19.918134+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.743801+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:19.743801+0000","last_peered":"2024-09-18T11:56:19.743801+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:19.743801+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13125,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.946992+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:19.946992+0000","last_peered":"2024-09-18T11:56:19.946992+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:19.946992+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13066,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.967353+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:19.967353+0000","last_peered":"2024-09-18T11:56:19.967353+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:19.967353+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.886947+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:19.886947+0000","last_peered":"2024-09-18T11:56:19.886947+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:19.886947+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17192,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.974257+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:19.974257+0000","last_peered":"2024-09-18T11:56:19.974257+0000","last_clean":"2024-09-18T11:56:19.974257+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:19.974257+0000","last_undegraded":"2024-09-18T11:56:19.974257+0000","last_fullsized":"2024-09-18T11:56:19.974257+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.019447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:20.019447+0000","last_peered":"2024-09-18T11:56:20.019447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:20.019447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23694,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9762,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":405504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923852,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447988,"statfs":{"total":96636764160,"available":95690739712,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.246},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.25800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919180,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452660,"statfs":{"total":96636764160,"available":95695523840,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.251},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.27400000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.191}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.222},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.25800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":938696,"kb_used_data":704,"kb_used_omap":4,"kb_used_meta":937979,"kb_avail":93433144,"statfs":{"total":96636764160,"available":95675539456,"internally_reserved":0,"allocated":720896,"data_stored":528646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":960491073},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":1.431},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":8.2509999999999994}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":7.1100000000000003},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":7.0529999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":16.347999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":8.4410000000000007}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915468,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456372,"statfs":{"total":96636764160,"available":95699324928,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.17599999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.218}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:27.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:27.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13353, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:20.018250+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:20.018250+0000', 'last_peered': '2024-09-18T11:56:20.018250+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:20.018250+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:27.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:56:27.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12808", 'reported_seq': 15142, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:24.838465+0000', 'last_change': '2024-09-18T11:55:51.694654+0000', 'last_active': '2024-09-18T11:56:24.838465+0000', 'last_peered': '2024-09-18T11:56:24.838465+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:56:24.838465+0000', 'last_undegraded': '2024-09-18T11:44:47.189859+0000', 'last_fullsized': '2024-09-18T11:44:47.170917+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 934, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12807, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1122, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -933}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13020, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.980113+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:19.980113+0000', 'last_peered': '2024-09-18T11:56:19.980113+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:19.980113+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13016, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:20.004935+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:20.004935+0000', 'last_peered': '2024-09-18T11:56:20.004935+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:20.004935+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12907, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.721283+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:19.721283+0000', 'last_peered': '2024-09-18T11:56:19.721283+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:19.721283+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'13014", 'reported_seq': 13123, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.942305+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:56:19.942305+0000', 'last_peered': '2024-09-18T11:56:19.942305+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:19.942305+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13012, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2114}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12708, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.918134+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:19.918134+0000', 'last_peered': '2024-09-18T11:56:19.918134+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:19.918134+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:27.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:27.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13030", 'reported_seq': 13153, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.743801+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:56:19.743801+0000', 'last_peered': '2024-09-18T11:56:19.743801+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:56:19.743801+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20337, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13027, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:27.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:27.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13125, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.946992+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:19.946992+0000', 'last_peered': '2024-09-18T11:56:19.946992+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:19.946992+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:27.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:27.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 13066, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.967353+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:19.967353+0000', 'last_peered': '2024-09-18T11:56:19.967353+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:19.967353+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:27.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:27.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13215, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.886947+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:19.886947+0000', 'last_peered': '2024-09-18T11:56:19.886947+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:19.886947+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:27.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:56:27.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12705", 'reported_seq': 12797, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:20.019447+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:56:20.019447+0000', 'last_peered': '2024-09-18T11:56:20.019447+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:56:20.019447+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12703, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2000}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:27.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:28.061 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:28.061 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23694/112 objects degraded (21155.357%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":786,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3785928704,"bytes_avail":382761127936,"bytes_total":386547056640,"degraded_objects":23694,"degraded_total":112,"degraded_ratio":211.55357142857142,"recovering_objects_per_sec":77,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":934,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":155},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:28.077 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:28.078 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:28.366 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:28.367 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:28.378 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-18T11:56:28.108638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23694,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9762,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697196,"kb_used_data":1676,"kb_used_omap":14,"kb_used_meta":3695473,"kb_avail":373790164,"statfs":{"total":386547056640,"available":382761127936,"internally_reserved":0,"allocated":1716224,"data_stored":947224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3784164379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8437784,"num_objects":-1031,"num_object_clones":0,"num_object_copies":-4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1031,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":890,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002314"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13353,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.018250+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:20.018250+0000","last_peered":"2024-09-18T11:56:20.018250+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:20.018250+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17348,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.932196+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:19.932196+0000","last_peered":"2024-09-18T11:56:19.932196+0000","last_clean":"2024-09-18T11:56:19.932196+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:19.932196+0000","last_undegraded":"2024-09-18T11:56:19.932196+0000","last_fullsized":"2024-09-18T11:56:19.932196+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":15142,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:24.838465+0000","last_change":"2024-09-18T11:55:51.694654+0000","last_active":"2024-09-18T11:56:24.838465+0000","last_peered":"2024-09-18T11:56:24.838465+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:24.838465+0000","last_undegraded":"2024-09-18T11:44:47.189859+0000","last_fullsized":"2024-09-18T11:44:47.170917+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-933}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.980113+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:19.980113+0000","last_peered":"2024-09-18T11:56:19.980113+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:19.980113+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.004935+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:20.004935+0000","last_peered":"2024-09-18T11:56:20.004935+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:20.004935+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12907,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.721283+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:19.721283+0000","last_peered":"2024-09-18T11:56:19.721283+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:19.721283+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13123,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.942305+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:19.942305+0000","last_peered":"2024-09-18T11:56:19.942305+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:19.942305+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.905625+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:19.905625+0000","last_peered":"2024-09-18T11:56:19.905625+0000","last_clean":"2024-09-18T11:56:19.905625+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:19.905625+0000","last_undegraded":"2024-09-18T11:56:19.905625+0000","last_fullsized":"2024-09-18T11:56:19.905625+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17527,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.936443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:19.936443+0000","last_peered":"2024-09-18T11:56:19.936443+0000","last_clean":"2024-09-18T11:56:19.936443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:19.936443+0000","last_undegraded":"2024-09-18T11:56:19.936443+0000","last_fullsized":"2024-09-18T11:56:19.936443+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12708,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.918134+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:19.918134+0000","last_peered":"2024-09-18T11:56:19.918134+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:19.918134+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.743801+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:19.743801+0000","last_peered":"2024-09-18T11:56:19.743801+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:19.743801+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13125,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.946992+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:19.946992+0000","last_peered":"2024-09-18T11:56:19.946992+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:19.946992+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13066,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.967353+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:19.967353+0000","last_peered":"2024-09-18T11:56:19.967353+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:19.967353+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.886947+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:19.886947+0000","last_peered":"2024-09-18T11:56:19.886947+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:19.886947+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17192,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.974257+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:19.974257+0000","last_peered":"2024-09-18T11:56:19.974257+0000","last_clean":"2024-09-18T11:56:19.974257+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:19.974257+0000","last_undegraded":"2024-09-18T11:56:19.974257+0000","last_fullsized":"2024-09-18T11:56:19.974257+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.019447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:20.019447+0000","last_peered":"2024-09-18T11:56:20.019447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:20.019447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23694,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9762,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":405504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923852,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447988,"statfs":{"total":96636764160,"available":95690739712,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.246},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.25800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919180,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452660,"statfs":{"total":96636764160,"available":95695523840,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.251},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.27400000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.191}]},{"osd":3,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":6.2000000000000002,"15min":4.2409999999999997},"min":{"1min":0.32400000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":22.047000000000001,"5min":22.73,"15min":22.73},"last":0.222},{"interface":"front","average":{"1min":6.0540000000000003,"5min":6.1189999999999998,"15min":3.6789999999999998},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":24.135000000000002,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.25800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":938696,"kb_used_data":704,"kb_used_omap":4,"kb_used_meta":937979,"kb_avail":93433144,"statfs":{"total":96636764160,"available":95675539456,"internally_reserved":0,"allocated":720896,"data_stored":528646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":960491073},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":1.431},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":8.2509999999999994}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":7.1100000000000003},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":7.0529999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":16.347999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":8.4410000000000007}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915468,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456372,"statfs":{"total":96636764160,"available":95699324928,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.17599999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.218}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:31.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:31.715 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:31.716 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:31.727 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-09-18T11:56:30.183189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10696,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688888,"kb_used_data":1684,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373798472,"statfs":{"total":386547056640,"available":382769635328,"internally_reserved":0,"allocated":1724416,"data_stored":951175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-548888,"num_objects":-68,"num_object_clones":0,"num_object_copies":-272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.075627"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13353,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.018250+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:20.018250+0000","last_peered":"2024-09-18T11:56:20.018250+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:20.018250+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17348,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.932196+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:19.932196+0000","last_peered":"2024-09-18T11:56:19.932196+0000","last_clean":"2024-09-18T11:56:19.932196+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:19.932196+0000","last_undegraded":"2024-09-18T11:56:19.932196+0000","last_fullsized":"2024-09-18T11:56:19.932196+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17014,"reported_epoch":40,"state":"peering","last_fresh":"2024-09-18T11:56:29.179792+0000","last_change":"2024-09-18T11:56:29.183009+0000","last_active":"2024-09-18T11:56:29.179753+0000","last_peered":"2024-09-18T11:56:28.892437+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:29.179792+0000","last_undegraded":"2024-09-18T11:56:29.179792+0000","last_fullsized":"2024-09-18T11:56:29.179792+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.980113+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:19.980113+0000","last_peered":"2024-09-18T11:56:19.980113+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:19.980113+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.004935+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:20.004935+0000","last_peered":"2024-09-18T11:56:20.004935+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:20.004935+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12907,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.721283+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:19.721283+0000","last_peered":"2024-09-18T11:56:19.721283+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:19.721283+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13123,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.942305+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:19.942305+0000","last_peered":"2024-09-18T11:56:19.942305+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:19.942305+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.905625+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:19.905625+0000","last_peered":"2024-09-18T11:56:19.905625+0000","last_clean":"2024-09-18T11:56:19.905625+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:19.905625+0000","last_undegraded":"2024-09-18T11:56:19.905625+0000","last_fullsized":"2024-09-18T11:56:19.905625+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17527,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.936443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:19.936443+0000","last_peered":"2024-09-18T11:56:19.936443+0000","last_clean":"2024-09-18T11:56:19.936443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:19.936443+0000","last_undegraded":"2024-09-18T11:56:19.936443+0000","last_fullsized":"2024-09-18T11:56:19.936443+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12709,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.009148+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:20.009148+0000","last_peered":"2024-09-18T11:56:20.009148+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:20.009148+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.743801+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:19.743801+0000","last_peered":"2024-09-18T11:56:19.743801+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:19.743801+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13125,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.946992+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:19.946992+0000","last_peered":"2024-09-18T11:56:19.946992+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:19.946992+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13067,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.983313+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:19.983313+0000","last_peered":"2024-09-18T11:56:19.983313+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:19.983313+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.886947+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:19.886947+0000","last_peered":"2024-09-18T11:56:19.886947+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:19.886947+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17192,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.974257+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:19.974257+0000","last_peered":"2024-09-18T11:56:19.974257+0000","last_clean":"2024-09-18T11:56:19.974257+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:19.974257+0000","last_undegraded":"2024-09-18T11:56:19.974257+0000","last_fullsized":"2024-09-18T11:56:19.974257+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.019447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:20.019447+0000","last_peered":"2024-09-18T11:56:20.019447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:20.019447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10696,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":405504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923852,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447988,"statfs":{"total":96636764160,"available":95690739712,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.246},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.25800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919192,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452648,"statfs":{"total":96636764160,"available":95695511552,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.28899999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.20499999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.249},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.224}]}]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930376,"kb_used_data":704,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441464,"statfs":{"total":96636764160,"available":95684059136,"internally_reserved":0,"allocated":720896,"data_stored":528646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.218},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.26400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23200000000000001},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.245}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.30199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915468,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456372,"statfs":{"total":96636764160,"available":95699324928,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.17599999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.218}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:31.729 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:32.008 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:32.008 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:32.021 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-09-18T11:56:30.183189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10696,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688888,"kb_used_data":1684,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373798472,"statfs":{"total":386547056640,"available":382769635328,"internally_reserved":0,"allocated":1724416,"data_stored":951175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-548888,"num_objects":-68,"num_object_clones":0,"num_object_copies":-272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.075627"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13353,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.018250+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:20.018250+0000","last_peered":"2024-09-18T11:56:20.018250+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:20.018250+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17348,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.932196+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:19.932196+0000","last_peered":"2024-09-18T11:56:19.932196+0000","last_clean":"2024-09-18T11:56:19.932196+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:19.932196+0000","last_undegraded":"2024-09-18T11:56:19.932196+0000","last_fullsized":"2024-09-18T11:56:19.932196+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17014,"reported_epoch":40,"state":"peering","last_fresh":"2024-09-18T11:56:29.179792+0000","last_change":"2024-09-18T11:56:29.183009+0000","last_active":"2024-09-18T11:56:29.179753+0000","last_peered":"2024-09-18T11:56:28.892437+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:29.179792+0000","last_undegraded":"2024-09-18T11:56:29.179792+0000","last_fullsized":"2024-09-18T11:56:29.179792+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.980113+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:19.980113+0000","last_peered":"2024-09-18T11:56:19.980113+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:19.980113+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.004935+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:20.004935+0000","last_peered":"2024-09-18T11:56:20.004935+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:20.004935+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12907,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.721283+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:19.721283+0000","last_peered":"2024-09-18T11:56:19.721283+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:19.721283+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13123,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.942305+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:19.942305+0000","last_peered":"2024-09-18T11:56:19.942305+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:19.942305+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.905625+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:19.905625+0000","last_peered":"2024-09-18T11:56:19.905625+0000","last_clean":"2024-09-18T11:56:19.905625+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:19.905625+0000","last_undegraded":"2024-09-18T11:56:19.905625+0000","last_fullsized":"2024-09-18T11:56:19.905625+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17527,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.936443+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:19.936443+0000","last_peered":"2024-09-18T11:56:19.936443+0000","last_clean":"2024-09-18T11:56:19.936443+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:19.936443+0000","last_undegraded":"2024-09-18T11:56:19.936443+0000","last_fullsized":"2024-09-18T11:56:19.936443+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12709,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.009148+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:20.009148+0000","last_peered":"2024-09-18T11:56:20.009148+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:20.009148+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.743801+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:19.743801+0000","last_peered":"2024-09-18T11:56:19.743801+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:19.743801+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13125,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.946992+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:19.946992+0000","last_peered":"2024-09-18T11:56:19.946992+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:19.946992+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13067,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.983313+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:19.983313+0000","last_peered":"2024-09-18T11:56:19.983313+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:19.983313+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.886947+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:19.886947+0000","last_peered":"2024-09-18T11:56:19.886947+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:19.886947+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17192,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.974257+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:19.974257+0000","last_peered":"2024-09-18T11:56:19.974257+0000","last_clean":"2024-09-18T11:56:19.974257+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:19.974257+0000","last_undegraded":"2024-09-18T11:56:19.974257+0000","last_fullsized":"2024-09-18T11:56:19.974257+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.019447+0000","last_change":"2024-09-18T11:44:47.469518+0000","last_active":"2024-09-18T11:56:20.019447+0000","last_peered":"2024-09-18T11:56:20.019447+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:20.019447+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10696,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":405504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923852,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447988,"statfs":{"total":96636764160,"available":95690739712,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.246},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.25800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919192,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452648,"statfs":{"total":96636764160,"available":95695511552,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.28899999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.20499999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.249},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.224}]}]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930376,"kb_used_data":704,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441464,"statfs":{"total":96636764160,"available":95684059136,"internally_reserved":0,"allocated":720896,"data_stored":528646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.218},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.26400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23200000000000001},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.245}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.30199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915468,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456372,"statfs":{"total":96636764160,"available":95699324928,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.17599999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.218}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:55:24 2024","interfaces":[{"interface":"back","average":{"1min":6.923,"5min":6.7640000000000002,"15min":4.3659999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":24.960999999999999,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":6.6479999999999997,"5min":8.5530000000000008,"15min":5.2869999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":20.881,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:32.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:32.022 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13353, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:20.018250+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:20.018250+0000', 'last_peered': '2024-09-18T11:56:20.018250+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:20.018250+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:32.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-18T11:56:32.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12808", 'reported_seq': 17014, 'reported_epoch': 40, 'state': 'peering', 'last_fresh': '2024-09-18T11:56:29.179792+0000', 'last_change': '2024-09-18T11:56:29.183009+0000', 'last_active': '2024-09-18T11:56:29.179753+0000', 'last_peered': '2024-09-18T11:56:28.892437+0000', 'last_clean': '2024-09-18T11:43:44.593892+0000', 'last_became_active': '2024-09-18T11:44:47.429842+0000', 'last_became_peered': '2024-09-18T11:44:47.429842+0000', 'last_unstale': '2024-09-18T11:56:29.179792+0000', 'last_undegraded': '2024-09-18T11:56:29.179792+0000', 'last_fullsized': '2024-09-18T11:56:29.179792+0000', 'mapping_epoch': 40, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12807, 'num_write_kb': 51223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2056, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 1, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:32.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:32.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13020, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.980113+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:19.980113+0000', 'last_peered': '2024-09-18T11:56:19.980113+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:19.980113+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:32.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:32.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13016, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:20.004935+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:20.004935+0000', 'last_peered': '2024-09-18T11:56:20.004935+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:20.004935+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:32.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:32.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12907, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.721283+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:19.721283+0000', 'last_peered': '2024-09-18T11:56:19.721283+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:19.721283+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:32.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:32.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'13014", 'reported_seq': 13123, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.942305+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:56:19.942305+0000', 'last_peered': '2024-09-18T11:56:19.942305+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:19.942305+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13012, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2114}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:32.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:32.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12709, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:20.009148+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:20.009148+0000', 'last_peered': '2024-09-18T11:56:20.009148+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:20.009148+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:32.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:32.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13030", 'reported_seq': 13153, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.743801+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:56:19.743801+0000', 'last_peered': '2024-09-18T11:56:19.743801+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:56:19.743801+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20337, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13027, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:32.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:32.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13125, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.946992+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:19.946992+0000', 'last_peered': '2024-09-18T11:56:19.946992+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:19.946992+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:32.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:32.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 13067, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.983313+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:19.983313+0000', 'last_peered': '2024-09-18T11:56:19.983313+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:19.983313+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:32.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:32.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13215, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:19.886947+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:19.886947+0000', 'last_peered': '2024-09-18T11:56:19.886947+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:19.886947+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:32.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:56:32.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12705", 'reported_seq': 12797, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:20.019447+0000', 'last_change': '2024-09-18T11:44:47.469518+0000', 'last_active': '2024-09-18T11:56:20.019447+0000', 'last_peered': '2024-09-18T11:56:20.019447+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:56:20.019447+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12703, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2000}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:32.025 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:32.364 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:32.364 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22392/112 objects degraded (19992.857%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":790,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":4},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3777458176,"bytes_avail":382769598464,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":22392,"degraded_total":112,"degraded_ratio":199.92857142857142,"recovering_objects_per_sec":267,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2141,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:32.377 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:32.378 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:32.666 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:32.667 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:32.679 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-18T11:56:32.109312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11064,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688924,"kb_used_data":1708,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373798436,"statfs":{"total":386547056640,"available":382769598464,"internally_reserved":0,"allocated":1748992,"data_stored":963007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-49176,"num_objects":-7,"num_object_clones":0,"num_object_copies":-28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001401"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13355,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:29.175345+0000","last_peered":"2024-09-18T11:56:29.175345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:29.175345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17348,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.932196+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:19.932196+0000","last_peered":"2024-09-18T11:56:19.932196+0000","last_clean":"2024-09-18T11:56:19.932196+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:19.932196+0000","last_undegraded":"2024-09-18T11:56:19.932196+0000","last_fullsized":"2024-09-18T11:56:19.932196+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17014,"reported_epoch":40,"state":"peering","last_fresh":"2024-09-18T11:56:29.179792+0000","last_change":"2024-09-18T11:56:29.183009+0000","last_active":"2024-09-18T11:56:29.179753+0000","last_peered":"2024-09-18T11:56:28.892437+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:29.179792+0000","last_undegraded":"2024-09-18T11:56:29.179792+0000","last_fullsized":"2024-09-18T11:56:29.179792+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.174803+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:29.174803+0000","last_peered":"2024-09-18T11:56:29.174803+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:29.174803+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13018,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176569+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:29.176569+0000","last_peered":"2024-09-18T11:56:29.176569+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:29.176569+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175434+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:29.175434+0000","last_peered":"2024-09-18T11:56:29.175434+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:29.175434+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13125,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175862+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:29.175862+0000","last_peered":"2024-09-18T11:56:29.175862+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:29.175862+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.905625+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:19.905625+0000","last_peered":"2024-09-18T11:56:19.905625+0000","last_clean":"2024-09-18T11:56:19.905625+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:19.905625+0000","last_undegraded":"2024-09-18T11:56:19.905625+0000","last_fullsized":"2024-09-18T11:56:19.905625+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17529,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.175162+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:29.175162+0000","last_peered":"2024-09-18T11:56:29.175162+0000","last_clean":"2024-09-18T11:56:29.175162+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:29.175162+0000","last_undegraded":"2024-09-18T11:56:29.175162+0000","last_fullsized":"2024-09-18T11:56:29.175162+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12709,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.009148+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:20.009148+0000","last_peered":"2024-09-18T11:56:20.009148+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:20.009148+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13155,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176792+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:29.176792+0000","last_peered":"2024-09-18T11:56:29.176792+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:29.176792+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13127,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175723+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:29.175723+0000","last_peered":"2024-09-18T11:56:29.175723+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:29.175723+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13067,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.983313+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:19.983313+0000","last_peered":"2024-09-18T11:56:19.983313+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:19.983313+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13217,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175464+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:29.175464+0000","last_peered":"2024-09-18T11:56:29.175464+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:29.175464+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17192,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:19.974257+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:19.974257+0000","last_peered":"2024-09-18T11:56:19.974257+0000","last_clean":"2024-09-18T11:56:19.974257+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:19.974257+0000","last_undegraded":"2024-09-18T11:56:19.974257+0000","last_fullsized":"2024-09-18T11:56:19.974257+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":13536,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.899768+0000","last_change":"2024-09-18T11:56:28.893308+0000","last_active":"2024-09-18T11:56:30.899768+0000","last_peered":"2024-09-18T11:56:30.899768+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:30.899768+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":381,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11064,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":405504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923864,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447976,"statfs":{"total":96636764160,"available":95690727424,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.246},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.25800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919192,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452648,"statfs":{"total":96636764160,"available":95695511552,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.28899999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.20499999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.249},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.224}]}]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930376,"kb_used_data":704,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441464,"statfs":{"total":96636764160,"available":95684059136,"internally_reserved":0,"allocated":720896,"data_stored":528646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.218},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.26400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23200000000000001},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.245}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.30199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915492,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456348,"statfs":{"total":96636764160,"available":95699300352,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.17000000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.253}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:35.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:35.952 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:35.952 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:35.963 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-09-18T11:56:34.109737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11064,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693092,"kb_used_data":1716,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794268,"statfs":{"total":386547056640,"available":382765330432,"internally_reserved":0,"allocated":1757184,"data_stored":962696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1302,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001488"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13355,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:29.175345+0000","last_peered":"2024-09-18T11:56:29.175345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:29.175345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17350,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181792+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:29.181792+0000","last_peered":"2024-09-18T11:56:29.181792+0000","last_clean":"2024-09-18T11:56:29.181792+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:29.181792+0000","last_undegraded":"2024-09-18T11:56:29.181792+0000","last_fullsized":"2024-09-18T11:56:29.181792+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17014,"reported_epoch":40,"state":"peering","last_fresh":"2024-09-18T11:56:29.179792+0000","last_change":"2024-09-18T11:56:29.183009+0000","last_active":"2024-09-18T11:56:29.179753+0000","last_peered":"2024-09-18T11:56:28.892437+0000","last_clean":"2024-09-18T11:43:44.593892+0000","last_became_active":"2024-09-18T11:44:47.429842+0000","last_became_peered":"2024-09-18T11:44:47.429842+0000","last_unstale":"2024-09-18T11:56:29.179792+0000","last_undegraded":"2024-09-18T11:56:29.179792+0000","last_fullsized":"2024-09-18T11:56:29.179792+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.174803+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:29.174803+0000","last_peered":"2024-09-18T11:56:29.174803+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:29.174803+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13018,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176569+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:29.176569+0000","last_peered":"2024-09-18T11:56:29.176569+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:29.176569+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175434+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:29.175434+0000","last_peered":"2024-09-18T11:56:29.175434+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:29.175434+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13125,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175862+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:29.175862+0000","last_peered":"2024-09-18T11:56:29.175862+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:29.175862+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.182612+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:29.182612+0000","last_peered":"2024-09-18T11:56:29.182612+0000","last_clean":"2024-09-18T11:56:29.182612+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:29.182612+0000","last_undegraded":"2024-09-18T11:56:29.182612+0000","last_fullsized":"2024-09-18T11:56:29.182612+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17529,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.175162+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:29.175162+0000","last_peered":"2024-09-18T11:56:29.175162+0000","last_clean":"2024-09-18T11:56:29.175162+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:29.175162+0000","last_undegraded":"2024-09-18T11:56:29.175162+0000","last_fullsized":"2024-09-18T11:56:29.175162+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12709,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:20.009148+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:20.009148+0000","last_peered":"2024-09-18T11:56:20.009148+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:20.009148+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13155,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176792+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:29.176792+0000","last_peered":"2024-09-18T11:56:29.176792+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:29.176792+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13127,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175723+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:29.175723+0000","last_peered":"2024-09-18T11:56:29.175723+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:29.175723+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13067,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:19.983313+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:19.983313+0000","last_peered":"2024-09-18T11:56:19.983313+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:19.983313+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13217,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175464+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:29.175464+0000","last_peered":"2024-09-18T11:56:29.175464+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:29.175464+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17194,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181827+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:29.181827+0000","last_peered":"2024-09-18T11:56:29.181827+0000","last_clean":"2024-09-18T11:56:29.181827+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:29.181827+0000","last_undegraded":"2024-09-18T11:56:29.181827+0000","last_fullsized":"2024-09-18T11:56:29.181827+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":13536,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.899768+0000","last_change":"2024-09-18T11:56:28.893308+0000","last_active":"2024-09-18T11:56:30.899768+0000","last_peered":"2024-09-18T11:56:30.899768+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:30.899768+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":381,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11064,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":397312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923864,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447976,"statfs":{"total":96636764160,"available":95690727424,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4320000000000004,"5min":7.0510000000000002,"15min":5.0350000000000001},"min":{"1min":0.224,"5min":0.186,"15min":0.155},"max":{"1min":16.913,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.246},{"interface":"front","average":{"1min":4.806,"5min":6.1239999999999997,"15min":4.7880000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":13.340999999999999,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:25 2024","interfaces":[{"interface":"back","average":{"1min":6.5279999999999996,"5min":7.4560000000000004,"15min":4.5609999999999999},"min":{"1min":0.49199999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":14.784000000000001,"5min":42.752000000000002,"15min":42.752000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.863,"5min":4.1619999999999999,"15min":3.7679999999999998},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.20200000000000001},"max":{"1min":10.042,"5min":24.974,"15min":24.974},"last":0.25800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919192,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452648,"statfs":{"total":96636764160,"available":95695511552,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.28899999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.20499999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.249},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.224}]}]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934544,"kb_used_data":712,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437296,"statfs":{"total":96636764160,"available":95679791104,"internally_reserved":0,"allocated":729088,"data_stored":528335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.17599999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.19900000000000001},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.251}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.28599999999999998},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.32900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915492,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456348,"statfs":{"total":96636764160,"available":95699300352,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.17000000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.253}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:35.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:36.243 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:36.255 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-18T11:56:36.110065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11966,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693116,"kb_used_data":1732,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794244,"statfs":{"total":386547056640,"available":382765305856,"internally_reserved":0,"allocated":1773568,"data_stored":970556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001426"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13355,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:29.175345+0000","last_peered":"2024-09-18T11:56:29.175345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:29.175345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17350,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181792+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:29.181792+0000","last_peered":"2024-09-18T11:56:29.181792+0000","last_clean":"2024-09-18T11:56:29.181792+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:29.181792+0000","last_undegraded":"2024-09-18T11:56:29.181792+0000","last_fullsized":"2024-09-18T11:56:29.181792+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17024,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:56:30.259041+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:30.259041+0000","last_peered":"2024-09-18T11:56:30.259041+0000","last_clean":"2024-09-18T11:56:30.259041+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:30.259041+0000","last_undegraded":"2024-09-18T11:56:30.259041+0000","last_fullsized":"2024-09-18T11:56:30.259041+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.174803+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:29.174803+0000","last_peered":"2024-09-18T11:56:29.174803+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:29.174803+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13018,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176569+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:29.176569+0000","last_peered":"2024-09-18T11:56:29.176569+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:29.176569+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175434+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:29.175434+0000","last_peered":"2024-09-18T11:56:29.175434+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:29.175434+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13125,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175862+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:29.175862+0000","last_peered":"2024-09-18T11:56:29.175862+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:29.175862+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.182612+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:29.182612+0000","last_peered":"2024-09-18T11:56:29.182612+0000","last_clean":"2024-09-18T11:56:29.182612+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:29.182612+0000","last_undegraded":"2024-09-18T11:56:29.182612+0000","last_fullsized":"2024-09-18T11:56:29.182612+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17529,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.175162+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:29.175162+0000","last_peered":"2024-09-18T11:56:29.175162+0000","last_clean":"2024-09-18T11:56:29.175162+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:29.175162+0000","last_undegraded":"2024-09-18T11:56:29.175162+0000","last_fullsized":"2024-09-18T11:56:29.175162+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12710,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.184123+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:29.184123+0000","last_peered":"2024-09-18T11:56:29.184123+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:29.184123+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13155,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176792+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:29.176792+0000","last_peered":"2024-09-18T11:56:29.176792+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:29.176792+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13127,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175723+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:29.175723+0000","last_peered":"2024-09-18T11:56:29.175723+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:29.175723+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13068,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.178246+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:29.178246+0000","last_peered":"2024-09-18T11:56:29.178246+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:29.178246+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13217,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175464+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:29.175464+0000","last_peered":"2024-09-18T11:56:29.175464+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:29.175464+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17194,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181827+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:29.181827+0000","last_peered":"2024-09-18T11:56:29.181827+0000","last_clean":"2024-09-18T11:56:29.181827+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:29.181827+0000","last_undegraded":"2024-09-18T11:56:29.181827+0000","last_fullsized":"2024-09-18T11:56:29.181827+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":15340,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:35.900963+0000","last_change":"2024-09-18T11:56:28.893308+0000","last_active":"2024-09-18T11:56:35.900963+0000","last_peered":"2024-09-18T11:56:35.900963+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:35.900963+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11966,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":397312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923876,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447964,"statfs":{"total":96636764160,"available":95690715136,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.25},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.21299999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.309}]}]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919204,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452636,"statfs":{"total":96636764160,"available":95695499264,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.317}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.249},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.28499999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.23499999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.32600000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934544,"kb_used_data":712,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437296,"statfs":{"total":96636764160,"available":95679791104,"internally_reserved":0,"allocated":729088,"data_stored":528335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.17599999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.19900000000000001},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.251}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.28599999999999998},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.32900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915492,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456348,"statfs":{"total":96636764160,"available":95699300352,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.219},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.245},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.32200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13355, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.175345+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:29.175345+0000', 'last_peered': '2024-09-18T11:56:29.175345+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:29.175345+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13022, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.174803+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:29.174803+0000', 'last_peered': '2024-09-18T11:56:29.174803+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:29.174803+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13018, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.176569+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:29.176569+0000', 'last_peered': '2024-09-18T11:56:29.176569+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:29.176569+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12909, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.175434+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:29.175434+0000', 'last_peered': '2024-09-18T11:56:29.175434+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:29.175434+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'13014", 'reported_seq': 13125, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.175862+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:56:29.175862+0000', 'last_peered': '2024-09-18T11:56:29.175862+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:29.175862+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13012, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2114}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12710, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.184123+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:29.184123+0000', 'last_peered': '2024-09-18T11:56:29.184123+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:29.184123+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13030", 'reported_seq': 13155, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.176792+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:56:29.176792+0000', 'last_peered': '2024-09-18T11:56:29.176792+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:56:29.176792+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20337, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13027, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13127, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.175723+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:29.175723+0000', 'last_peered': '2024-09-18T11:56:29.175723+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:29.175723+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:36.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 13068, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.178246+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:29.178246+0000', 'last_peered': '2024-09-18T11:56:29.178246+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:29.178246+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:36.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:36.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13217, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.175464+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:29.175464+0000', 'last_peered': '2024-09-18T11:56:29.175464+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:29.175464+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:36.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:56:36.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12705", 'reported_seq': 15340, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:35.900963+0000', 'last_change': '2024-09-18T11:56:28.893308+0000', 'last_active': '2024-09-18T11:56:35.900963+0000', 'last_peered': '2024-09-18T11:56:35.900963+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:56:35.900963+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 734, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12703, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1283, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -732}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:36.258 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:36.601 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:36.601 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21490/112 objects degraded (19187.500%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":794,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3781750784,"bytes_avail":382765305856,"bytes_total":386547056640,"degraded_objects":21490,"degraded_total":112,"degraded_ratio":191.875,"recovering_objects_per_sec":275,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:36.614 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:36.614 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:36.889 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:36.889 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:36.900 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-18T11:56:36.110065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11966,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693116,"kb_used_data":1732,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794244,"statfs":{"total":386547056640,"available":382765305856,"internally_reserved":0,"allocated":1773568,"data_stored":970556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001426"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13355,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:29.175345+0000","last_peered":"2024-09-18T11:56:29.175345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:29.175345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17350,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181792+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:29.181792+0000","last_peered":"2024-09-18T11:56:29.181792+0000","last_clean":"2024-09-18T11:56:29.181792+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:29.181792+0000","last_undegraded":"2024-09-18T11:56:29.181792+0000","last_fullsized":"2024-09-18T11:56:29.181792+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17024,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:56:30.259041+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:30.259041+0000","last_peered":"2024-09-18T11:56:30.259041+0000","last_clean":"2024-09-18T11:56:30.259041+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:30.259041+0000","last_undegraded":"2024-09-18T11:56:30.259041+0000","last_fullsized":"2024-09-18T11:56:30.259041+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.174803+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:29.174803+0000","last_peered":"2024-09-18T11:56:29.174803+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:29.174803+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13018,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176569+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:29.176569+0000","last_peered":"2024-09-18T11:56:29.176569+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:29.176569+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175434+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:29.175434+0000","last_peered":"2024-09-18T11:56:29.175434+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:29.175434+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13125,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175862+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:29.175862+0000","last_peered":"2024-09-18T11:56:29.175862+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:29.175862+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.182612+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:29.182612+0000","last_peered":"2024-09-18T11:56:29.182612+0000","last_clean":"2024-09-18T11:56:29.182612+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:29.182612+0000","last_undegraded":"2024-09-18T11:56:29.182612+0000","last_fullsized":"2024-09-18T11:56:29.182612+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17529,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.175162+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:29.175162+0000","last_peered":"2024-09-18T11:56:29.175162+0000","last_clean":"2024-09-18T11:56:29.175162+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:29.175162+0000","last_undegraded":"2024-09-18T11:56:29.175162+0000","last_fullsized":"2024-09-18T11:56:29.175162+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12710,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.184123+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:29.184123+0000","last_peered":"2024-09-18T11:56:29.184123+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:29.184123+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13155,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176792+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:29.176792+0000","last_peered":"2024-09-18T11:56:29.176792+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:29.176792+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13127,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175723+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:29.175723+0000","last_peered":"2024-09-18T11:56:29.175723+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:29.175723+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13068,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.178246+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:29.178246+0000","last_peered":"2024-09-18T11:56:29.178246+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:29.178246+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13217,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175464+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:29.175464+0000","last_peered":"2024-09-18T11:56:29.175464+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:29.175464+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17194,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181827+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:29.181827+0000","last_peered":"2024-09-18T11:56:29.181827+0000","last_clean":"2024-09-18T11:56:29.181827+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:29.181827+0000","last_undegraded":"2024-09-18T11:56:29.181827+0000","last_fullsized":"2024-09-18T11:56:29.181827+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":15340,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:35.900963+0000","last_change":"2024-09-18T11:56:28.893308+0000","last_active":"2024-09-18T11:56:35.900963+0000","last_peered":"2024-09-18T11:56:35.900963+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:35.900963+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11966,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":397312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923876,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447964,"statfs":{"total":96636764160,"available":95690715136,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.25},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.21299999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.309}]}]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919204,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452636,"statfs":{"total":96636764160,"available":95695499264,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.317}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.249},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.28499999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.23499999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.32600000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934544,"kb_used_data":712,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437296,"statfs":{"total":96636764160,"available":95679791104,"internally_reserved":0,"allocated":729088,"data_stored":528335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.17599999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.19900000000000001},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.251}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.28599999999999998},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.32900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915492,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456348,"statfs":{"total":96636764160,"available":95699300352,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.219},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.245},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.32200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:39.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:40.187 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:40.188 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:40.199 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-18T11:56:40.110758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11966,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684800,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802560,"statfs":{"total":386547056640,"available":382773821440,"internally_reserved":0,"allocated":1781760,"data_stored":974470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.927568"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13355,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:29.175345+0000","last_peered":"2024-09-18T11:56:29.175345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:29.175345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17350,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181792+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:29.181792+0000","last_peered":"2024-09-18T11:56:29.181792+0000","last_clean":"2024-09-18T11:56:29.181792+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:29.181792+0000","last_undegraded":"2024-09-18T11:56:29.181792+0000","last_fullsized":"2024-09-18T11:56:29.181792+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17025,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:56:30.259649+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:30.259649+0000","last_peered":"2024-09-18T11:56:30.259649+0000","last_clean":"2024-09-18T11:56:30.259649+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:30.259649+0000","last_undegraded":"2024-09-18T11:56:30.259649+0000","last_fullsized":"2024-09-18T11:56:30.259649+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.174803+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:29.174803+0000","last_peered":"2024-09-18T11:56:29.174803+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:29.174803+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13018,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176569+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:29.176569+0000","last_peered":"2024-09-18T11:56:29.176569+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:29.176569+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175434+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:29.175434+0000","last_peered":"2024-09-18T11:56:29.175434+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:29.175434+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13125,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175862+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:29.175862+0000","last_peered":"2024-09-18T11:56:29.175862+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:29.175862+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.182612+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:29.182612+0000","last_peered":"2024-09-18T11:56:29.182612+0000","last_clean":"2024-09-18T11:56:29.182612+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:29.182612+0000","last_undegraded":"2024-09-18T11:56:29.182612+0000","last_fullsized":"2024-09-18T11:56:29.182612+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17529,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.175162+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:29.175162+0000","last_peered":"2024-09-18T11:56:29.175162+0000","last_clean":"2024-09-18T11:56:29.175162+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:29.175162+0000","last_undegraded":"2024-09-18T11:56:29.175162+0000","last_fullsized":"2024-09-18T11:56:29.175162+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12711,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.190414+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:30.190414+0000","last_peered":"2024-09-18T11:56:30.190414+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:30.190414+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13155,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176792+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:29.176792+0000","last_peered":"2024-09-18T11:56:29.176792+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:29.176792+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13127,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175723+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:29.175723+0000","last_peered":"2024-09-18T11:56:29.175723+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:29.175723+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13069,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.187746+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:30.187746+0000","last_peered":"2024-09-18T11:56:30.187746+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:30.187746+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13217,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175464+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:29.175464+0000","last_peered":"2024-09-18T11:56:29.175464+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:29.175464+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17194,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181827+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:29.181827+0000","last_peered":"2024-09-18T11:56:29.181827+0000","last_clean":"2024-09-18T11:56:29.181827+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:29.181827+0000","last_undegraded":"2024-09-18T11:56:29.181827+0000","last_fullsized":"2024-09-18T11:56:29.181827+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":15340,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:35.900963+0000","last_change":"2024-09-18T11:56:28.893308+0000","last_active":"2024-09-18T11:56:35.900963+0000","last_peered":"2024-09-18T11:56:35.900963+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:35.900963+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11966,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":397312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923876,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447964,"statfs":{"total":96636764160,"available":95690715136,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.25},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.21299999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.309}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919208,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452632,"statfs":{"total":96636764160,"available":95695495168,"internally_reserved":0,"allocated":356352,"data_stored":151321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.26200000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.311}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.28899999999999998},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27300000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926224,"kb_used_data":712,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445616,"statfs":{"total":96636764160,"available":95688310784,"internally_reserved":0,"allocated":729088,"data_stored":528335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.248}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.29799999999999999},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.23300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.27200000000000002},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.25900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915492,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456348,"statfs":{"total":96636764160,"available":95699300352,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.219},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.245},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.32200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:40.201 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:40.531 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:40.531 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:40.544 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-18T11:56:40.225187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11966,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684800,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802560,"statfs":{"total":386547056640,"available":382773821440,"internally_reserved":0,"allocated":1781760,"data_stored":974470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.115874"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13355,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:29.175345+0000","last_peered":"2024-09-18T11:56:29.175345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:29.175345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17350,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181792+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:29.181792+0000","last_peered":"2024-09-18T11:56:29.181792+0000","last_clean":"2024-09-18T11:56:29.181792+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:29.181792+0000","last_undegraded":"2024-09-18T11:56:29.181792+0000","last_fullsized":"2024-09-18T11:56:29.181792+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17025,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:56:30.259649+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:30.259649+0000","last_peered":"2024-09-18T11:56:30.259649+0000","last_clean":"2024-09-18T11:56:30.259649+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:30.259649+0000","last_undegraded":"2024-09-18T11:56:30.259649+0000","last_fullsized":"2024-09-18T11:56:30.259649+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.174803+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:29.174803+0000","last_peered":"2024-09-18T11:56:29.174803+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:29.174803+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13018,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176569+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:29.176569+0000","last_peered":"2024-09-18T11:56:29.176569+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:29.176569+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175434+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:29.175434+0000","last_peered":"2024-09-18T11:56:29.175434+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:29.175434+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13125,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175862+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:29.175862+0000","last_peered":"2024-09-18T11:56:29.175862+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:29.175862+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.182612+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:29.182612+0000","last_peered":"2024-09-18T11:56:29.182612+0000","last_clean":"2024-09-18T11:56:29.182612+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:29.182612+0000","last_undegraded":"2024-09-18T11:56:29.182612+0000","last_fullsized":"2024-09-18T11:56:29.182612+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17529,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.175162+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:29.175162+0000","last_peered":"2024-09-18T11:56:29.175162+0000","last_clean":"2024-09-18T11:56:29.175162+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:29.175162+0000","last_undegraded":"2024-09-18T11:56:29.175162+0000","last_fullsized":"2024-09-18T11:56:29.175162+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12711,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.190414+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:30.190414+0000","last_peered":"2024-09-18T11:56:30.190414+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:30.190414+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13155,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176792+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:29.176792+0000","last_peered":"2024-09-18T11:56:29.176792+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:29.176792+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13127,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175723+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:29.175723+0000","last_peered":"2024-09-18T11:56:29.175723+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:29.175723+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13069,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.187746+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:30.187746+0000","last_peered":"2024-09-18T11:56:30.187746+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:30.187746+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13217,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175464+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:29.175464+0000","last_peered":"2024-09-18T11:56:29.175464+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:29.175464+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17194,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181827+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:29.181827+0000","last_peered":"2024-09-18T11:56:29.181827+0000","last_clean":"2024-09-18T11:56:29.181827+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:29.181827+0000","last_undegraded":"2024-09-18T11:56:29.181827+0000","last_fullsized":"2024-09-18T11:56:29.181827+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":15340,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:35.900963+0000","last_change":"2024-09-18T11:56:28.893308+0000","last_active":"2024-09-18T11:56:35.900963+0000","last_peered":"2024-09-18T11:56:35.900963+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:35.900963+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11966,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":397312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923876,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447964,"statfs":{"total":96636764160,"available":95690715136,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.25},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.21299999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.309}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919208,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452632,"statfs":{"total":96636764160,"available":95695495168,"internally_reserved":0,"allocated":356352,"data_stored":151321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.26200000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.311}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.28899999999999998},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27300000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926224,"kb_used_data":712,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445616,"statfs":{"total":96636764160,"available":95688310784,"internally_reserved":0,"allocated":729088,"data_stored":528335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.248}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.29799999999999999},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.23300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.27200000000000002},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.25900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915492,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456348,"statfs":{"total":96636764160,"available":95699300352,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.219},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.245},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.32200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:40.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:40.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13355, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.175345+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:29.175345+0000', 'last_peered': '2024-09-18T11:56:29.175345+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:29.175345+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:40.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:40.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13022, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.174803+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:29.174803+0000', 'last_peered': '2024-09-18T11:56:29.174803+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:29.174803+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:40.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:40.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13018, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.176569+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:29.176569+0000', 'last_peered': '2024-09-18T11:56:29.176569+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:29.176569+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12909, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.175434+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:29.175434+0000', 'last_peered': '2024-09-18T11:56:29.175434+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:29.175434+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'13014", 'reported_seq': 13125, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.175862+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:56:29.175862+0000', 'last_peered': '2024-09-18T11:56:29.175862+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:29.175862+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13012, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2114}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12711, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:30.190414+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:30.190414+0000', 'last_peered': '2024-09-18T11:56:30.190414+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:30.190414+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13030", 'reported_seq': 13155, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.176792+0000', 'last_change': '2024-09-18T11:44:47.366982+0000', 'last_active': '2024-09-18T11:56:29.176792+0000', 'last_peered': '2024-09-18T11:56:29.176792+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:56:29.176792+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20337, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13027, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13127, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.175723+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:29.175723+0000', 'last_peered': '2024-09-18T11:56:29.175723+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:29.175723+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:40.546 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:40.547 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 13069, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:30.187746+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:30.187746+0000', 'last_peered': '2024-09-18T11:56:30.187746+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:30.187746+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:40.547 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:40.547 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13217, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:29.175464+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:29.175464+0000', 'last_peered': '2024-09-18T11:56:29.175464+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:29.175464+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:40.547 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-18T11:56:40.547 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12705", 'reported_seq': 15340, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:35.900963+0000', 'last_change': '2024-09-18T11:56:28.893308+0000', 'last_active': '2024-09-18T11:56:35.900963+0000', 'last_peered': '2024-09-18T11:56:35.900963+0000', 'last_clean': '2024-09-18T11:43:44.009098+0000', 'last_became_active': '2024-09-18T11:44:47.451914+0000', 'last_became_peered': '2024-09-18T11:44:47.451914+0000', 'last_unstale': '2024-09-18T11:56:35.900963+0000', 'last_undegraded': '2024-09-18T11:44:47.156228+0000', 'last_fullsized': '2024-09-18T11:44:47.123387+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 734, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12703, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1283, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -732}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:40.547 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:40.932 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:40.932 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21490/112 objects degraded (19187.500%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":799,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3773235200,"bytes_avail":382773821440,"bytes_total":386547056640,"degraded_objects":21490,"degraded_total":112,"degraded_ratio":191.875,"recovering_objects_per_sec":127,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1270,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:40.945 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:40.945 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:41.223 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:41.223 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:41.236 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-18T11:56:40.225187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11966,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684800,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802560,"statfs":{"total":386547056640,"available":382773821440,"internally_reserved":0,"allocated":1781760,"data_stored":974470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.115874"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13355,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175345+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:29.175345+0000","last_peered":"2024-09-18T11:56:29.175345+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:29.175345+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17350,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181792+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:29.181792+0000","last_peered":"2024-09-18T11:56:29.181792+0000","last_clean":"2024-09-18T11:56:29.181792+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:29.181792+0000","last_undegraded":"2024-09-18T11:56:29.181792+0000","last_fullsized":"2024-09-18T11:56:29.181792+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17025,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:56:30.259649+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:30.259649+0000","last_peered":"2024-09-18T11:56:30.259649+0000","last_clean":"2024-09-18T11:56:30.259649+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:30.259649+0000","last_undegraded":"2024-09-18T11:56:30.259649+0000","last_fullsized":"2024-09-18T11:56:30.259649+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.174803+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:29.174803+0000","last_peered":"2024-09-18T11:56:29.174803+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:29.174803+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13018,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176569+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:29.176569+0000","last_peered":"2024-09-18T11:56:29.176569+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:29.176569+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175434+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:29.175434+0000","last_peered":"2024-09-18T11:56:29.175434+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:29.175434+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13125,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175862+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:29.175862+0000","last_peered":"2024-09-18T11:56:29.175862+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:29.175862+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.182612+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:29.182612+0000","last_peered":"2024-09-18T11:56:29.182612+0000","last_clean":"2024-09-18T11:56:29.182612+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:29.182612+0000","last_undegraded":"2024-09-18T11:56:29.182612+0000","last_fullsized":"2024-09-18T11:56:29.182612+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17529,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.175162+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:29.175162+0000","last_peered":"2024-09-18T11:56:29.175162+0000","last_clean":"2024-09-18T11:56:29.175162+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:29.175162+0000","last_undegraded":"2024-09-18T11:56:29.175162+0000","last_fullsized":"2024-09-18T11:56:29.175162+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12711,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.190414+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:30.190414+0000","last_peered":"2024-09-18T11:56:30.190414+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:30.190414+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13155,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.176792+0000","last_change":"2024-09-18T11:44:47.366982+0000","last_active":"2024-09-18T11:56:29.176792+0000","last_peered":"2024-09-18T11:56:29.176792+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:29.176792+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13127,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175723+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:29.175723+0000","last_peered":"2024-09-18T11:56:29.175723+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:29.175723+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13069,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.187746+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:30.187746+0000","last_peered":"2024-09-18T11:56:30.187746+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:30.187746+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13217,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:29.175464+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:29.175464+0000","last_peered":"2024-09-18T11:56:29.175464+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:29.175464+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17194,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:56:29.181827+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:29.181827+0000","last_peered":"2024-09-18T11:56:29.181827+0000","last_clean":"2024-09-18T11:56:29.181827+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:29.181827+0000","last_undegraded":"2024-09-18T11:56:29.181827+0000","last_fullsized":"2024-09-18T11:56:29.181827+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":15340,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:35.900963+0000","last_change":"2024-09-18T11:56:28.893308+0000","last_active":"2024-09-18T11:56:35.900963+0000","last_peered":"2024-09-18T11:56:35.900963+0000","last_clean":"2024-09-18T11:43:44.009098+0000","last_became_active":"2024-09-18T11:44:47.451914+0000","last_became_peered":"2024-09-18T11:44:47.451914+0000","last_unstale":"2024-09-18T11:56:35.900963+0000","last_undegraded":"2024-09-18T11:44:47.156228+0000","last_fullsized":"2024-09-18T11:44:47.123387+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11966,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":397312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923876,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447964,"statfs":{"total":96636764160,"available":95690715136,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.25},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.21299999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.309}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919208,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452632,"statfs":{"total":96636764160,"available":95695495168,"internally_reserved":0,"allocated":356352,"data_stored":151321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.26200000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.311}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.28899999999999998},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27300000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926224,"kb_used_data":712,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445616,"statfs":{"total":96636764160,"available":95688310784,"internally_reserved":0,"allocated":729088,"data_stored":528335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.248}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.29799999999999999},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.23300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.27200000000000002},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.25900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915492,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456348,"statfs":{"total":96636764160,"available":95699300352,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.219},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.245},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.32200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:44.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:44.563 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:44.564 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:44.575 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-18T11:56:44.111459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12932,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693144,"kb_used_data":1768,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794216,"statfs":{"total":386547056640,"available":382765277184,"internally_reserved":0,"allocated":1810432,"data_stored":981710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001394"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13357,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222270+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:39.222270+0000","last_peered":"2024-09-18T11:56:39.222270+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:39.222270+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17352,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219620+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:39.219620+0000","last_peered":"2024-09-18T11:56:39.219620+0000","last_clean":"2024-09-18T11:56:39.219620+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:39.219620+0000","last_undegraded":"2024-09-18T11:56:39.219620+0000","last_fullsized":"2024-09-18T11:56:39.219620+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17025,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:56:30.259649+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:30.259649+0000","last_peered":"2024-09-18T11:56:30.259649+0000","last_clean":"2024-09-18T11:56:30.259649+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:30.259649+0000","last_undegraded":"2024-09-18T11:56:30.259649+0000","last_fullsized":"2024-09-18T11:56:30.259649+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13024,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.217483+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:39.217483+0000","last_peered":"2024-09-18T11:56:39.217483+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:39.217483+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13020,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:39.222177+0000","last_peered":"2024-09-18T11:56:39.222177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:39.222177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13127,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.218062+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:39.218062+0000","last_peered":"2024-09-18T11:56:39.218062+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:39.218062+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:39.219653+0000","last_peered":"2024-09-18T11:56:39.219653+0000","last_clean":"2024-09-18T11:56:39.219653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:39.219653+0000","last_undegraded":"2024-09-18T11:56:39.219653+0000","last_fullsized":"2024-09-18T11:56:39.219653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17531,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.222148+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:39.222148+0000","last_peered":"2024-09-18T11:56:39.222148+0000","last_clean":"2024-09-18T11:56:39.222148+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:39.222148+0000","last_undegraded":"2024-09-18T11:56:39.222148+0000","last_fullsized":"2024-09-18T11:56:39.222148+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12711,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.190414+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:30.190414+0000","last_peered":"2024-09-18T11:56:30.190414+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:30.190414+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13622,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:40.257448+0000","last_change":"2024-09-18T11:56:38.980663+0000","last_active":"2024-09-18T11:56:40.257448+0000","last_peered":"2024-09-18T11:56:40.257448+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:40.257448+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13129,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13069,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.187746+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:30.187746+0000","last_peered":"2024-09-18T11:56:30.187746+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:30.187746+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222196+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:39.222196+0000","last_peered":"2024-09-18T11:56:39.222196+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:39.222196+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17196,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:39.219637+0000","last_peered":"2024-09-18T11:56:39.219637+0000","last_clean":"2024-09-18T11:56:39.219637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:39.219637+0000","last_undegraded":"2024-09-18T11:56:39.219637+0000","last_fullsized":"2024-09-18T11:56:39.219637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12812,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.305343+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:40.305343+0000","last_peered":"2024-09-18T11:56:40.305343+0000","last_clean":"2024-09-18T11:56:40.305343+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:40.305343+0000","last_undegraded":"2024-09-18T11:56:40.305343+0000","last_fullsized":"2024-09-18T11:56:40.305343+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12932,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923880,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447960,"statfs":{"total":96636764160,"available":95690711040,"internally_reserved":0,"allocated":356352,"data_stored":151321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.23400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.19700000000000001},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.25600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919208,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452632,"statfs":{"total":96636764160,"available":95695495168,"internally_reserved":0,"allocated":356352,"data_stored":151321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.26200000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.311}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.28899999999999998},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27300000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934552,"kb_used_data":716,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437288,"statfs":{"total":96636764160,"available":95679782912,"internally_reserved":0,"allocated":733184,"data_stored":523854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.18099999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.28199999999999997},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.20100000000000001},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.308}]}]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915504,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456336,"statfs":{"total":96636764160,"available":95699288064,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28899999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.316}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:44.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:44.865 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:44.866 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:44.878 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-18T11:56:44.111459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12932,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693144,"kb_used_data":1768,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794216,"statfs":{"total":386547056640,"available":382765277184,"internally_reserved":0,"allocated":1810432,"data_stored":981710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001394"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13357,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222270+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:39.222270+0000","last_peered":"2024-09-18T11:56:39.222270+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:39.222270+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17352,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219620+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:39.219620+0000","last_peered":"2024-09-18T11:56:39.219620+0000","last_clean":"2024-09-18T11:56:39.219620+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:39.219620+0000","last_undegraded":"2024-09-18T11:56:39.219620+0000","last_fullsized":"2024-09-18T11:56:39.219620+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17025,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:56:30.259649+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:30.259649+0000","last_peered":"2024-09-18T11:56:30.259649+0000","last_clean":"2024-09-18T11:56:30.259649+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:30.259649+0000","last_undegraded":"2024-09-18T11:56:30.259649+0000","last_fullsized":"2024-09-18T11:56:30.259649+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13024,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.217483+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:39.217483+0000","last_peered":"2024-09-18T11:56:39.217483+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:39.217483+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13020,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:39.222177+0000","last_peered":"2024-09-18T11:56:39.222177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:39.222177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13127,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.218062+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:39.218062+0000","last_peered":"2024-09-18T11:56:39.218062+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:39.218062+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:39.219653+0000","last_peered":"2024-09-18T11:56:39.219653+0000","last_clean":"2024-09-18T11:56:39.219653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:39.219653+0000","last_undegraded":"2024-09-18T11:56:39.219653+0000","last_fullsized":"2024-09-18T11:56:39.219653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17531,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.222148+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:39.222148+0000","last_peered":"2024-09-18T11:56:39.222148+0000","last_clean":"2024-09-18T11:56:39.222148+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:39.222148+0000","last_undegraded":"2024-09-18T11:56:39.222148+0000","last_fullsized":"2024-09-18T11:56:39.222148+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12711,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.190414+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:30.190414+0000","last_peered":"2024-09-18T11:56:30.190414+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:30.190414+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13622,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:40.257448+0000","last_change":"2024-09-18T11:56:38.980663+0000","last_active":"2024-09-18T11:56:40.257448+0000","last_peered":"2024-09-18T11:56:40.257448+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:40.257448+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13129,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13069,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.187746+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:30.187746+0000","last_peered":"2024-09-18T11:56:30.187746+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:30.187746+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222196+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:39.222196+0000","last_peered":"2024-09-18T11:56:39.222196+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:39.222196+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17196,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:39.219637+0000","last_peered":"2024-09-18T11:56:39.219637+0000","last_clean":"2024-09-18T11:56:39.219637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:39.219637+0000","last_undegraded":"2024-09-18T11:56:39.219637+0000","last_fullsized":"2024-09-18T11:56:39.219637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12812,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.305343+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:40.305343+0000","last_peered":"2024-09-18T11:56:40.305343+0000","last_clean":"2024-09-18T11:56:40.305343+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:40.305343+0000","last_undegraded":"2024-09-18T11:56:40.305343+0000","last_fullsized":"2024-09-18T11:56:40.305343+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12932,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923880,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447960,"statfs":{"total":96636764160,"available":95690711040,"internally_reserved":0,"allocated":356352,"data_stored":151321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.23400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.19700000000000001},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.25600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919208,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452632,"statfs":{"total":96636764160,"available":95695495168,"internally_reserved":0,"allocated":356352,"data_stored":151321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.26200000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.311}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.28899999999999998},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27300000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934552,"kb_used_data":716,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437288,"statfs":{"total":96636764160,"available":95679782912,"internally_reserved":0,"allocated":733184,"data_stored":523854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.18099999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.28199999999999997},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.20100000000000001},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.308}]}]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915504,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456336,"statfs":{"total":96636764160,"available":95699288064,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28899999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.316}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:44.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:44.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13357, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.222270+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:39.222270+0000', 'last_peered': '2024-09-18T11:56:39.222270+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:39.222270+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:44.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:44.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13024, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.217483+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:39.217483+0000', 'last_peered': '2024-09-18T11:56:39.217483+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:39.217483+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:44.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:44.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13020, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.222177+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:39.222177+0000', 'last_peered': '2024-09-18T11:56:39.222177+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:39.222177+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:44.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:44.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12911, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.222435+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:39.222435+0000', 'last_peered': '2024-09-18T11:56:39.222435+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:39.222435+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:44.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:44.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'13014", 'reported_seq': 13127, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.218062+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:56:39.218062+0000', 'last_peered': '2024-09-18T11:56:39.218062+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:39.218062+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13012, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2114}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:44.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:44.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12711, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:30.190414+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:30.190414+0000', 'last_peered': '2024-09-18T11:56:30.190414+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:30.190414+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:44.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:44.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13030", 'reported_seq': 13622, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:40.257448+0000', 'last_change': '2024-09-18T11:56:38.980663+0000', 'last_active': '2024-09-18T11:56:40.257448+0000', 'last_peered': '2024-09-18T11:56:40.257448+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:56:40.257448+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20337, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13027, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 264, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1802}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:44.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:44.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13129, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.222435+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:39.222435+0000', 'last_peered': '2024-09-18T11:56:39.222435+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:39.222435+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:44.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:44.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 13069, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:30.187746+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:30.187746+0000', 'last_peered': '2024-09-18T11:56:30.187746+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:30.187746+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:44.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:44.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13219, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.222196+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:39.222196+0000', 'last_peered': '2024-09-18T11:56:39.222196+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:39.222196+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:44.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:45.244 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:45.244 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20524/112 objects degraded (18325.000%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":803,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3781779456,"bytes_avail":382765277184,"bytes_total":386547056640,"degraded_objects":20524,"degraded_total":112,"degraded_ratio":183.25,"recovering_objects_per_sec":120,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:45.258 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:45.536 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:45.537 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:45.549 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-18T11:56:44.111459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12932,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693144,"kb_used_data":1768,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794216,"statfs":{"total":386547056640,"available":382765277184,"internally_reserved":0,"allocated":1810432,"data_stored":981710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001394"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13357,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222270+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:39.222270+0000","last_peered":"2024-09-18T11:56:39.222270+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:39.222270+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17352,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219620+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:39.219620+0000","last_peered":"2024-09-18T11:56:39.219620+0000","last_clean":"2024-09-18T11:56:39.219620+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:39.219620+0000","last_undegraded":"2024-09-18T11:56:39.219620+0000","last_fullsized":"2024-09-18T11:56:39.219620+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17025,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:56:30.259649+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:30.259649+0000","last_peered":"2024-09-18T11:56:30.259649+0000","last_clean":"2024-09-18T11:56:30.259649+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:30.259649+0000","last_undegraded":"2024-09-18T11:56:30.259649+0000","last_fullsized":"2024-09-18T11:56:30.259649+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13024,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.217483+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:39.217483+0000","last_peered":"2024-09-18T11:56:39.217483+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:39.217483+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13020,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:39.222177+0000","last_peered":"2024-09-18T11:56:39.222177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:39.222177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13127,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.218062+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:39.218062+0000","last_peered":"2024-09-18T11:56:39.218062+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:39.218062+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:39.219653+0000","last_peered":"2024-09-18T11:56:39.219653+0000","last_clean":"2024-09-18T11:56:39.219653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:39.219653+0000","last_undegraded":"2024-09-18T11:56:39.219653+0000","last_fullsized":"2024-09-18T11:56:39.219653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17531,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.222148+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:39.222148+0000","last_peered":"2024-09-18T11:56:39.222148+0000","last_clean":"2024-09-18T11:56:39.222148+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:39.222148+0000","last_undegraded":"2024-09-18T11:56:39.222148+0000","last_fullsized":"2024-09-18T11:56:39.222148+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12711,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.190414+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:30.190414+0000","last_peered":"2024-09-18T11:56:30.190414+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:30.190414+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":13622,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:40.257448+0000","last_change":"2024-09-18T11:56:38.980663+0000","last_active":"2024-09-18T11:56:40.257448+0000","last_peered":"2024-09-18T11:56:40.257448+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:40.257448+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13129,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13069,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:30.187746+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:30.187746+0000","last_peered":"2024-09-18T11:56:30.187746+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:30.187746+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222196+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:39.222196+0000","last_peered":"2024-09-18T11:56:39.222196+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:39.222196+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17196,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:39.219637+0000","last_peered":"2024-09-18T11:56:39.219637+0000","last_clean":"2024-09-18T11:56:39.219637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:39.219637+0000","last_undegraded":"2024-09-18T11:56:39.219637+0000","last_fullsized":"2024-09-18T11:56:39.219637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12812,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.305343+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:40.305343+0000","last_peered":"2024-09-18T11:56:40.305343+0000","last_clean":"2024-09-18T11:56:40.305343+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:40.305343+0000","last_undegraded":"2024-09-18T11:56:40.305343+0000","last_fullsized":"2024-09-18T11:56:40.305343+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12932,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923880,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447960,"statfs":{"total":96636764160,"available":95690711040,"internally_reserved":0,"allocated":356352,"data_stored":151321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.23400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.19700000000000001},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.25600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919208,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452632,"statfs":{"total":96636764160,"available":95695495168,"internally_reserved":0,"allocated":356352,"data_stored":151321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.26200000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.311}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.28899999999999998},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27300000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934552,"kb_used_data":716,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437288,"statfs":{"total":96636764160,"available":95679782912,"internally_reserved":0,"allocated":733184,"data_stored":523854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.18099999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.28199999999999997},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.20100000000000001},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.308}]}]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915504,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456336,"statfs":{"total":96636764160,"available":95699288064,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28899999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.316}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:48.551 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:48.820 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:48.821 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:48.833 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-09-18T11:56:48.112141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13840,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693160,"kb_used_data":1784,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794200,"statfs":{"total":386547056640,"available":382765260800,"internally_reserved":0,"allocated":1826816,"data_stored":989496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.898128"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13357,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222270+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:39.222270+0000","last_peered":"2024-09-18T11:56:39.222270+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:39.222270+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17352,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219620+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:39.219620+0000","last_peered":"2024-09-18T11:56:39.219620+0000","last_clean":"2024-09-18T11:56:39.219620+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:39.219620+0000","last_undegraded":"2024-09-18T11:56:39.219620+0000","last_fullsized":"2024-09-18T11:56:39.219620+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17026,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219781+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:39.219781+0000","last_peered":"2024-09-18T11:56:39.219781+0000","last_clean":"2024-09-18T11:56:39.219781+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:39.219781+0000","last_undegraded":"2024-09-18T11:56:39.219781+0000","last_fullsized":"2024-09-18T11:56:39.219781+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13024,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.217483+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:39.217483+0000","last_peered":"2024-09-18T11:56:39.217483+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:39.217483+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13020,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:39.222177+0000","last_peered":"2024-09-18T11:56:39.222177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:39.222177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13127,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.218062+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:39.218062+0000","last_peered":"2024-09-18T11:56:39.218062+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:39.218062+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:39.219653+0000","last_peered":"2024-09-18T11:56:39.219653+0000","last_clean":"2024-09-18T11:56:39.219653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:39.219653+0000","last_undegraded":"2024-09-18T11:56:39.219653+0000","last_fullsized":"2024-09-18T11:56:39.219653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17531,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.222148+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:39.222148+0000","last_peered":"2024-09-18T11:56:39.222148+0000","last_clean":"2024-09-18T11:56:39.222148+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:39.222148+0000","last_undegraded":"2024-09-18T11:56:39.222148+0000","last_fullsized":"2024-09-18T11:56:39.222148+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12712,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.220036+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:39.220036+0000","last_peered":"2024-09-18T11:56:39.220036+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:39.220036+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":15438,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:45.259532+0000","last_change":"2024-09-18T11:56:38.980663+0000","last_active":"2024-09-18T11:56:45.259532+0000","last_peered":"2024-09-18T11:56:45.259532+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:45.259532+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-894}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13129,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13070,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.219484+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:39.219484+0000","last_peered":"2024-09-18T11:56:39.219484+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:39.219484+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222196+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:39.222196+0000","last_peered":"2024-09-18T11:56:39.222196+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:39.222196+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17196,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:39.219637+0000","last_peered":"2024-09-18T11:56:39.219637+0000","last_clean":"2024-09-18T11:56:39.219637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:39.219637+0000","last_undegraded":"2024-09-18T11:56:39.219637+0000","last_fullsized":"2024-09-18T11:56:39.219637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12812,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.305343+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:40.305343+0000","last_peered":"2024-09-18T11:56:40.305343+0000","last_clean":"2024-09-18T11:56:40.305343+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:40.305343+0000","last_undegraded":"2024-09-18T11:56:40.305343+0000","last_fullsized":"2024-09-18T11:56:40.305343+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13840,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923888,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447952,"statfs":{"total":96636764160,"available":95690702848,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.185},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.247}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.222},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.28100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919216,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452624,"statfs":{"total":96636764160,"available":95695486976,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.30499999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.188},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.215}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.252},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.23100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934552,"kb_used_data":716,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437288,"statfs":{"total":96636764160,"available":95679782912,"internally_reserved":0,"allocated":733184,"data_stored":523854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.18099999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.28199999999999997},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.20100000000000001},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.308}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915504,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456336,"statfs":{"total":96636764160,"available":95699288064,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.16700000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.246},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:48.835 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:49.113 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:49.126 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-09-18T11:56:48.112141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13840,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693160,"kb_used_data":1784,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794200,"statfs":{"total":386547056640,"available":382765260800,"internally_reserved":0,"allocated":1826816,"data_stored":989496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.898128"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13357,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222270+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:39.222270+0000","last_peered":"2024-09-18T11:56:39.222270+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:39.222270+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17352,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219620+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:39.219620+0000","last_peered":"2024-09-18T11:56:39.219620+0000","last_clean":"2024-09-18T11:56:39.219620+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:39.219620+0000","last_undegraded":"2024-09-18T11:56:39.219620+0000","last_fullsized":"2024-09-18T11:56:39.219620+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17026,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219781+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:39.219781+0000","last_peered":"2024-09-18T11:56:39.219781+0000","last_clean":"2024-09-18T11:56:39.219781+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:39.219781+0000","last_undegraded":"2024-09-18T11:56:39.219781+0000","last_fullsized":"2024-09-18T11:56:39.219781+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13024,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.217483+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:39.217483+0000","last_peered":"2024-09-18T11:56:39.217483+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:39.217483+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13020,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:39.222177+0000","last_peered":"2024-09-18T11:56:39.222177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:39.222177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13127,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.218062+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:39.218062+0000","last_peered":"2024-09-18T11:56:39.218062+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:39.218062+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:39.219653+0000","last_peered":"2024-09-18T11:56:39.219653+0000","last_clean":"2024-09-18T11:56:39.219653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:39.219653+0000","last_undegraded":"2024-09-18T11:56:39.219653+0000","last_fullsized":"2024-09-18T11:56:39.219653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17531,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.222148+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:39.222148+0000","last_peered":"2024-09-18T11:56:39.222148+0000","last_clean":"2024-09-18T11:56:39.222148+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:39.222148+0000","last_undegraded":"2024-09-18T11:56:39.222148+0000","last_fullsized":"2024-09-18T11:56:39.222148+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12712,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.220036+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:39.220036+0000","last_peered":"2024-09-18T11:56:39.220036+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:39.220036+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":15438,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:45.259532+0000","last_change":"2024-09-18T11:56:38.980663+0000","last_active":"2024-09-18T11:56:45.259532+0000","last_peered":"2024-09-18T11:56:45.259532+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:45.259532+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-894}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13129,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13070,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.219484+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:39.219484+0000","last_peered":"2024-09-18T11:56:39.219484+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:39.219484+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222196+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:39.222196+0000","last_peered":"2024-09-18T11:56:39.222196+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:39.222196+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17196,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:39.219637+0000","last_peered":"2024-09-18T11:56:39.219637+0000","last_clean":"2024-09-18T11:56:39.219637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:39.219637+0000","last_undegraded":"2024-09-18T11:56:39.219637+0000","last_fullsized":"2024-09-18T11:56:39.219637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12812,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.305343+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:40.305343+0000","last_peered":"2024-09-18T11:56:40.305343+0000","last_clean":"2024-09-18T11:56:40.305343+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:40.305343+0000","last_undegraded":"2024-09-18T11:56:40.305343+0000","last_fullsized":"2024-09-18T11:56:40.305343+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13840,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923888,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447952,"statfs":{"total":96636764160,"available":95690702848,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.185},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.247}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.222},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.28100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919216,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452624,"statfs":{"total":96636764160,"available":95695486976,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.30499999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.188},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.215}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.252},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.23100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934552,"kb_used_data":716,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437288,"statfs":{"total":96636764160,"available":95679782912,"internally_reserved":0,"allocated":733184,"data_stored":523854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.18099999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.28199999999999997},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.20100000000000001},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.308}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915504,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456336,"statfs":{"total":96636764160,"available":95699288064,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.16700000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.246},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:49.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:49.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13357, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.222270+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:39.222270+0000', 'last_peered': '2024-09-18T11:56:39.222270+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:39.222270+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:49.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:49.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13024, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.217483+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:39.217483+0000', 'last_peered': '2024-09-18T11:56:39.217483+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:39.217483+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:49.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:49.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13020, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.222177+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:39.222177+0000', 'last_peered': '2024-09-18T11:56:39.222177+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:39.222177+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12911, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.222435+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:39.222435+0000', 'last_peered': '2024-09-18T11:56:39.222435+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:39.222435+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'13014", 'reported_seq': 13127, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.218062+0000', 'last_change': '2024-09-18T11:44:47.503126+0000', 'last_active': '2024-09-18T11:56:39.218062+0000', 'last_peered': '2024-09-18T11:56:39.218062+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:39.218062+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13012, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2114}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12712, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.220036+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:39.220036+0000', 'last_peered': '2024-09-18T11:56:39.220036+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:39.220036+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13030", 'reported_seq': 15438, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:45.259532+0000', 'last_change': '2024-09-18T11:56:38.980663+0000', 'last_active': '2024-09-18T11:56:45.259532+0000', 'last_peered': '2024-09-18T11:56:45.259532+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:44:47.359963+0000', 'last_became_peered': '2024-09-18T11:44:47.359963+0000', 'last_unstale': '2024-09-18T11:56:45.259532+0000', 'last_undegraded': '2024-09-18T11:44:47.169839+0000', 'last_fullsized': '2024-09-18T11:44:47.118676+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20337, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 897, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13027, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1172, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -894}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13129, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.222435+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:39.222435+0000', 'last_peered': '2024-09-18T11:56:39.222435+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:39.222435+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:49.128 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:49.129 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 13070, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.219484+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:39.219484+0000', 'last_peered': '2024-09-18T11:56:39.219484+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:39.219484+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:49.129 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:49.129 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13219, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:39.222196+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:39.222196+0000', 'last_peered': '2024-09-18T11:56:39.222196+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:39.222196+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:49.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:49.579 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:49.579 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19616/112 objects degraded (17514.286%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":807,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3781795840,"bytes_avail":382765260800,"bytes_total":386547056640,"degraded_objects":19616,"degraded_total":112,"degraded_ratio":175.14285714285714,"recovering_objects_per_sec":210,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1874,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:49.592 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:49.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:49.874 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:49.874 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:49.886 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-18T11:56:49.249166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13840,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684836,"kb_used_data":1784,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802524,"statfs":{"total":386547056640,"available":382773784576,"internally_reserved":0,"allocated":1826816,"data_stored":989496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.138408"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13357,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222270+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:39.222270+0000","last_peered":"2024-09-18T11:56:39.222270+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:39.222270+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17352,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219620+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:39.219620+0000","last_peered":"2024-09-18T11:56:39.219620+0000","last_clean":"2024-09-18T11:56:39.219620+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:39.219620+0000","last_undegraded":"2024-09-18T11:56:39.219620+0000","last_fullsized":"2024-09-18T11:56:39.219620+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17026,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219781+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:39.219781+0000","last_peered":"2024-09-18T11:56:39.219781+0000","last_clean":"2024-09-18T11:56:39.219781+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:39.219781+0000","last_undegraded":"2024-09-18T11:56:39.219781+0000","last_fullsized":"2024-09-18T11:56:39.219781+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13024,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.217483+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:39.217483+0000","last_peered":"2024-09-18T11:56:39.217483+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:39.217483+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13020,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222177+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:39.222177+0000","last_peered":"2024-09-18T11:56:39.222177+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:39.222177+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12911,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13127,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.218062+0000","last_change":"2024-09-18T11:44:47.503126+0000","last_active":"2024-09-18T11:56:39.218062+0000","last_peered":"2024-09-18T11:56:39.218062+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:39.218062+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:39.219653+0000","last_peered":"2024-09-18T11:56:39.219653+0000","last_clean":"2024-09-18T11:56:39.219653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:39.219653+0000","last_undegraded":"2024-09-18T11:56:39.219653+0000","last_fullsized":"2024-09-18T11:56:39.219653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17531,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.222148+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:39.222148+0000","last_peered":"2024-09-18T11:56:39.222148+0000","last_clean":"2024-09-18T11:56:39.222148+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:39.222148+0000","last_undegraded":"2024-09-18T11:56:39.222148+0000","last_fullsized":"2024-09-18T11:56:39.222148+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12712,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.220036+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:39.220036+0000","last_peered":"2024-09-18T11:56:39.220036+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:39.220036+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":15438,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:45.259532+0000","last_change":"2024-09-18T11:56:38.980663+0000","last_active":"2024-09-18T11:56:45.259532+0000","last_peered":"2024-09-18T11:56:45.259532+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:44:47.359963+0000","last_became_peered":"2024-09-18T11:44:47.359963+0000","last_unstale":"2024-09-18T11:56:45.259532+0000","last_undegraded":"2024-09-18T11:44:47.169839+0000","last_fullsized":"2024-09-18T11:44:47.118676+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-894}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13129,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222435+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:39.222435+0000","last_peered":"2024-09-18T11:56:39.222435+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:39.222435+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13070,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.219484+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:39.219484+0000","last_peered":"2024-09-18T11:56:39.219484+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:39.219484+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13219,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:39.222196+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:39.222196+0000","last_peered":"2024-09-18T11:56:39.222196+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:39.222196+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17196,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:39.219637+0000","last_peered":"2024-09-18T11:56:39.219637+0000","last_clean":"2024-09-18T11:56:39.219637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:39.219637+0000","last_undegraded":"2024-09-18T11:56:39.219637+0000","last_fullsized":"2024-09-18T11:56:39.219637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12812,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.305343+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:40.305343+0000","last_peered":"2024-09-18T11:56:40.305343+0000","last_clean":"2024-09-18T11:56:40.305343+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:40.305343+0000","last_undegraded":"2024-09-18T11:56:40.305343+0000","last_fullsized":"2024-09-18T11:56:40.305343+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13840,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923888,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447952,"statfs":{"total":96636764160,"available":95690702848,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.185},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.247}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.222},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.28100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919216,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452624,"statfs":{"total":96636764160,"available":95695486976,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.30499999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.188},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.215}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.252},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.23100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926228,"kb_used_data":716,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445612,"statfs":{"total":96636764160,"available":95688306688,"internally_reserved":0,"allocated":733184,"data_stored":523854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.20399999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.219},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26500000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.27800000000000002},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.255}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915504,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456336,"statfs":{"total":96636764160,"available":95699288064,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.16700000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.246},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:52.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:53.187 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:53.187 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:53.200 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-18T11:56:52.112784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15086,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684872,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802488,"statfs":{"total":386547056640,"available":382773747712,"internally_reserved":0,"allocated":1859584,"data_stored":1004983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001324"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13359,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253553+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:49.253553+0000","last_peered":"2024-09-18T11:56:49.253553+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:49.253553+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17352,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219620+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:39.219620+0000","last_peered":"2024-09-18T11:56:39.219620+0000","last_clean":"2024-09-18T11:56:39.219620+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:39.219620+0000","last_undegraded":"2024-09-18T11:56:39.219620+0000","last_fullsized":"2024-09-18T11:56:39.219620+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17027,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.228655+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:40.228655+0000","last_peered":"2024-09-18T11:56:40.228655+0000","last_clean":"2024-09-18T11:56:40.228655+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:40.228655+0000","last_undegraded":"2024-09-18T11:56:40.228655+0000","last_fullsized":"2024-09-18T11:56:40.228655+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13026,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.260300+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:49.260300+0000","last_peered":"2024-09-18T11:56:49.260300+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:49.260300+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13022,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253813+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:49.253813+0000","last_peered":"2024-09-18T11:56:49.253813+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:49.253813+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12913,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253671+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:49.253671+0000","last_peered":"2024-09-18T11:56:49.253671+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:49.253671+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13828,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.904627+0000","last_change":"2024-09-18T11:56:49.045365+0000","last_active":"2024-09-18T11:56:50.904627+0000","last_peered":"2024-09-18T11:56:50.904627+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:50.904627+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1767}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:39.219653+0000","last_peered":"2024-09-18T11:56:39.219653+0000","last_clean":"2024-09-18T11:56:39.219653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:39.219653+0000","last_undegraded":"2024-09-18T11:56:39.219653+0000","last_fullsized":"2024-09-18T11:56:39.219653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17533,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.253379+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:49.253379+0000","last_peered":"2024-09-18T11:56:49.253379+0000","last_clean":"2024-09-18T11:56:49.253379+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:49.253379+0000","last_undegraded":"2024-09-18T11:56:49.253379+0000","last_fullsized":"2024-09-18T11:56:49.253379+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12713,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:40.228916+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:40.228916+0000","last_peered":"2024-09-18T11:56:40.228916+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:40.228916+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17243,"reported_epoch":45,"state":"active","last_fresh":"2024-09-18T11:56:50.261849+0000","last_change":"2024-09-18T11:56:50.261849+0000","last_active":"2024-09-18T11:56:50.261849+0000","last_peered":"2024-09-18T11:56:50.261849+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:56:50.261849+0000","last_undegraded":"2024-09-18T11:56:50.261849+0000","last_fullsized":"2024-09-18T11:56:50.261849+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":["3(0)","0(1)","1(2)","2(3)"],"object_location_counts":[{"shards":"0(1),1(2),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13131,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253767+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:49.253767+0000","last_peered":"2024-09-18T11:56:49.253767+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:49.253767+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13071,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:40.228384+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:40.228384+0000","last_peered":"2024-09-18T11:56:40.228384+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:40.228384+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13221,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.254786+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:49.254786+0000","last_peered":"2024-09-18T11:56:49.254786+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:49.254786+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17196,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:39.219637+0000","last_peered":"2024-09-18T11:56:39.219637+0000","last_clean":"2024-09-18T11:56:39.219637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:39.219637+0000","last_undegraded":"2024-09-18T11:56:39.219637+0000","last_fullsized":"2024-09-18T11:56:39.219637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12812,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.305343+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:40.305343+0000","last_peered":"2024-09-18T11:56:40.305343+0000","last_clean":"2024-09-18T11:56:40.305343+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:40.305343+0000","last_undegraded":"2024-09-18T11:56:40.305343+0000","last_fullsized":"2024-09-18T11:56:40.305343+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15086,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923896,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447944,"statfs":{"total":96636764160,"available":95690694656,"internally_reserved":0,"allocated":372736,"data_stored":159091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.60299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.443}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.32400000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.28399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919224,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452616,"statfs":{"total":96636764160,"available":95695478784,"internally_reserved":0,"allocated":372736,"data_stored":159091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.217},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.29899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.23999999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.28000000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926228,"kb_used_data":716,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445612,"statfs":{"total":96636764160,"available":95688306688,"internally_reserved":0,"allocated":733184,"data_stored":523854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.20399999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.219},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26500000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.27800000000000002},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.255}]}]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915524,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456316,"statfs":{"total":96636764160,"available":95699267584,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.189}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.22800000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.27900000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:53.202 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:53.480 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:53.480 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:53.492 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-18T11:56:52.112784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15086,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684872,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802488,"statfs":{"total":386547056640,"available":382773747712,"internally_reserved":0,"allocated":1859584,"data_stored":1004983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001324"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13359,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253553+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:49.253553+0000","last_peered":"2024-09-18T11:56:49.253553+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:49.253553+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17352,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219620+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:39.219620+0000","last_peered":"2024-09-18T11:56:39.219620+0000","last_clean":"2024-09-18T11:56:39.219620+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:39.219620+0000","last_undegraded":"2024-09-18T11:56:39.219620+0000","last_fullsized":"2024-09-18T11:56:39.219620+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17027,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.228655+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:40.228655+0000","last_peered":"2024-09-18T11:56:40.228655+0000","last_clean":"2024-09-18T11:56:40.228655+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:40.228655+0000","last_undegraded":"2024-09-18T11:56:40.228655+0000","last_fullsized":"2024-09-18T11:56:40.228655+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13026,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.260300+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:49.260300+0000","last_peered":"2024-09-18T11:56:49.260300+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:49.260300+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13022,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253813+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:49.253813+0000","last_peered":"2024-09-18T11:56:49.253813+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:49.253813+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12913,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253671+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:49.253671+0000","last_peered":"2024-09-18T11:56:49.253671+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:49.253671+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13828,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.904627+0000","last_change":"2024-09-18T11:56:49.045365+0000","last_active":"2024-09-18T11:56:50.904627+0000","last_peered":"2024-09-18T11:56:50.904627+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:50.904627+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1767}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:39.219653+0000","last_peered":"2024-09-18T11:56:39.219653+0000","last_clean":"2024-09-18T11:56:39.219653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:39.219653+0000","last_undegraded":"2024-09-18T11:56:39.219653+0000","last_fullsized":"2024-09-18T11:56:39.219653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17533,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.253379+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:49.253379+0000","last_peered":"2024-09-18T11:56:49.253379+0000","last_clean":"2024-09-18T11:56:49.253379+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:49.253379+0000","last_undegraded":"2024-09-18T11:56:49.253379+0000","last_fullsized":"2024-09-18T11:56:49.253379+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12713,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:40.228916+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:40.228916+0000","last_peered":"2024-09-18T11:56:40.228916+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:40.228916+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17243,"reported_epoch":45,"state":"active","last_fresh":"2024-09-18T11:56:50.261849+0000","last_change":"2024-09-18T11:56:50.261849+0000","last_active":"2024-09-18T11:56:50.261849+0000","last_peered":"2024-09-18T11:56:50.261849+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:56:50.261849+0000","last_undegraded":"2024-09-18T11:56:50.261849+0000","last_fullsized":"2024-09-18T11:56:50.261849+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":["3(0)","0(1)","1(2)","2(3)"],"object_location_counts":[{"shards":"0(1),1(2),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13131,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253767+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:49.253767+0000","last_peered":"2024-09-18T11:56:49.253767+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:49.253767+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13071,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:40.228384+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:40.228384+0000","last_peered":"2024-09-18T11:56:40.228384+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:40.228384+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13221,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.254786+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:49.254786+0000","last_peered":"2024-09-18T11:56:49.254786+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:49.254786+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17196,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T11:56:39.219637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:39.219637+0000","last_peered":"2024-09-18T11:56:39.219637+0000","last_clean":"2024-09-18T11:56:39.219637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:39.219637+0000","last_undegraded":"2024-09-18T11:56:39.219637+0000","last_fullsized":"2024-09-18T11:56:39.219637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12812,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.305343+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:40.305343+0000","last_peered":"2024-09-18T11:56:40.305343+0000","last_clean":"2024-09-18T11:56:40.305343+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:40.305343+0000","last_undegraded":"2024-09-18T11:56:40.305343+0000","last_fullsized":"2024-09-18T11:56:40.305343+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15086,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923896,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447944,"statfs":{"total":96636764160,"available":95690694656,"internally_reserved":0,"allocated":372736,"data_stored":159091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.60299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.443}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.32400000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.28399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919224,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452616,"statfs":{"total":96636764160,"available":95695478784,"internally_reserved":0,"allocated":372736,"data_stored":159091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.217},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.29899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.23999999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.28000000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926228,"kb_used_data":716,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445612,"statfs":{"total":96636764160,"available":95688306688,"internally_reserved":0,"allocated":733184,"data_stored":523854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.20399999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.219},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26500000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.27800000000000002},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.255}]}]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915524,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456316,"statfs":{"total":96636764160,"available":95699267584,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.189}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.22800000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.27900000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:53.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:53.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13359, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.253553+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:49.253553+0000', 'last_peered': '2024-09-18T11:56:49.253553+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:49.253553+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:53.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:53.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13026, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.260300+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:49.260300+0000', 'last_peered': '2024-09-18T11:56:49.260300+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:49.260300+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:53.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13022, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.253813+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:49.253813+0000', 'last_peered': '2024-09-18T11:56:49.253813+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:49.253813+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12913, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.253671+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:49.253671+0000', 'last_peered': '2024-09-18T11:56:49.253671+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:49.253671+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'13014", 'reported_seq': 13828, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:50.904627+0000', 'last_change': '2024-09-18T11:56:49.045365+0000', 'last_active': '2024-09-18T11:56:50.904627+0000', 'last_peered': '2024-09-18T11:56:50.904627+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:50.904627+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1769, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13012, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 369, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1767}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12713, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:40.228916+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:40.228916+0000', 'last_peered': '2024-09-18T11:56:40.228916+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:40.228916+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13030", 'reported_seq': 17243, 'reported_epoch': 45, 'state': 'active', 'last_fresh': '2024-09-18T11:56:50.261849+0000', 'last_change': '2024-09-18T11:56:50.261849+0000', 'last_active': '2024-09-18T11:56:50.261849+0000', 'last_peered': '2024-09-18T11:56:50.261849+0000', 'last_clean': '2024-09-18T11:43:44.730784+0000', 'last_became_active': '2024-09-18T11:56:50.261849+0000', 'last_became_peered': '2024-09-18T11:56:50.261849+0000', 'last_unstale': '2024-09-18T11:56:50.261849+0000', 'last_undegraded': '2024-09-18T11:56:50.261849+0000', 'last_fullsized': '2024-09-18T11:56:50.261849+0000', 'mapping_epoch': 44, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20337, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13027, 'num_write_kb': 52116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2069, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 1, 2], 'avail_no_missing': ['3(0)', '0(1)', '1(2)', '2(3)'], 'object_location_counts': [{'shards': '0(1),1(2),2(3),3(0)', 'objects': 3}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:53.494 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13131, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.253767+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:49.253767+0000', 'last_peered': '2024-09-18T11:56:49.253767+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:49.253767+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:53.495 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:53.495 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 13071, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:40.228384+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:40.228384+0000', 'last_peered': '2024-09-18T11:56:40.228384+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:40.228384+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:53.495 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:53.495 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13221, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.254786+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:49.254786+0000', 'last_peered': '2024-09-18T11:56:49.254786+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:49.254786+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:53.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:53.832 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:53.833 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18370/112 objects degraded (16401.786%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":811,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":6},{"state_name":"active","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3773308928,"bytes_avail":382773747712,"bytes_total":386547056640,"degraded_objects":18370,"degraded_total":112,"degraded_ratio":164.01785714285714,"recovering_objects_per_sec":269,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:53.845 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:53.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:54.116 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:54.116 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:54.128 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-18T11:56:54.113223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15086,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693200,"kb_used_data":1824,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794160,"statfs":{"total":386547056640,"available":382765219840,"internally_reserved":0,"allocated":1867776,"data_stored":1004524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001414"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13359,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253553+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:49.253553+0000","last_peered":"2024-09-18T11:56:49.253553+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:49.253553+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17354,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250909+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:49.250909+0000","last_peered":"2024-09-18T11:56:49.250909+0000","last_clean":"2024-09-18T11:56:49.250909+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:49.250909+0000","last_undegraded":"2024-09-18T11:56:49.250909+0000","last_fullsized":"2024-09-18T11:56:49.250909+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17027,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T11:56:40.228655+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:40.228655+0000","last_peered":"2024-09-18T11:56:40.228655+0000","last_clean":"2024-09-18T11:56:40.228655+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:40.228655+0000","last_undegraded":"2024-09-18T11:56:40.228655+0000","last_fullsized":"2024-09-18T11:56:40.228655+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13026,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.260300+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:49.260300+0000","last_peered":"2024-09-18T11:56:49.260300+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:49.260300+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13022,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253813+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:49.253813+0000","last_peered":"2024-09-18T11:56:49.253813+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:49.253813+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12913,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253671+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:49.253671+0000","last_peered":"2024-09-18T11:56:49.253671+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:49.253671+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":13828,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.904627+0000","last_change":"2024-09-18T11:56:49.045365+0000","last_active":"2024-09-18T11:56:50.904627+0000","last_peered":"2024-09-18T11:56:50.904627+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:50.904627+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1767}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250714+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:49.250714+0000","last_peered":"2024-09-18T11:56:49.250714+0000","last_clean":"2024-09-18T11:56:49.250714+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:49.250714+0000","last_undegraded":"2024-09-18T11:56:49.250714+0000","last_fullsized":"2024-09-18T11:56:49.250714+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17533,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.253379+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:49.253379+0000","last_peered":"2024-09-18T11:56:49.253379+0000","last_clean":"2024-09-18T11:56:49.253379+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:49.253379+0000","last_undegraded":"2024-09-18T11:56:49.253379+0000","last_fullsized":"2024-09-18T11:56:49.253379+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12713,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:40.228916+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:40.228916+0000","last_peered":"2024-09-18T11:56:40.228916+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:40.228916+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17243,"reported_epoch":45,"state":"active","last_fresh":"2024-09-18T11:56:50.261849+0000","last_change":"2024-09-18T11:56:50.261849+0000","last_active":"2024-09-18T11:56:50.261849+0000","last_peered":"2024-09-18T11:56:50.261849+0000","last_clean":"2024-09-18T11:43:44.730784+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:56:50.261849+0000","last_undegraded":"2024-09-18T11:56:50.261849+0000","last_fullsized":"2024-09-18T11:56:50.261849+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":["3(0)","0(1)","1(2)","2(3)"],"object_location_counts":[{"shards":"0(1),1(2),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13131,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253767+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:49.253767+0000","last_peered":"2024-09-18T11:56:49.253767+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:49.253767+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13071,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:40.228384+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:40.228384+0000","last_peered":"2024-09-18T11:56:40.228384+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:40.228384+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13221,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.254786+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:49.254786+0000","last_peered":"2024-09-18T11:56:49.254786+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:49.254786+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17198,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.255016+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:49.255016+0000","last_peered":"2024-09-18T11:56:49.255016+0000","last_clean":"2024-09-18T11:56:49.255016+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:49.255016+0000","last_undegraded":"2024-09-18T11:56:49.255016+0000","last_fullsized":"2024-09-18T11:56:49.255016+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12814,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250802+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:49.250802+0000","last_peered":"2024-09-18T11:56:49.250802+0000","last_clean":"2024-09-18T11:56:49.250802+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:49.250802+0000","last_undegraded":"2024-09-18T11:56:49.250802+0000","last_fullsized":"2024-09-18T11:56:49.250802+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15086,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":376832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923896,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447944,"statfs":{"total":96636764160,"available":95690694656,"internally_reserved":0,"allocated":372736,"data_stored":159091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.60299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.443}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.32400000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.28399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919224,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452616,"statfs":{"total":96636764160,"available":95695478784,"internally_reserved":0,"allocated":372736,"data_stored":159091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.217},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.29899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.23999999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.28000000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934556,"kb_used_data":724,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437284,"statfs":{"total":96636764160,"available":95679778816,"internally_reserved":0,"allocated":741376,"data_stored":523395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.19800000000000001},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.223},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.24399999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.311},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915524,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456316,"statfs":{"total":96636764160,"available":95699267584,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.189}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.22800000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.27900000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:57.130 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:57.450 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:57.451 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:57.463 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2024-09-18T11:56:56.113593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15999,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693224,"kb_used_data":1840,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794136,"statfs":{"total":386547056640,"available":382765195264,"internally_reserved":0,"allocated":1884160,"data_stored":1012236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001451"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13359,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253553+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:49.253553+0000","last_peered":"2024-09-18T11:56:49.253553+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:49.253553+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17354,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250909+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:49.250909+0000","last_peered":"2024-09-18T11:56:49.250909+0000","last_clean":"2024-09-18T11:56:49.250909+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:49.250909+0000","last_undegraded":"2024-09-18T11:56:49.250909+0000","last_fullsized":"2024-09-18T11:56:49.250909+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17028,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.251592+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:49.251592+0000","last_peered":"2024-09-18T11:56:49.251592+0000","last_clean":"2024-09-18T11:56:49.251592+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:49.251592+0000","last_undegraded":"2024-09-18T11:56:49.251592+0000","last_fullsized":"2024-09-18T11:56:49.251592+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13026,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.260300+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:49.260300+0000","last_peered":"2024-09-18T11:56:49.260300+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:49.260300+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13022,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253813+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:49.253813+0000","last_peered":"2024-09-18T11:56:49.253813+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:49.253813+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12913,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253671+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:49.253671+0000","last_peered":"2024-09-18T11:56:49.253671+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:49.253671+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":15654,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:55.901779+0000","last_change":"2024-09-18T11:56:49.045365+0000","last_active":"2024-09-18T11:56:55.901779+0000","last_peered":"2024-09-18T11:56:55.901779+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:55.901779+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1282,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-854}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250714+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:49.250714+0000","last_peered":"2024-09-18T11:56:49.250714+0000","last_clean":"2024-09-18T11:56:49.250714+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:49.250714+0000","last_undegraded":"2024-09-18T11:56:49.250714+0000","last_fullsized":"2024-09-18T11:56:49.250714+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17533,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.253379+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:49.253379+0000","last_peered":"2024-09-18T11:56:49.253379+0000","last_clean":"2024-09-18T11:56:49.253379+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:49.253379+0000","last_undegraded":"2024-09-18T11:56:49.253379+0000","last_fullsized":"2024-09-18T11:56:49.253379+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12714,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.251606+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:49.251606+0000","last_peered":"2024-09-18T11:56:49.251606+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:49.251606+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17246,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.330277+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:56:50.330277+0000","last_peered":"2024-09-18T11:56:50.330277+0000","last_clean":"2024-09-18T11:56:50.330277+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:56:50.330277+0000","last_undegraded":"2024-09-18T11:56:50.330277+0000","last_fullsized":"2024-09-18T11:56:50.330277+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13131,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253767+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:49.253767+0000","last_peered":"2024-09-18T11:56:49.253767+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:49.253767+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13072,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.250981+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:49.250981+0000","last_peered":"2024-09-18T11:56:49.250981+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:49.250981+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13221,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.254786+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:49.254786+0000","last_peered":"2024-09-18T11:56:49.254786+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:49.254786+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17198,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.255016+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:49.255016+0000","last_peered":"2024-09-18T11:56:49.255016+0000","last_clean":"2024-09-18T11:56:49.255016+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:49.255016+0000","last_undegraded":"2024-09-18T11:56:49.255016+0000","last_fullsized":"2024-09-18T11:56:49.255016+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12814,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250802+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:49.250802+0000","last_peered":"2024-09-18T11:56:49.250802+0000","last_clean":"2024-09-18T11:56:49.250802+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:49.250802+0000","last_undegraded":"2024-09-18T11:56:49.250802+0000","last_fullsized":"2024-09-18T11:56:49.250802+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15999,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":376832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923908,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447932,"statfs":{"total":96636764160,"available":95690682368,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.252}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.20999999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.26600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919236,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452604,"statfs":{"total":96636764160,"available":95695466496,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.19},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.34899999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.246},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.22700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.26300000000000001},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.28199999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934556,"kb_used_data":724,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437284,"statfs":{"total":96636764160,"available":95679778816,"internally_reserved":0,"allocated":741376,"data_stored":523395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.19800000000000001},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.223},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.24399999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.311},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915524,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456316,"statfs":{"total":96636764160,"available":95699267584,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.157},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.30099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.315},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:57.464 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:57.744 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:57.745 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:57.757 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2024-09-18T11:56:56.113593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15999,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693224,"kb_used_data":1840,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794136,"statfs":{"total":386547056640,"available":382765195264,"internally_reserved":0,"allocated":1884160,"data_stored":1012236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001451"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13359,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253553+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:49.253553+0000","last_peered":"2024-09-18T11:56:49.253553+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:49.253553+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17354,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250909+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:49.250909+0000","last_peered":"2024-09-18T11:56:49.250909+0000","last_clean":"2024-09-18T11:56:49.250909+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:49.250909+0000","last_undegraded":"2024-09-18T11:56:49.250909+0000","last_fullsized":"2024-09-18T11:56:49.250909+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17028,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.251592+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:49.251592+0000","last_peered":"2024-09-18T11:56:49.251592+0000","last_clean":"2024-09-18T11:56:49.251592+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:49.251592+0000","last_undegraded":"2024-09-18T11:56:49.251592+0000","last_fullsized":"2024-09-18T11:56:49.251592+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13026,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.260300+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:49.260300+0000","last_peered":"2024-09-18T11:56:49.260300+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:49.260300+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13022,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253813+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:49.253813+0000","last_peered":"2024-09-18T11:56:49.253813+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:49.253813+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12913,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253671+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:49.253671+0000","last_peered":"2024-09-18T11:56:49.253671+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:49.253671+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":15654,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:55.901779+0000","last_change":"2024-09-18T11:56:49.045365+0000","last_active":"2024-09-18T11:56:55.901779+0000","last_peered":"2024-09-18T11:56:55.901779+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:55.901779+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1282,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-854}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250714+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:49.250714+0000","last_peered":"2024-09-18T11:56:49.250714+0000","last_clean":"2024-09-18T11:56:49.250714+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:49.250714+0000","last_undegraded":"2024-09-18T11:56:49.250714+0000","last_fullsized":"2024-09-18T11:56:49.250714+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17533,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.253379+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:49.253379+0000","last_peered":"2024-09-18T11:56:49.253379+0000","last_clean":"2024-09-18T11:56:49.253379+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:49.253379+0000","last_undegraded":"2024-09-18T11:56:49.253379+0000","last_fullsized":"2024-09-18T11:56:49.253379+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12714,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.251606+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:49.251606+0000","last_peered":"2024-09-18T11:56:49.251606+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:49.251606+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17246,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.330277+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:56:50.330277+0000","last_peered":"2024-09-18T11:56:50.330277+0000","last_clean":"2024-09-18T11:56:50.330277+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:56:50.330277+0000","last_undegraded":"2024-09-18T11:56:50.330277+0000","last_fullsized":"2024-09-18T11:56:50.330277+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13131,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253767+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:49.253767+0000","last_peered":"2024-09-18T11:56:49.253767+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:49.253767+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13072,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.250981+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:49.250981+0000","last_peered":"2024-09-18T11:56:49.250981+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:49.250981+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13221,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.254786+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:49.254786+0000","last_peered":"2024-09-18T11:56:49.254786+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:49.254786+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17198,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.255016+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:49.255016+0000","last_peered":"2024-09-18T11:56:49.255016+0000","last_clean":"2024-09-18T11:56:49.255016+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:49.255016+0000","last_undegraded":"2024-09-18T11:56:49.255016+0000","last_fullsized":"2024-09-18T11:56:49.255016+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12814,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250802+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:49.250802+0000","last_peered":"2024-09-18T11:56:49.250802+0000","last_clean":"2024-09-18T11:56:49.250802+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:49.250802+0000","last_undegraded":"2024-09-18T11:56:49.250802+0000","last_fullsized":"2024-09-18T11:56:49.250802+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15999,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":376832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923908,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447932,"statfs":{"total":96636764160,"available":95690682368,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.252}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.20999999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.26600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919236,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452604,"statfs":{"total":96636764160,"available":95695466496,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.19},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.34899999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.246},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.22700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.26300000000000001},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.28199999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934556,"kb_used_data":724,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437284,"statfs":{"total":96636764160,"available":95679778816,"internally_reserved":0,"allocated":741376,"data_stored":523395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.19800000000000001},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.223},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.24399999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.311},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915524,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456316,"statfs":{"total":96636764160,"available":95699267584,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.157},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.30099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.315},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:57.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:56:57.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13359, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.253553+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:49.253553+0000', 'last_peered': '2024-09-18T11:56:49.253553+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:49.253553+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:57.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:56:57.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13026, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.260300+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:49.260300+0000', 'last_peered': '2024-09-18T11:56:49.260300+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:49.260300+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:57.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:56:57.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13022, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.253813+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:49.253813+0000', 'last_peered': '2024-09-18T11:56:49.253813+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:49.253813+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:57.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:56:57.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12913, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.253671+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:49.253671+0000', 'last_peered': '2024-09-18T11:56:49.253671+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:49.253671+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:57.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:56:57.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'13014", 'reported_seq': 15654, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:55.901779+0000', 'last_change': '2024-09-18T11:56:49.045365+0000', 'last_active': '2024-09-18T11:56:55.901779+0000', 'last_peered': '2024-09-18T11:56:55.901779+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:56:55.901779+0000', 'last_undegraded': '2024-09-18T11:44:47.192131+0000', 'last_fullsized': '2024-09-18T11:44:47.122446+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 856, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13012, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1282, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -854}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:57.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:56:57.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12714, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.251606+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:49.251606+0000', 'last_peered': '2024-09-18T11:56:49.251606+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:49.251606+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:57.760 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:56:57.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13131, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.253767+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:49.253767+0000', 'last_peered': '2024-09-18T11:56:49.253767+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:49.253767+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:57.760 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:56:57.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 13072, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.250981+0000', 'last_change': '2024-09-18T11:44:47.454537+0000', 'last_active': '2024-09-18T11:56:49.250981+0000', 'last_peered': '2024-09-18T11:56:49.250981+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:49.250981+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:57.760 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:56:57.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13221, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.254786+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:49.254786+0000', 'last_peered': '2024-09-18T11:56:49.254786+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:49.254786+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:56:57.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:58.105 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:58.105 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17457/112 objects degraded (15586.607%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":816,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3781861376,"bytes_avail":382765195264,"bytes_total":386547056640,"degraded_objects":17457,"degraded_total":112,"degraded_ratio":155.86607142857142,"recovering_objects_per_sec":269,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2159,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:58.119 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:58.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:58.424 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:56:58.424 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:56:58.436 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-18T11:56:58.114057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15999,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693224,"kb_used_data":1840,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794136,"statfs":{"total":386547056640,"available":382765195264,"internally_reserved":0,"allocated":1884160,"data_stored":1012236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.864890"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13359,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253553+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:49.253553+0000","last_peered":"2024-09-18T11:56:49.253553+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:49.253553+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17354,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250909+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:49.250909+0000","last_peered":"2024-09-18T11:56:49.250909+0000","last_clean":"2024-09-18T11:56:49.250909+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:49.250909+0000","last_undegraded":"2024-09-18T11:56:49.250909+0000","last_fullsized":"2024-09-18T11:56:49.250909+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17028,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.251592+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:49.251592+0000","last_peered":"2024-09-18T11:56:49.251592+0000","last_clean":"2024-09-18T11:56:49.251592+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:49.251592+0000","last_undegraded":"2024-09-18T11:56:49.251592+0000","last_fullsized":"2024-09-18T11:56:49.251592+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13026,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.260300+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:49.260300+0000","last_peered":"2024-09-18T11:56:49.260300+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:49.260300+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13022,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253813+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:49.253813+0000","last_peered":"2024-09-18T11:56:49.253813+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:49.253813+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12913,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253671+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:49.253671+0000","last_peered":"2024-09-18T11:56:49.253671+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:49.253671+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":15654,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:55.901779+0000","last_change":"2024-09-18T11:56:49.045365+0000","last_active":"2024-09-18T11:56:55.901779+0000","last_peered":"2024-09-18T11:56:55.901779+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:56:55.901779+0000","last_undegraded":"2024-09-18T11:44:47.192131+0000","last_fullsized":"2024-09-18T11:44:47.122446+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1282,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-854}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250714+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:49.250714+0000","last_peered":"2024-09-18T11:56:49.250714+0000","last_clean":"2024-09-18T11:56:49.250714+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:49.250714+0000","last_undegraded":"2024-09-18T11:56:49.250714+0000","last_fullsized":"2024-09-18T11:56:49.250714+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17533,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.253379+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:49.253379+0000","last_peered":"2024-09-18T11:56:49.253379+0000","last_clean":"2024-09-18T11:56:49.253379+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:49.253379+0000","last_undegraded":"2024-09-18T11:56:49.253379+0000","last_fullsized":"2024-09-18T11:56:49.253379+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12714,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.251606+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:49.251606+0000","last_peered":"2024-09-18T11:56:49.251606+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:49.251606+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17246,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.330277+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:56:50.330277+0000","last_peered":"2024-09-18T11:56:50.330277+0000","last_clean":"2024-09-18T11:56:50.330277+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:56:50.330277+0000","last_undegraded":"2024-09-18T11:56:50.330277+0000","last_fullsized":"2024-09-18T11:56:50.330277+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13131,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.253767+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:49.253767+0000","last_peered":"2024-09-18T11:56:49.253767+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:49.253767+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13072,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.250981+0000","last_change":"2024-09-18T11:44:47.454537+0000","last_active":"2024-09-18T11:56:49.250981+0000","last_peered":"2024-09-18T11:56:49.250981+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:49.250981+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13221,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.254786+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:49.254786+0000","last_peered":"2024-09-18T11:56:49.254786+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:49.254786+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17198,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.255016+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:49.255016+0000","last_peered":"2024-09-18T11:56:49.255016+0000","last_clean":"2024-09-18T11:56:49.255016+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:49.255016+0000","last_undegraded":"2024-09-18T11:56:49.255016+0000","last_fullsized":"2024-09-18T11:56:49.255016+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12814,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250802+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:49.250802+0000","last_peered":"2024-09-18T11:56:49.250802+0000","last_clean":"2024-09-18T11:56:49.250802+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:49.250802+0000","last_undegraded":"2024-09-18T11:56:49.250802+0000","last_fullsized":"2024-09-18T11:56:49.250802+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15999,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":376832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923908,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447932,"statfs":{"total":96636764160,"available":95690682368,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.252}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.20999999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.26600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919236,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452604,"statfs":{"total":96636764160,"available":95695466496,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.19},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.34899999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.246},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.22700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.26300000000000001},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.28199999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934556,"kb_used_data":724,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437284,"statfs":{"total":96636764160,"available":95679778816,"internally_reserved":0,"allocated":741376,"data_stored":523395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.19800000000000001},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.223},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.24399999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.311},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915524,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456316,"statfs":{"total":96636764160,"available":95699267584,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.157},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.30099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.315},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:01.439 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:01.777 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:01.777 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:01.789 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-18T11:57:01.288572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16877,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684908,"kb_used_data":1848,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802452,"statfs":{"total":386547056640,"available":382773710848,"internally_reserved":0,"allocated":1892352,"data_stored":1016076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1791,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.175787"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13360,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.256596+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:50.256596+0000","last_peered":"2024-09-18T11:56:50.256596+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:50.256596+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17354,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250909+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:49.250909+0000","last_peered":"2024-09-18T11:56:49.250909+0000","last_clean":"2024-09-18T11:56:49.250909+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:49.250909+0000","last_undegraded":"2024-09-18T11:56:49.250909+0000","last_fullsized":"2024-09-18T11:56:49.250909+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17028,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.251592+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:49.251592+0000","last_peered":"2024-09-18T11:56:49.251592+0000","last_clean":"2024-09-18T11:56:49.251592+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:49.251592+0000","last_undegraded":"2024-09-18T11:56:49.251592+0000","last_fullsized":"2024-09-18T11:56:49.251592+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13027,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.256076+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:50.256076+0000","last_peered":"2024-09-18T11:56:50.256076+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:50.256076+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13023,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255736+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:50.255736+0000","last_peered":"2024-09-18T11:56:50.255736+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:50.255736+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12914,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255889+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:50.255889+0000","last_peered":"2024-09-18T11:56:50.255889+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:50.255889+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17370,"reported_epoch":46,"state":"peering","last_fresh":"2024-09-18T11:57:00.287246+0000","last_change":"2024-09-18T11:57:00.289914+0000","last_active":"2024-09-18T11:57:00.287210+0000","last_peered":"2024-09-18T11:56:59.699934+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:57:00.287246+0000","last_undegraded":"2024-09-18T11:57:00.287246+0000","last_fullsized":"2024-09-18T11:57:00.287246+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250714+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:49.250714+0000","last_peered":"2024-09-18T11:56:49.250714+0000","last_clean":"2024-09-18T11:56:49.250714+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:49.250714+0000","last_undegraded":"2024-09-18T11:56:49.250714+0000","last_fullsized":"2024-09-18T11:56:49.250714+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17534,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.255321+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:50.255321+0000","last_peered":"2024-09-18T11:56:50.255321+0000","last_clean":"2024-09-18T11:56:50.255321+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:50.255321+0000","last_undegraded":"2024-09-18T11:56:50.255321+0000","last_fullsized":"2024-09-18T11:56:50.255321+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12714,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.251606+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:49.251606+0000","last_peered":"2024-09-18T11:56:49.251606+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:49.251606+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17247,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.331543+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:56:50.331543+0000","last_peered":"2024-09-18T11:56:50.331543+0000","last_clean":"2024-09-18T11:56:50.331543+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:56:50.331543+0000","last_undegraded":"2024-09-18T11:56:50.331543+0000","last_fullsized":"2024-09-18T11:56:50.331543+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13132,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.257448+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:50.257448+0000","last_peered":"2024-09-18T11:56:50.257448+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:50.257448+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13117,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:59.844880+0000","last_change":"2024-09-18T11:56:59.700508+0000","last_active":"2024-09-18T11:56:59.844880+0000","last_peered":"2024-09-18T11:56:59.844880+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:59.844880+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13222,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255903+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:50.255903+0000","last_peered":"2024-09-18T11:56:50.255903+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:50.255903+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17198,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.255016+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:49.255016+0000","last_peered":"2024-09-18T11:56:49.255016+0000","last_clean":"2024-09-18T11:56:49.255016+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:49.255016+0000","last_undegraded":"2024-09-18T11:56:49.255016+0000","last_fullsized":"2024-09-18T11:56:49.255016+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12814,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250802+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:49.250802+0000","last_peered":"2024-09-18T11:56:49.250802+0000","last_clean":"2024-09-18T11:56:49.250802+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:49.250802+0000","last_undegraded":"2024-09-18T11:56:49.250802+0000","last_fullsized":"2024-09-18T11:56:49.250802+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16877,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":376832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923908,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447932,"statfs":{"total":96636764160,"available":95690682368,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.27200000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.193}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.254},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.23000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919236,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452604,"statfs":{"total":96636764160,"available":95695466496,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.17999999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.245}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.21299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.27800000000000002},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926236,"kb_used_data":724,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445604,"statfs":{"total":96636764160,"available":95688298496,"internally_reserved":0,"allocated":741376,"data_stored":523395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.28199999999999997},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.183}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.23799999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915528,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456312,"statfs":{"total":96636764160,"available":95699263488,"internally_reserved":0,"allocated":389120,"data_stored":166787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.184}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.223}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.248}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:01.790 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:02.067 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:02.068 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:02.080 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-18T11:57:01.288572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16877,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684908,"kb_used_data":1848,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802452,"statfs":{"total":386547056640,"available":382773710848,"internally_reserved":0,"allocated":1892352,"data_stored":1016076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1791,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.175787"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13360,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.256596+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:50.256596+0000","last_peered":"2024-09-18T11:56:50.256596+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:50.256596+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17354,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250909+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:49.250909+0000","last_peered":"2024-09-18T11:56:49.250909+0000","last_clean":"2024-09-18T11:56:49.250909+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:49.250909+0000","last_undegraded":"2024-09-18T11:56:49.250909+0000","last_fullsized":"2024-09-18T11:56:49.250909+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17028,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.251592+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:49.251592+0000","last_peered":"2024-09-18T11:56:49.251592+0000","last_clean":"2024-09-18T11:56:49.251592+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:49.251592+0000","last_undegraded":"2024-09-18T11:56:49.251592+0000","last_fullsized":"2024-09-18T11:56:49.251592+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13027,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.256076+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:50.256076+0000","last_peered":"2024-09-18T11:56:50.256076+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:50.256076+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13023,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255736+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:50.255736+0000","last_peered":"2024-09-18T11:56:50.255736+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:50.255736+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12914,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255889+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:50.255889+0000","last_peered":"2024-09-18T11:56:50.255889+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:50.255889+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17370,"reported_epoch":46,"state":"peering","last_fresh":"2024-09-18T11:57:00.287246+0000","last_change":"2024-09-18T11:57:00.289914+0000","last_active":"2024-09-18T11:57:00.287210+0000","last_peered":"2024-09-18T11:56:59.699934+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:57:00.287246+0000","last_undegraded":"2024-09-18T11:57:00.287246+0000","last_fullsized":"2024-09-18T11:57:00.287246+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250714+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:49.250714+0000","last_peered":"2024-09-18T11:56:49.250714+0000","last_clean":"2024-09-18T11:56:49.250714+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:49.250714+0000","last_undegraded":"2024-09-18T11:56:49.250714+0000","last_fullsized":"2024-09-18T11:56:49.250714+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17534,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.255321+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:50.255321+0000","last_peered":"2024-09-18T11:56:50.255321+0000","last_clean":"2024-09-18T11:56:50.255321+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:50.255321+0000","last_undegraded":"2024-09-18T11:56:50.255321+0000","last_fullsized":"2024-09-18T11:56:50.255321+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12714,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.251606+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:49.251606+0000","last_peered":"2024-09-18T11:56:49.251606+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:49.251606+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17247,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.331543+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:56:50.331543+0000","last_peered":"2024-09-18T11:56:50.331543+0000","last_clean":"2024-09-18T11:56:50.331543+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:56:50.331543+0000","last_undegraded":"2024-09-18T11:56:50.331543+0000","last_fullsized":"2024-09-18T11:56:50.331543+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13132,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.257448+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:50.257448+0000","last_peered":"2024-09-18T11:56:50.257448+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:50.257448+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13117,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:59.844880+0000","last_change":"2024-09-18T11:56:59.700508+0000","last_active":"2024-09-18T11:56:59.844880+0000","last_peered":"2024-09-18T11:56:59.844880+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:59.844880+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13222,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255903+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:50.255903+0000","last_peered":"2024-09-18T11:56:50.255903+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:50.255903+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17198,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.255016+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:49.255016+0000","last_peered":"2024-09-18T11:56:49.255016+0000","last_clean":"2024-09-18T11:56:49.255016+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:49.255016+0000","last_undegraded":"2024-09-18T11:56:49.255016+0000","last_fullsized":"2024-09-18T11:56:49.255016+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12814,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250802+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:49.250802+0000","last_peered":"2024-09-18T11:56:49.250802+0000","last_clean":"2024-09-18T11:56:49.250802+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:49.250802+0000","last_undegraded":"2024-09-18T11:56:49.250802+0000","last_fullsized":"2024-09-18T11:56:49.250802+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16877,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":376832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923908,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447932,"statfs":{"total":96636764160,"available":95690682368,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.27200000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.193}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.254},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.23000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919236,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452604,"statfs":{"total":96636764160,"available":95695466496,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.17999999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.245}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.21299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.27800000000000002},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926236,"kb_used_data":724,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445604,"statfs":{"total":96636764160,"available":95688298496,"internally_reserved":0,"allocated":741376,"data_stored":523395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.28199999999999997},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.183}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.23799999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915528,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456312,"statfs":{"total":96636764160,"available":95699263488,"internally_reserved":0,"allocated":389120,"data_stored":166787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.184}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.223}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.248}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:02.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:02.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13360, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:50.256596+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:56:50.256596+0000', 'last_peered': '2024-09-18T11:56:50.256596+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:56:50.256596+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:02.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:57:02.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13027, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:50.256076+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:56:50.256076+0000', 'last_peered': '2024-09-18T11:56:50.256076+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:56:50.256076+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:02.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:02.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13023, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:50.255736+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:56:50.255736+0000', 'last_peered': '2024-09-18T11:56:50.255736+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:56:50.255736+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:02.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:57:02.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12914, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:50.255889+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:56:50.255889+0000', 'last_peered': '2024-09-18T11:56:50.255889+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:56:50.255889+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:02.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-18T11:57:02.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'13014", 'reported_seq': 17370, 'reported_epoch': 46, 'state': 'peering', 'last_fresh': '2024-09-18T11:57:00.287246+0000', 'last_change': '2024-09-18T11:57:00.289914+0000', 'last_active': '2024-09-18T11:57:00.287210+0000', 'last_peered': '2024-09-18T11:56:59.699934+0000', 'last_clean': '2024-09-18T11:43:44.911411+0000', 'last_became_active': '2024-09-18T11:44:47.481088+0000', 'last_became_peered': '2024-09-18T11:44:47.481088+0000', 'last_unstale': '2024-09-18T11:57:00.287246+0000', 'last_undegraded': '2024-09-18T11:57:00.287246+0000', 'last_fullsized': '2024-09-18T11:57:00.287246+0000', 'mapping_epoch': 46, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13012, 'num_write_kb': 52052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2138, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:02.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:02.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12714, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:49.251606+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:56:49.251606+0000', 'last_peered': '2024-09-18T11:56:49.251606+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:56:49.251606+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:02.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:02.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13132, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:50.257448+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:56:50.257448+0000', 'last_peered': '2024-09-18T11:56:50.257448+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:56:50.257448+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:02.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:57:02.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 13117, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:59.844880+0000', 'last_change': '2024-09-18T11:56:59.700508+0000', 'last_active': '2024-09-18T11:56:59.844880+0000', 'last_peered': '2024-09-18T11:56:59.844880+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:56:59.844880+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:02.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:02.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13222, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:50.255903+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:56:50.255903+0000', 'last_peered': '2024-09-18T11:56:50.255903+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:56:50.255903+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:02.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:02.434 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:02.434 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16579/112 objects degraded (14802.679%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":820,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3773345792,"bytes_avail":382773710848,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":16579,"degraded_total":112,"degraded_ratio":148.02678571428572,"recovering_objects_per_sec":223,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1791,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:02.447 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:02.448 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:02.743 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:02.744 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:02.757 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-18T11:57:02.114718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16877,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684908,"kb_used_data":1848,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802452,"statfs":{"total":386547056640,"available":382773710848,"internally_reserved":0,"allocated":1892352,"data_stored":1016076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1791,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001495"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13360,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.256596+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:50.256596+0000","last_peered":"2024-09-18T11:56:50.256596+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:50.256596+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17354,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250909+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:56:49.250909+0000","last_peered":"2024-09-18T11:56:49.250909+0000","last_clean":"2024-09-18T11:56:49.250909+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:56:49.250909+0000","last_undegraded":"2024-09-18T11:56:49.250909+0000","last_fullsized":"2024-09-18T11:56:49.250909+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17028,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.251592+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:49.251592+0000","last_peered":"2024-09-18T11:56:49.251592+0000","last_clean":"2024-09-18T11:56:49.251592+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:49.251592+0000","last_undegraded":"2024-09-18T11:56:49.251592+0000","last_fullsized":"2024-09-18T11:56:49.251592+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13027,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.256076+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:50.256076+0000","last_peered":"2024-09-18T11:56:50.256076+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:50.256076+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13023,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255736+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:50.255736+0000","last_peered":"2024-09-18T11:56:50.255736+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:50.255736+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12914,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255889+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:50.255889+0000","last_peered":"2024-09-18T11:56:50.255889+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:50.255889+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17370,"reported_epoch":46,"state":"peering","last_fresh":"2024-09-18T11:57:00.287246+0000","last_change":"2024-09-18T11:57:00.289914+0000","last_active":"2024-09-18T11:57:00.287210+0000","last_peered":"2024-09-18T11:56:59.699934+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:57:00.287246+0000","last_undegraded":"2024-09-18T11:57:00.287246+0000","last_fullsized":"2024-09-18T11:57:00.287246+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250714+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:56:49.250714+0000","last_peered":"2024-09-18T11:56:49.250714+0000","last_clean":"2024-09-18T11:56:49.250714+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:56:49.250714+0000","last_undegraded":"2024-09-18T11:56:49.250714+0000","last_fullsized":"2024-09-18T11:56:49.250714+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17534,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.255321+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:50.255321+0000","last_peered":"2024-09-18T11:56:50.255321+0000","last_clean":"2024-09-18T11:56:50.255321+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:50.255321+0000","last_undegraded":"2024-09-18T11:56:50.255321+0000","last_fullsized":"2024-09-18T11:56:50.255321+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12714,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.251606+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:49.251606+0000","last_peered":"2024-09-18T11:56:49.251606+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:49.251606+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17247,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.331543+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:56:50.331543+0000","last_peered":"2024-09-18T11:56:50.331543+0000","last_clean":"2024-09-18T11:56:50.331543+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:56:50.331543+0000","last_undegraded":"2024-09-18T11:56:50.331543+0000","last_fullsized":"2024-09-18T11:56:50.331543+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13132,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.257448+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:50.257448+0000","last_peered":"2024-09-18T11:56:50.257448+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:50.257448+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13117,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:59.844880+0000","last_change":"2024-09-18T11:56:59.700508+0000","last_active":"2024-09-18T11:56:59.844880+0000","last_peered":"2024-09-18T11:56:59.844880+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:59.844880+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13222,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255903+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:50.255903+0000","last_peered":"2024-09-18T11:56:50.255903+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:50.255903+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17198,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.255016+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:56:49.255016+0000","last_peered":"2024-09-18T11:56:49.255016+0000","last_clean":"2024-09-18T11:56:49.255016+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:56:49.255016+0000","last_undegraded":"2024-09-18T11:56:49.255016+0000","last_fullsized":"2024-09-18T11:56:49.255016+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12814,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.250802+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:56:49.250802+0000","last_peered":"2024-09-18T11:56:49.250802+0000","last_clean":"2024-09-18T11:56:49.250802+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:56:49.250802+0000","last_undegraded":"2024-09-18T11:56:49.250802+0000","last_fullsized":"2024-09-18T11:56:49.250802+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16877,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":376832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923908,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447932,"statfs":{"total":96636764160,"available":95690682368,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.27200000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.193}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.254},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.23000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919236,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452604,"statfs":{"total":96636764160,"available":95695466496,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.17999999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.245}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.21299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.27800000000000002},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926236,"kb_used_data":724,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445604,"statfs":{"total":96636764160,"available":95688298496,"internally_reserved":0,"allocated":741376,"data_stored":523395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.28199999999999997},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.183}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.23799999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915528,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456312,"statfs":{"total":96636764160,"available":95699263488,"internally_reserved":0,"allocated":389120,"data_stored":166787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.184}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.223}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.248}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:05.759 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:06.040 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:06.051 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-18T11:57:04.115014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16877,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693244,"kb_used_data":1860,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794116,"statfs":{"total":386547056640,"available":382765174784,"internally_reserved":0,"allocated":1904640,"data_stored":1019639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":878,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001421"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13360,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.256596+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:56:50.256596+0000","last_peered":"2024-09-18T11:56:50.256596+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:56:50.256596+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17356,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283887+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:00.283887+0000","last_peered":"2024-09-18T11:57:00.283887+0000","last_clean":"2024-09-18T11:57:00.283887+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:00.283887+0000","last_undegraded":"2024-09-18T11:57:00.283887+0000","last_fullsized":"2024-09-18T11:57:00.283887+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17028,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T11:56:49.251592+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:56:49.251592+0000","last_peered":"2024-09-18T11:56:49.251592+0000","last_clean":"2024-09-18T11:56:49.251592+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:56:49.251592+0000","last_undegraded":"2024-09-18T11:56:49.251592+0000","last_fullsized":"2024-09-18T11:56:49.251592+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13027,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.256076+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:56:50.256076+0000","last_peered":"2024-09-18T11:56:50.256076+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:56:50.256076+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13023,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255736+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:56:50.255736+0000","last_peered":"2024-09-18T11:56:50.255736+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:56:50.255736+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12914,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255889+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:56:50.255889+0000","last_peered":"2024-09-18T11:56:50.255889+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:56:50.255889+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17370,"reported_epoch":46,"state":"peering","last_fresh":"2024-09-18T11:57:00.287246+0000","last_change":"2024-09-18T11:57:00.289914+0000","last_active":"2024-09-18T11:57:00.287210+0000","last_peered":"2024-09-18T11:56:59.699934+0000","last_clean":"2024-09-18T11:43:44.911411+0000","last_became_active":"2024-09-18T11:44:47.481088+0000","last_became_peered":"2024-09-18T11:44:47.481088+0000","last_unstale":"2024-09-18T11:57:00.287246+0000","last_undegraded":"2024-09-18T11:57:00.287246+0000","last_fullsized":"2024-09-18T11:57:00.287246+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17037,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283668+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:00.283668+0000","last_peered":"2024-09-18T11:57:00.283668+0000","last_clean":"2024-09-18T11:57:00.283668+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:00.283668+0000","last_undegraded":"2024-09-18T11:57:00.283668+0000","last_fullsized":"2024-09-18T11:57:00.283668+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17534,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.255321+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:56:50.255321+0000","last_peered":"2024-09-18T11:56:50.255321+0000","last_clean":"2024-09-18T11:56:50.255321+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:56:50.255321+0000","last_undegraded":"2024-09-18T11:56:50.255321+0000","last_fullsized":"2024-09-18T11:56:50.255321+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12714,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:49.251606+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:56:49.251606+0000","last_peered":"2024-09-18T11:56:49.251606+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:56:49.251606+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17247,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T11:56:50.331543+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:56:50.331543+0000","last_peered":"2024-09-18T11:56:50.331543+0000","last_clean":"2024-09-18T11:56:50.331543+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:56:50.331543+0000","last_undegraded":"2024-09-18T11:56:50.331543+0000","last_fullsized":"2024-09-18T11:56:50.331543+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13132,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.257448+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:56:50.257448+0000","last_peered":"2024-09-18T11:56:50.257448+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:56:50.257448+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13117,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:59.844880+0000","last_change":"2024-09-18T11:56:59.700508+0000","last_active":"2024-09-18T11:56:59.844880+0000","last_peered":"2024-09-18T11:56:59.844880+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:56:59.844880+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13222,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:50.255903+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:56:50.255903+0000","last_peered":"2024-09-18T11:56:50.255903+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:56:50.255903+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17200,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283660+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:00.283660+0000","last_peered":"2024-09-18T11:57:00.283660+0000","last_clean":"2024-09-18T11:57:00.283660+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:00.283660+0000","last_undegraded":"2024-09-18T11:57:00.283660+0000","last_fullsized":"2024-09-18T11:57:00.283660+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12816,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283674+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:00.283674+0000","last_peered":"2024-09-18T11:57:00.283674+0000","last_clean":"2024-09-18T11:57:00.283674+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:00.283674+0000","last_undegraded":"2024-09-18T11:57:00.283674+0000","last_fullsized":"2024-09-18T11:57:00.283674+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16877,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923908,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447932,"statfs":{"total":96636764160,"available":95690682368,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.27200000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":5.1340000000000003,"5min":7.8339999999999996,"15min":6.3849999999999998},"min":{"1min":0.48799999999999999,"5min":0.32900000000000001,"15min":0.23100000000000001},"max":{"1min":12.217000000000001,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":4.96,"5min":6.7640000000000002,"15min":5.2089999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":14.680999999999999,"5min":57.372,"15min":57.372},"last":0.193}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.254},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.23000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919236,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452604,"statfs":{"total":96636764160,"available":95695466496,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.17999999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.245}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.21299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.27800000000000002},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934572,"kb_used_data":736,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437268,"statfs":{"total":96636764160,"available":95679762432,"internally_reserved":0,"allocated":753664,"data_stored":526958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23599999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.317}]}]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915528,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456312,"statfs":{"total":96636764160,"available":95699263488,"internally_reserved":0,"allocated":389120,"data_stored":166787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.184}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.223}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.248}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:06.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:06.340 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:06.340 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:06.353 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-18T11:57:06.115336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17716,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693276,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794084,"statfs":{"total":386547056640,"available":382765142016,"internally_reserved":0,"allocated":1945600,"data_stored":1038776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001279"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13361,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291975+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:00.291975+0000","last_peered":"2024-09-18T11:57:00.291975+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:00.291975+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17356,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283887+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:00.283887+0000","last_peered":"2024-09-18T11:57:00.283887+0000","last_clean":"2024-09-18T11:57:00.283887+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:00.283887+0000","last_undegraded":"2024-09-18T11:57:00.283887+0000","last_fullsized":"2024-09-18T11:57:00.283887+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17030,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.282361+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:00.282361+0000","last_peered":"2024-09-18T11:57:00.282361+0000","last_clean":"2024-09-18T11:57:00.282361+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:00.282361+0000","last_undegraded":"2024-09-18T11:57:00.282361+0000","last_fullsized":"2024-09-18T11:57:00.282361+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13028,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.286177+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:57:00.286177+0000","last_peered":"2024-09-18T11:57:00.286177+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:00.286177+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13024,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291672+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:00.291672+0000","last_peered":"2024-09-18T11:57:00.291672+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:00.291672+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12915,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291782+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:00.291782+0000","last_peered":"2024-09-18T11:57:00.291782+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:00.291782+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.376777+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:01.376777+0000","last_peered":"2024-09-18T11:57:01.376777+0000","last_clean":"2024-09-18T11:57:01.376777+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:01.376777+0000","last_undegraded":"2024-09-18T11:57:01.376777+0000","last_fullsized":"2024-09-18T11:57:01.376777+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17037,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283668+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:00.283668+0000","last_peered":"2024-09-18T11:57:00.283668+0000","last_clean":"2024-09-18T11:57:00.283668+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:00.283668+0000","last_undegraded":"2024-09-18T11:57:00.283668+0000","last_fullsized":"2024-09-18T11:57:00.283668+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17535,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.291517+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:00.291517+0000","last_peered":"2024-09-18T11:57:00.291517+0000","last_clean":"2024-09-18T11:57:00.291517+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:00.291517+0000","last_undegraded":"2024-09-18T11:57:00.291517+0000","last_fullsized":"2024-09-18T11:57:00.291517+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.282581+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:00.282581+0000","last_peered":"2024-09-18T11:57:00.282581+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:00.282581+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17248,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.292639+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:00.292639+0000","last_peered":"2024-09-18T11:57:00.292639+0000","last_clean":"2024-09-18T11:57:00.292639+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:00.292639+0000","last_undegraded":"2024-09-18T11:57:00.292639+0000","last_fullsized":"2024-09-18T11:57:00.292639+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13133,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.292253+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:00.292253+0000","last_peered":"2024-09-18T11:57:00.292253+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:00.292253+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":14797,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:04.845160+0000","last_change":"2024-09-18T11:56:59.700508+0000","last_active":"2024-09-18T11:57:04.845160+0000","last_peered":"2024-09-18T11:57:04.845160+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:57:04.845160+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":870,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1185}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13223,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291801+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:00.291801+0000","last_peered":"2024-09-18T11:57:00.291801+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:00.291801+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17200,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283660+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:00.283660+0000","last_peered":"2024-09-18T11:57:00.283660+0000","last_clean":"2024-09-18T11:57:00.283660+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:00.283660+0000","last_undegraded":"2024-09-18T11:57:00.283660+0000","last_fullsized":"2024-09-18T11:57:00.283660+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12816,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283674+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:00.283674+0000","last_peered":"2024-09-18T11:57:00.283674+0000","last_clean":"2024-09-18T11:57:00.283674+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:00.283674+0000","last_undegraded":"2024-09-18T11:57:00.283674+0000","last_fullsized":"2024-09-18T11:57:00.283674+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17716,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923920,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447920,"statfs":{"total":96636764160,"available":95690670080,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.20000000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.22900000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.255},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.17999999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.245}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.21299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.27800000000000002},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934572,"kb_used_data":736,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437268,"statfs":{"total":96636764160,"available":95679762432,"internally_reserved":0,"allocated":753664,"data_stored":526958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23599999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.317}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915536,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456304,"statfs":{"total":96636764160,"available":95699255296,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.185},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.22600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.253}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:06.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:06.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13361, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.291975+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:00.291975+0000', 'last_peered': '2024-09-18T11:57:00.291975+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:00.291975+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:06.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:57:06.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13028, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.286177+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:57:00.286177+0000', 'last_peered': '2024-09-18T11:57:00.286177+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:57:00.286177+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:06.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:06.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13024, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.291672+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:00.291672+0000', 'last_peered': '2024-09-18T11:57:00.291672+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:00.291672+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:06.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:57:06.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12915, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.291782+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:57:00.291782+0000', 'last_peered': '2024-09-18T11:57:00.291782+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:57:00.291782+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:06.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:06.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12716, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.282581+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:57:00.282581+0000', 'last_peered': '2024-09-18T11:57:00.282581+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:57:00.282581+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:06.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:06.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13133, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.292253+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:00.292253+0000', 'last_peered': '2024-09-18T11:57:00.292253+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:00.292253+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:06.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:57:06.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 14797, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:04.845160+0000', 'last_change': '2024-09-18T11:56:59.700508+0000', 'last_active': '2024-09-18T11:57:04.845160+0000', 'last_peered': '2024-09-18T11:57:04.845160+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:57:04.845160+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 870, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1185}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:06.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:06.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13223, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.291801+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:57:00.291801+0000', 'last_peered': '2024-09-18T11:57:00.291801+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:00.291801+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:06.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:06.690 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:06.690 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15740/112 objects degraded (14053.571%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":824,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3781914624,"bytes_avail":382765142016,"bytes_total":386547056640,"degraded_objects":15740,"degraded_total":112,"degraded_ratio":140.53571428571428,"recovering_objects_per_sec":214,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1717,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:06.703 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:06.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:07.003 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:07.003 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:07.016 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-18T11:57:06.115336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17716,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693276,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794084,"statfs":{"total":386547056640,"available":382765142016,"internally_reserved":0,"allocated":1945600,"data_stored":1038776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001279"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13361,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291975+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:00.291975+0000","last_peered":"2024-09-18T11:57:00.291975+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:00.291975+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17356,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283887+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:00.283887+0000","last_peered":"2024-09-18T11:57:00.283887+0000","last_clean":"2024-09-18T11:57:00.283887+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:00.283887+0000","last_undegraded":"2024-09-18T11:57:00.283887+0000","last_fullsized":"2024-09-18T11:57:00.283887+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17030,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.282361+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:00.282361+0000","last_peered":"2024-09-18T11:57:00.282361+0000","last_clean":"2024-09-18T11:57:00.282361+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:00.282361+0000","last_undegraded":"2024-09-18T11:57:00.282361+0000","last_fullsized":"2024-09-18T11:57:00.282361+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13028,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.286177+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:57:00.286177+0000","last_peered":"2024-09-18T11:57:00.286177+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:00.286177+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13024,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291672+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:00.291672+0000","last_peered":"2024-09-18T11:57:00.291672+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:00.291672+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12915,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291782+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:00.291782+0000","last_peered":"2024-09-18T11:57:00.291782+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:00.291782+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.376777+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:01.376777+0000","last_peered":"2024-09-18T11:57:01.376777+0000","last_clean":"2024-09-18T11:57:01.376777+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:01.376777+0000","last_undegraded":"2024-09-18T11:57:01.376777+0000","last_fullsized":"2024-09-18T11:57:01.376777+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17037,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283668+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:00.283668+0000","last_peered":"2024-09-18T11:57:00.283668+0000","last_clean":"2024-09-18T11:57:00.283668+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:00.283668+0000","last_undegraded":"2024-09-18T11:57:00.283668+0000","last_fullsized":"2024-09-18T11:57:00.283668+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17535,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.291517+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:00.291517+0000","last_peered":"2024-09-18T11:57:00.291517+0000","last_clean":"2024-09-18T11:57:00.291517+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:00.291517+0000","last_undegraded":"2024-09-18T11:57:00.291517+0000","last_fullsized":"2024-09-18T11:57:00.291517+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.282581+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:00.282581+0000","last_peered":"2024-09-18T11:57:00.282581+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:00.282581+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17248,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.292639+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:00.292639+0000","last_peered":"2024-09-18T11:57:00.292639+0000","last_clean":"2024-09-18T11:57:00.292639+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:00.292639+0000","last_undegraded":"2024-09-18T11:57:00.292639+0000","last_fullsized":"2024-09-18T11:57:00.292639+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13133,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.292253+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:00.292253+0000","last_peered":"2024-09-18T11:57:00.292253+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:00.292253+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":14797,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:04.845160+0000","last_change":"2024-09-18T11:56:59.700508+0000","last_active":"2024-09-18T11:57:04.845160+0000","last_peered":"2024-09-18T11:57:04.845160+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:57:04.845160+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":870,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1185}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13223,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291801+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:00.291801+0000","last_peered":"2024-09-18T11:57:00.291801+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:00.291801+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17200,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283660+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:00.283660+0000","last_peered":"2024-09-18T11:57:00.283660+0000","last_clean":"2024-09-18T11:57:00.283660+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:00.283660+0000","last_undegraded":"2024-09-18T11:57:00.283660+0000","last_fullsized":"2024-09-18T11:57:00.283660+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12816,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283674+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:00.283674+0000","last_peered":"2024-09-18T11:57:00.283674+0000","last_clean":"2024-09-18T11:57:00.283674+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:00.283674+0000","last_undegraded":"2024-09-18T11:57:00.283674+0000","last_fullsized":"2024-09-18T11:57:00.283674+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17716,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923920,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447920,"statfs":{"total":96636764160,"available":95690670080,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.20000000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.22900000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.255},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.17999999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.245}]},{"osd":1,"last update":"Wed Sep 18 11:56:05 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":8.4440000000000008,"15min":6.9450000000000003},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":31.425999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":6.5919999999999996,"5min":5.6550000000000002,"15min":4.6200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":21.504999999999999,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.21299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.27800000000000002},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934572,"kb_used_data":736,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437268,"statfs":{"total":96636764160,"available":95679762432,"internally_reserved":0,"allocated":753664,"data_stored":526958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23599999999999999},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.317}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915536,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456304,"statfs":{"total":96636764160,"available":95699255296,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.185},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.22600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.253}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:10.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:10.310 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:10.310 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:10.326 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-09-18T11:57:10.116021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18797,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684952,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802408,"statfs":{"total":386547056640,"available":382773665792,"internally_reserved":0,"allocated":1945600,"data_stored":1038776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2776,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.835982"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13361,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291975+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:00.291975+0000","last_peered":"2024-09-18T11:57:00.291975+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:00.291975+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17356,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283887+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:00.283887+0000","last_peered":"2024-09-18T11:57:00.283887+0000","last_clean":"2024-09-18T11:57:00.283887+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:00.283887+0000","last_undegraded":"2024-09-18T11:57:00.283887+0000","last_fullsized":"2024-09-18T11:57:00.283887+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17030,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.282361+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:00.282361+0000","last_peered":"2024-09-18T11:57:00.282361+0000","last_clean":"2024-09-18T11:57:00.282361+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:00.282361+0000","last_undegraded":"2024-09-18T11:57:00.282361+0000","last_fullsized":"2024-09-18T11:57:00.282361+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13028,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.286177+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:57:00.286177+0000","last_peered":"2024-09-18T11:57:00.286177+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:00.286177+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13024,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291672+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:00.291672+0000","last_peered":"2024-09-18T11:57:00.291672+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:00.291672+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12915,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291782+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:00.291782+0000","last_peered":"2024-09-18T11:57:00.291782+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:00.291782+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.376777+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:01.376777+0000","last_peered":"2024-09-18T11:57:01.376777+0000","last_clean":"2024-09-18T11:57:01.376777+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:01.376777+0000","last_undegraded":"2024-09-18T11:57:01.376777+0000","last_fullsized":"2024-09-18T11:57:01.376777+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17037,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283668+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:00.283668+0000","last_peered":"2024-09-18T11:57:00.283668+0000","last_clean":"2024-09-18T11:57:00.283668+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:00.283668+0000","last_undegraded":"2024-09-18T11:57:00.283668+0000","last_fullsized":"2024-09-18T11:57:00.283668+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17535,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.291517+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:00.291517+0000","last_peered":"2024-09-18T11:57:00.291517+0000","last_clean":"2024-09-18T11:57:00.291517+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:00.291517+0000","last_undegraded":"2024-09-18T11:57:00.291517+0000","last_fullsized":"2024-09-18T11:57:00.291517+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.282581+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:00.282581+0000","last_peered":"2024-09-18T11:57:00.282581+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:00.282581+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17248,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.292639+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:00.292639+0000","last_peered":"2024-09-18T11:57:00.292639+0000","last_clean":"2024-09-18T11:57:00.292639+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:00.292639+0000","last_undegraded":"2024-09-18T11:57:00.292639+0000","last_fullsized":"2024-09-18T11:57:00.292639+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13133,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.292253+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:00.292253+0000","last_peered":"2024-09-18T11:57:00.292253+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:00.292253+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":16959,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:09.844907+0000","last_change":"2024-09-18T11:56:59.700508+0000","last_active":"2024-09-18T11:57:09.844907+0000","last_peered":"2024-09-18T11:57:09.844907+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:57:09.844907+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1951,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13223,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291801+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:00.291801+0000","last_peered":"2024-09-18T11:57:00.291801+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:00.291801+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17200,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283660+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:00.283660+0000","last_peered":"2024-09-18T11:57:00.283660+0000","last_clean":"2024-09-18T11:57:00.283660+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:00.283660+0000","last_undegraded":"2024-09-18T11:57:00.283660+0000","last_fullsized":"2024-09-18T11:57:00.283660+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12816,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283674+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:00.283674+0000","last_peered":"2024-09-18T11:57:00.283674+0000","last_clean":"2024-09-18T11:57:00.283674+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:00.283674+0000","last_undegraded":"2024-09-18T11:57:00.283674+0000","last_fullsized":"2024-09-18T11:57:00.283674+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18797,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923920,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447920,"statfs":{"total":96636764160,"available":95690670080,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.20000000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.22900000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.255},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.245}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.23499999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926248,"kb_used_data":736,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445592,"statfs":{"total":96636764160,"available":95688286208,"internally_reserved":0,"allocated":753664,"data_stored":526958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.20000000000000001},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.35699999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.23999999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915536,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456304,"statfs":{"total":96636764160,"available":95699255296,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.185},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.22600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.253}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:10.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:10.621 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:10.621 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:10.633 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-18T11:57:10.315011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18797,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684952,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802408,"statfs":{"total":386547056640,"available":382773665792,"internally_reserved":0,"allocated":1945600,"data_stored":1038776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1920,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.026439"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13361,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291975+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:00.291975+0000","last_peered":"2024-09-18T11:57:00.291975+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:00.291975+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17356,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283887+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:00.283887+0000","last_peered":"2024-09-18T11:57:00.283887+0000","last_clean":"2024-09-18T11:57:00.283887+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:00.283887+0000","last_undegraded":"2024-09-18T11:57:00.283887+0000","last_fullsized":"2024-09-18T11:57:00.283887+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17030,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.282361+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:00.282361+0000","last_peered":"2024-09-18T11:57:00.282361+0000","last_clean":"2024-09-18T11:57:00.282361+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:00.282361+0000","last_undegraded":"2024-09-18T11:57:00.282361+0000","last_fullsized":"2024-09-18T11:57:00.282361+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13028,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.286177+0000","last_change":"2024-09-18T11:44:47.508855+0000","last_active":"2024-09-18T11:57:00.286177+0000","last_peered":"2024-09-18T11:57:00.286177+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:00.286177+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13024,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291672+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:00.291672+0000","last_peered":"2024-09-18T11:57:00.291672+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:00.291672+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12915,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291782+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:00.291782+0000","last_peered":"2024-09-18T11:57:00.291782+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:00.291782+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.376777+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:01.376777+0000","last_peered":"2024-09-18T11:57:01.376777+0000","last_clean":"2024-09-18T11:57:01.376777+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:01.376777+0000","last_undegraded":"2024-09-18T11:57:01.376777+0000","last_fullsized":"2024-09-18T11:57:01.376777+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17037,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283668+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:00.283668+0000","last_peered":"2024-09-18T11:57:00.283668+0000","last_clean":"2024-09-18T11:57:00.283668+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:00.283668+0000","last_undegraded":"2024-09-18T11:57:00.283668+0000","last_fullsized":"2024-09-18T11:57:00.283668+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17535,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.291517+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:00.291517+0000","last_peered":"2024-09-18T11:57:00.291517+0000","last_clean":"2024-09-18T11:57:00.291517+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:00.291517+0000","last_undegraded":"2024-09-18T11:57:00.291517+0000","last_fullsized":"2024-09-18T11:57:00.291517+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.282581+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:00.282581+0000","last_peered":"2024-09-18T11:57:00.282581+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:00.282581+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17248,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.292639+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:00.292639+0000","last_peered":"2024-09-18T11:57:00.292639+0000","last_clean":"2024-09-18T11:57:00.292639+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:00.292639+0000","last_undegraded":"2024-09-18T11:57:00.292639+0000","last_fullsized":"2024-09-18T11:57:00.292639+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13133,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.292253+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:00.292253+0000","last_peered":"2024-09-18T11:57:00.292253+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:00.292253+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":16959,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:09.844907+0000","last_change":"2024-09-18T11:56:59.700508+0000","last_active":"2024-09-18T11:57:09.844907+0000","last_peered":"2024-09-18T11:57:09.844907+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:57:09.844907+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1951,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13223,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.291801+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:00.291801+0000","last_peered":"2024-09-18T11:57:00.291801+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:00.291801+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17200,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283660+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:00.283660+0000","last_peered":"2024-09-18T11:57:00.283660+0000","last_clean":"2024-09-18T11:57:00.283660+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:00.283660+0000","last_undegraded":"2024-09-18T11:57:00.283660+0000","last_fullsized":"2024-09-18T11:57:00.283660+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12816,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283674+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:00.283674+0000","last_peered":"2024-09-18T11:57:00.283674+0000","last_clean":"2024-09-18T11:57:00.283674+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:00.283674+0000","last_undegraded":"2024-09-18T11:57:00.283674+0000","last_fullsized":"2024-09-18T11:57:00.283674+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18797,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923920,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447920,"statfs":{"total":96636764160,"available":95690670080,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.20000000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.22900000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.255},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.245}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.23499999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926248,"kb_used_data":736,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445592,"statfs":{"total":96636764160,"available":95688286208,"internally_reserved":0,"allocated":753664,"data_stored":526958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.20000000000000001},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.35699999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.23999999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915536,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456304,"statfs":{"total":96636764160,"available":95699255296,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.185},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.22600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.253}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:10.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:10.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13361, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.291975+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:00.291975+0000', 'last_peered': '2024-09-18T11:57:00.291975+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:00.291975+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:10.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:57:10.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13028, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.286177+0000', 'last_change': '2024-09-18T11:44:47.508855+0000', 'last_active': '2024-09-18T11:57:00.286177+0000', 'last_peered': '2024-09-18T11:57:00.286177+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:57:00.286177+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:10.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:10.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13024, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.291672+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:00.291672+0000', 'last_peered': '2024-09-18T11:57:00.291672+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:00.291672+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:10.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:57:10.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12915, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.291782+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:57:00.291782+0000', 'last_peered': '2024-09-18T11:57:00.291782+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:57:00.291782+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:10.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:10.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12716, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.282581+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:57:00.282581+0000', 'last_peered': '2024-09-18T11:57:00.282581+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:57:00.282581+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:10.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:10.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13133, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.292253+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:00.292253+0000', 'last_peered': '2024-09-18T11:57:00.292253+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:00.292253+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:10.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-18T11:57:10.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "39'12966", 'reported_seq': 16959, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:09.844907+0000', 'last_change': '2024-09-18T11:56:59.700508+0000', 'last_active': '2024-09-18T11:57:09.844907+0000', 'last_peered': '2024-09-18T11:57:09.844907+0000', 'last_clean': '2024-09-18T11:43:39.896512+0000', 'last_became_active': '2024-09-18T11:44:47.447164+0000', 'last_became_peered': '2024-09-18T11:44:47.447164+0000', 'last_unstale': '2024-09-18T11:57:09.844907+0000', 'last_undegraded': '2024-09-18T11:44:47.155015+0000', 'last_fullsized': '2024-09-18T11:44:47.121871+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 105, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12965, 'num_write_kb': 51863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1951, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:10.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:10.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13223, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.291801+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:57:00.291801+0000', 'last_peered': '2024-09-18T11:57:00.291801+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:00.291801+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:10.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:11.004 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:11.005 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14659/112 objects degraded (13088.393%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":829,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3773390848,"bytes_avail":382773665792,"bytes_total":386547056640,"degraded_objects":14659,"degraded_total":112,"degraded_ratio":130.88392857142858,"recovering_objects_per_sec":282,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2776,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:11.017 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:11.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:11.339 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:11.339 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:11.350 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2024-09-18T11:57:11.319733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18872,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684960,"kb_used_data":1908,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802400,"statfs":{"total":386547056640,"available":382773657600,"internally_reserved":0,"allocated":1953792,"data_stored":1042579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1995,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.205015"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13362,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291319+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:01.291319+0000","last_peered":"2024-09-18T11:57:01.291319+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:01.291319+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17356,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283887+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:00.283887+0000","last_peered":"2024-09-18T11:57:00.283887+0000","last_clean":"2024-09-18T11:57:00.283887+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:00.283887+0000","last_undegraded":"2024-09-18T11:57:00.283887+0000","last_fullsized":"2024-09-18T11:57:00.283887+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17030,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.282361+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:00.282361+0000","last_peered":"2024-09-18T11:57:00.282361+0000","last_clean":"2024-09-18T11:57:00.282361+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:00.282361+0000","last_undegraded":"2024-09-18T11:57:00.282361+0000","last_fullsized":"2024-09-18T11:57:00.282361+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13180,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.906412+0000","last_change":"2024-09-18T11:57:10.283259+0000","last_active":"2024-09-18T11:57:10.906412+0000","last_peered":"2024-09-18T11:57:10.906412+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:10.906412+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2026}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13025,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291458+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:01.291458+0000","last_peered":"2024-09-18T11:57:01.291458+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:01.291458+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12916,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291290+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:01.291290+0000","last_peered":"2024-09-18T11:57:01.291290+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:01.291290+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17381,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.377206+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:01.377206+0000","last_peered":"2024-09-18T11:57:01.377206+0000","last_clean":"2024-09-18T11:57:01.377206+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:01.377206+0000","last_undegraded":"2024-09-18T11:57:01.377206+0000","last_fullsized":"2024-09-18T11:57:01.377206+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17037,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283668+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:00.283668+0000","last_peered":"2024-09-18T11:57:00.283668+0000","last_clean":"2024-09-18T11:57:00.283668+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:00.283668+0000","last_undegraded":"2024-09-18T11:57:00.283668+0000","last_fullsized":"2024-09-18T11:57:00.283668+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.290992+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:01.290992+0000","last_peered":"2024-09-18T11:57:01.290992+0000","last_clean":"2024-09-18T11:57:01.290992+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:01.290992+0000","last_undegraded":"2024-09-18T11:57:01.290992+0000","last_fullsized":"2024-09-18T11:57:01.290992+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.282581+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:00.282581+0000","last_peered":"2024-09-18T11:57:00.282581+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:00.282581+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17249,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.291078+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:01.291078+0000","last_peered":"2024-09-18T11:57:01.291078+0000","last_clean":"2024-09-18T11:57:01.291078+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:01.291078+0000","last_undegraded":"2024-09-18T11:57:01.291078+0000","last_fullsized":"2024-09-18T11:57:01.291078+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13134,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.292435+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:01.292435+0000","last_peered":"2024-09-18T11:57:01.292435+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:01.292435+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":16959,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:09.844907+0000","last_change":"2024-09-18T11:56:59.700508+0000","last_active":"2024-09-18T11:57:09.844907+0000","last_peered":"2024-09-18T11:57:09.844907+0000","last_clean":"2024-09-18T11:43:39.896512+0000","last_became_active":"2024-09-18T11:44:47.447164+0000","last_became_peered":"2024-09-18T11:44:47.447164+0000","last_unstale":"2024-09-18T11:57:09.844907+0000","last_undegraded":"2024-09-18T11:44:47.155015+0000","last_fullsized":"2024-09-18T11:44:47.121871+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1951,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13224,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291871+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:01.291871+0000","last_peered":"2024-09-18T11:57:01.291871+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:01.291871+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17200,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283660+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:00.283660+0000","last_peered":"2024-09-18T11:57:00.283660+0000","last_clean":"2024-09-18T11:57:00.283660+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:00.283660+0000","last_undegraded":"2024-09-18T11:57:00.283660+0000","last_fullsized":"2024-09-18T11:57:00.283660+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12816,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.283674+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:00.283674+0000","last_peered":"2024-09-18T11:57:00.283674+0000","last_clean":"2024-09-18T11:57:00.283674+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:00.283674+0000","last_undegraded":"2024-09-18T11:57:00.283674+0000","last_fullsized":"2024-09-18T11:57:00.283674+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18872,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923920,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447920,"statfs":{"total":96636764160,"available":95690670080,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.32400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.22}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.245}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.23499999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926248,"kb_used_data":736,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445592,"statfs":{"total":96636764160,"available":95688286208,"internally_reserved":0,"allocated":753664,"data_stored":526958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.601,"5min":7.5510000000000002,"15min":5.3280000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.159},"max":{"1min":22.119,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":8.2159999999999993,"5min":7.6040000000000001,"15min":5.8490000000000002},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.18099999999999999},"max":{"1min":20.704000000000001,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":8.1379999999999999,"15min":5.8819999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":19.995000000000001,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.20000000000000001},{"interface":"front","average":{"1min":7.351,"5min":6.5700000000000003,"15min":4.9870000000000001},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.19},"max":{"1min":22.224,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.35699999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:56:09 2024","interfaces":[{"interface":"back","average":{"1min":6.1440000000000001,"5min":6.5590000000000002,"15min":5.2000000000000002},"min":{"1min":0.53500000000000003,"5min":0.216,"15min":0.19},"max":{"1min":21.132999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.23999999999999999},{"interface":"front","average":{"1min":7.0810000000000004,"5min":7.8879999999999999,"15min":6.2290000000000001},"min":{"1min":0.84899999999999998,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":19.960000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915544,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456296,"statfs":{"total":96636764160,"available":95699247104,"internally_reserved":0,"allocated":405504,"data_stored":174409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.317}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.217},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.34599999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:14.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:14.729 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:14.730 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:14.742 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-18T11:57:14.116737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16930,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693296,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794064,"statfs":{"total":386547056640,"available":382765121536,"internally_reserved":0,"allocated":1961984,"data_stored":1041972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-786,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001401"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13362,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291319+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:01.291319+0000","last_peered":"2024-09-18T11:57:01.291319+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:01.291319+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17358,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320458+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:10.320458+0000","last_peered":"2024-09-18T11:57:10.320458+0000","last_clean":"2024-09-18T11:57:10.320458+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:10.320458+0000","last_undegraded":"2024-09-18T11:57:10.320458+0000","last_fullsized":"2024-09-18T11:57:10.320458+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17030,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.282361+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:00.282361+0000","last_peered":"2024-09-18T11:57:00.282361+0000","last_clean":"2024-09-18T11:57:00.282361+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:00.282361+0000","last_undegraded":"2024-09-18T11:57:00.282361+0000","last_fullsized":"2024-09-18T11:57:00.282361+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13180,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.906412+0000","last_change":"2024-09-18T11:57:10.283259+0000","last_active":"2024-09-18T11:57:10.906412+0000","last_peered":"2024-09-18T11:57:10.906412+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:10.906412+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2026}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13025,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291458+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:01.291458+0000","last_peered":"2024-09-18T11:57:01.291458+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:01.291458+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12916,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291290+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:01.291290+0000","last_peered":"2024-09-18T11:57:01.291290+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:01.291290+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17381,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.377206+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:01.377206+0000","last_peered":"2024-09-18T11:57:01.377206+0000","last_clean":"2024-09-18T11:57:01.377206+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:01.377206+0000","last_undegraded":"2024-09-18T11:57:01.377206+0000","last_fullsized":"2024-09-18T11:57:01.377206+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320180+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:10.320180+0000","last_peered":"2024-09-18T11:57:10.320180+0000","last_clean":"2024-09-18T11:57:10.320180+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:10.320180+0000","last_undegraded":"2024-09-18T11:57:10.320180+0000","last_fullsized":"2024-09-18T11:57:10.320180+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.290992+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:01.290992+0000","last_peered":"2024-09-18T11:57:01.290992+0000","last_clean":"2024-09-18T11:57:01.290992+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:01.290992+0000","last_undegraded":"2024-09-18T11:57:01.290992+0000","last_fullsized":"2024-09-18T11:57:01.290992+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.282581+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:00.282581+0000","last_peered":"2024-09-18T11:57:00.282581+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:00.282581+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17249,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.291078+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:01.291078+0000","last_peered":"2024-09-18T11:57:01.291078+0000","last_clean":"2024-09-18T11:57:01.291078+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:01.291078+0000","last_undegraded":"2024-09-18T11:57:01.291078+0000","last_fullsized":"2024-09-18T11:57:01.291078+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13134,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.292435+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:01.292435+0000","last_peered":"2024-09-18T11:57:01.292435+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:01.292435+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13081,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.388493+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:11.388493+0000","last_peered":"2024-09-18T11:57:11.388493+0000","last_clean":"2024-09-18T11:57:11.388493+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:11.388493+0000","last_undegraded":"2024-09-18T11:57:11.388493+0000","last_fullsized":"2024-09-18T11:57:11.388493+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13224,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291871+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:01.291871+0000","last_peered":"2024-09-18T11:57:01.291871+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:01.291871+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17202,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320115+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:10.320115+0000","last_peered":"2024-09-18T11:57:10.320115+0000","last_clean":"2024-09-18T11:57:10.320115+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:10.320115+0000","last_undegraded":"2024-09-18T11:57:10.320115+0000","last_fullsized":"2024-09-18T11:57:10.320115+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12818,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320224+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:10.320224+0000","last_peered":"2024-09-18T11:57:10.320224+0000","last_clean":"2024-09-18T11:57:10.320224+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:10.320224+0000","last_undegraded":"2024-09-18T11:57:10.320224+0000","last_fullsized":"2024-09-18T11:57:10.320224+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16930,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923920,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447920,"statfs":{"total":96636764160,"available":95690670080,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.32400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.22}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.245}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.23499999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934584,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437256,"statfs":{"total":96636764160,"available":95679750144,"internally_reserved":0,"allocated":761856,"data_stored":526351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.222},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.317},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915544,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456296,"statfs":{"total":96636764160,"available":95699247104,"internally_reserved":0,"allocated":405504,"data_stored":174409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.317}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.217},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.34599999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:14.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:15.020 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:15.020 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:15.032 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-18T11:57:14.116737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16930,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693296,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794064,"statfs":{"total":386547056640,"available":382765121536,"internally_reserved":0,"allocated":1961984,"data_stored":1041972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-786,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001401"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13362,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291319+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:01.291319+0000","last_peered":"2024-09-18T11:57:01.291319+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:01.291319+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17358,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320458+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:10.320458+0000","last_peered":"2024-09-18T11:57:10.320458+0000","last_clean":"2024-09-18T11:57:10.320458+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:10.320458+0000","last_undegraded":"2024-09-18T11:57:10.320458+0000","last_fullsized":"2024-09-18T11:57:10.320458+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17030,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.282361+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:00.282361+0000","last_peered":"2024-09-18T11:57:00.282361+0000","last_clean":"2024-09-18T11:57:00.282361+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:00.282361+0000","last_undegraded":"2024-09-18T11:57:00.282361+0000","last_fullsized":"2024-09-18T11:57:00.282361+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13180,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.906412+0000","last_change":"2024-09-18T11:57:10.283259+0000","last_active":"2024-09-18T11:57:10.906412+0000","last_peered":"2024-09-18T11:57:10.906412+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:10.906412+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2026}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13025,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291458+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:01.291458+0000","last_peered":"2024-09-18T11:57:01.291458+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:01.291458+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12916,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291290+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:01.291290+0000","last_peered":"2024-09-18T11:57:01.291290+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:01.291290+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17381,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.377206+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:01.377206+0000","last_peered":"2024-09-18T11:57:01.377206+0000","last_clean":"2024-09-18T11:57:01.377206+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:01.377206+0000","last_undegraded":"2024-09-18T11:57:01.377206+0000","last_fullsized":"2024-09-18T11:57:01.377206+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320180+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:10.320180+0000","last_peered":"2024-09-18T11:57:10.320180+0000","last_clean":"2024-09-18T11:57:10.320180+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:10.320180+0000","last_undegraded":"2024-09-18T11:57:10.320180+0000","last_fullsized":"2024-09-18T11:57:10.320180+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.290992+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:01.290992+0000","last_peered":"2024-09-18T11:57:01.290992+0000","last_clean":"2024-09-18T11:57:01.290992+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:01.290992+0000","last_undegraded":"2024-09-18T11:57:01.290992+0000","last_fullsized":"2024-09-18T11:57:01.290992+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.282581+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:00.282581+0000","last_peered":"2024-09-18T11:57:00.282581+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:00.282581+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17249,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.291078+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:01.291078+0000","last_peered":"2024-09-18T11:57:01.291078+0000","last_clean":"2024-09-18T11:57:01.291078+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:01.291078+0000","last_undegraded":"2024-09-18T11:57:01.291078+0000","last_fullsized":"2024-09-18T11:57:01.291078+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13134,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.292435+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:01.292435+0000","last_peered":"2024-09-18T11:57:01.292435+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:01.292435+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13081,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.388493+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:11.388493+0000","last_peered":"2024-09-18T11:57:11.388493+0000","last_clean":"2024-09-18T11:57:11.388493+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:11.388493+0000","last_undegraded":"2024-09-18T11:57:11.388493+0000","last_fullsized":"2024-09-18T11:57:11.388493+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13224,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291871+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:01.291871+0000","last_peered":"2024-09-18T11:57:01.291871+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:01.291871+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17202,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320115+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:10.320115+0000","last_peered":"2024-09-18T11:57:10.320115+0000","last_clean":"2024-09-18T11:57:10.320115+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:10.320115+0000","last_undegraded":"2024-09-18T11:57:10.320115+0000","last_fullsized":"2024-09-18T11:57:10.320115+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12818,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320224+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:10.320224+0000","last_peered":"2024-09-18T11:57:10.320224+0000","last_clean":"2024-09-18T11:57:10.320224+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:10.320224+0000","last_undegraded":"2024-09-18T11:57:10.320224+0000","last_fullsized":"2024-09-18T11:57:10.320224+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16930,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923920,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447920,"statfs":{"total":96636764160,"available":95690670080,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.32400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.22}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.245}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.23499999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934584,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437256,"statfs":{"total":96636764160,"available":95679750144,"internally_reserved":0,"allocated":761856,"data_stored":526351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.222},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.317},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915544,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456296,"statfs":{"total":96636764160,"available":95699247104,"internally_reserved":0,"allocated":405504,"data_stored":174409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.317}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.217},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.34599999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:15.033 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:15.033 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13362, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:01.291319+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:01.291319+0000', 'last_peered': '2024-09-18T11:57:01.291319+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:01.291319+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:15.033 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:57:15.033 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 13180, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.906412+0000', 'last_change': '2024-09-18T11:57:10.283259+0000', 'last_active': '2024-09-18T11:57:10.906412+0000', 'last_peered': '2024-09-18T11:57:10.906412+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:57:10.906412+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2026}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:15.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:15.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13025, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:01.291458+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:01.291458+0000', 'last_peered': '2024-09-18T11:57:01.291458+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:01.291458+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:15.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:57:15.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12916, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:01.291290+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:57:01.291290+0000', 'last_peered': '2024-09-18T11:57:01.291290+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:57:01.291290+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:15.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:15.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12716, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:00.282581+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:57:00.282581+0000', 'last_peered': '2024-09-18T11:57:00.282581+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:57:00.282581+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:15.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:15.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13134, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:01.292435+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:01.292435+0000', 'last_peered': '2024-09-18T11:57:01.292435+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:01.292435+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:15.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:15.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13224, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:01.291871+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:57:01.291871+0000', 'last_peered': '2024-09-18T11:57:01.291871+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:01.291871+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:15.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:15.380 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:15.380 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14479/112 objects degraded (12927.679%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":833,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3781935104,"bytes_avail":382765121536,"bytes_total":386547056640,"degraded_objects":14479,"degraded_total":112,"degraded_ratio":129.27678571428572},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:15.395 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T11:57:15.395 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:15.676 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:15.676 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:15.688 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-18T11:57:14.116737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16930,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693296,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794064,"statfs":{"total":386547056640,"available":382765121536,"internally_reserved":0,"allocated":1961984,"data_stored":1041972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-786,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001401"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13362,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291319+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:01.291319+0000","last_peered":"2024-09-18T11:57:01.291319+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:01.291319+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17358,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320458+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:10.320458+0000","last_peered":"2024-09-18T11:57:10.320458+0000","last_clean":"2024-09-18T11:57:10.320458+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:10.320458+0000","last_undegraded":"2024-09-18T11:57:10.320458+0000","last_fullsized":"2024-09-18T11:57:10.320458+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17030,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T11:57:00.282361+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:00.282361+0000","last_peered":"2024-09-18T11:57:00.282361+0000","last_clean":"2024-09-18T11:57:00.282361+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:00.282361+0000","last_undegraded":"2024-09-18T11:57:00.282361+0000","last_fullsized":"2024-09-18T11:57:00.282361+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13180,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.906412+0000","last_change":"2024-09-18T11:57:10.283259+0000","last_active":"2024-09-18T11:57:10.906412+0000","last_peered":"2024-09-18T11:57:10.906412+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:10.906412+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2026}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13025,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291458+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:01.291458+0000","last_peered":"2024-09-18T11:57:01.291458+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:01.291458+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12916,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291290+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:01.291290+0000","last_peered":"2024-09-18T11:57:01.291290+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:01.291290+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17381,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.377206+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:01.377206+0000","last_peered":"2024-09-18T11:57:01.377206+0000","last_clean":"2024-09-18T11:57:01.377206+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:01.377206+0000","last_undegraded":"2024-09-18T11:57:01.377206+0000","last_fullsized":"2024-09-18T11:57:01.377206+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320180+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:10.320180+0000","last_peered":"2024-09-18T11:57:10.320180+0000","last_clean":"2024-09-18T11:57:10.320180+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:10.320180+0000","last_undegraded":"2024-09-18T11:57:10.320180+0000","last_fullsized":"2024-09-18T11:57:10.320180+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.290992+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:01.290992+0000","last_peered":"2024-09-18T11:57:01.290992+0000","last_clean":"2024-09-18T11:57:01.290992+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:01.290992+0000","last_undegraded":"2024-09-18T11:57:01.290992+0000","last_fullsized":"2024-09-18T11:57:01.290992+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:00.282581+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:00.282581+0000","last_peered":"2024-09-18T11:57:00.282581+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:00.282581+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17249,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.291078+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:01.291078+0000","last_peered":"2024-09-18T11:57:01.291078+0000","last_clean":"2024-09-18T11:57:01.291078+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:01.291078+0000","last_undegraded":"2024-09-18T11:57:01.291078+0000","last_fullsized":"2024-09-18T11:57:01.291078+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13134,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.292435+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:01.292435+0000","last_peered":"2024-09-18T11:57:01.292435+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:01.292435+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13081,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.388493+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:11.388493+0000","last_peered":"2024-09-18T11:57:11.388493+0000","last_clean":"2024-09-18T11:57:11.388493+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:11.388493+0000","last_undegraded":"2024-09-18T11:57:11.388493+0000","last_fullsized":"2024-09-18T11:57:11.388493+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13224,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.291871+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:01.291871+0000","last_peered":"2024-09-18T11:57:01.291871+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:01.291871+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17202,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320115+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:10.320115+0000","last_peered":"2024-09-18T11:57:10.320115+0000","last_clean":"2024-09-18T11:57:10.320115+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:10.320115+0000","last_undegraded":"2024-09-18T11:57:10.320115+0000","last_fullsized":"2024-09-18T11:57:10.320115+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12818,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320224+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:10.320224+0000","last_peered":"2024-09-18T11:57:10.320224+0000","last_clean":"2024-09-18T11:57:10.320224+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:10.320224+0000","last_undegraded":"2024-09-18T11:57:10.320224+0000","last_fullsized":"2024-09-18T11:57:10.320224+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16930,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923920,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447920,"statfs":{"total":96636764160,"available":95690670080,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.32400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.22}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.245}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.23499999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934584,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437256,"statfs":{"total":96636764160,"available":95679750144,"internally_reserved":0,"allocated":761856,"data_stored":526351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.222},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.317},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915544,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456296,"statfs":{"total":96636764160,"available":95699247104,"internally_reserved":0,"allocated":405504,"data_stored":174409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:07 2024","interfaces":[{"interface":"back","average":{"1min":6.734,"5min":8.3360000000000003,"15min":5.569},"min":{"1min":0.16200000000000001,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":25.125,"5min":28.059000000000001,"15min":28.059000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":5.9139999999999997,"5min":6.883,"15min":5.1369999999999996},"min":{"1min":0.51800000000000002,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":27.818000000000001,"5min":34.786999999999999,"15min":34.786999999999999},"last":0.317}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.217},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.34599999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:18.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:18.964 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:18.965 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:18.976 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-09-18T11:57:18.117503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17728,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693348,"kb_used_data":1956,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794012,"statfs":{"total":386547056640,"available":382765068288,"internally_reserved":0,"allocated":2002944,"data_stored":1060924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001481"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13363,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322480+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:10.322480+0000","last_peered":"2024-09-18T11:57:10.322480+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:10.322480+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17358,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320458+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:10.320458+0000","last_peered":"2024-09-18T11:57:10.320458+0000","last_clean":"2024-09-18T11:57:10.320458+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:10.320458+0000","last_undegraded":"2024-09-18T11:57:10.320458+0000","last_fullsized":"2024-09-18T11:57:10.320458+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17032,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.318863+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:10.318863+0000","last_peered":"2024-09-18T11:57:10.318863+0000","last_clean":"2024-09-18T11:57:10.318863+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:10.318863+0000","last_undegraded":"2024-09-18T11:57:10.318863+0000","last_fullsized":"2024-09-18T11:57:10.318863+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":14777,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:15.907586+0000","last_change":"2024-09-18T11:57:10.283259+0000","last_active":"2024-09-18T11:57:15.907586+0000","last_peered":"2024-09-18T11:57:15.907586+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:15.907586+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":888,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1228}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13026,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322372+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:10.322372+0000","last_peered":"2024-09-18T11:57:10.322372+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:10.322372+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12917,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323147+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:10.323147+0000","last_peered":"2024-09-18T11:57:10.323147+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:10.323147+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17382,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.316575+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:10.316575+0000","last_peered":"2024-09-18T11:57:10.316575+0000","last_clean":"2024-09-18T11:57:10.316575+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:10.316575+0000","last_undegraded":"2024-09-18T11:57:10.316575+0000","last_fullsized":"2024-09-18T11:57:10.316575+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320180+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:10.320180+0000","last_peered":"2024-09-18T11:57:10.320180+0000","last_clean":"2024-09-18T11:57:10.320180+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:10.320180+0000","last_undegraded":"2024-09-18T11:57:10.320180+0000","last_fullsized":"2024-09-18T11:57:10.320180+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17537,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322870+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:10.322870+0000","last_peered":"2024-09-18T11:57:10.322870+0000","last_clean":"2024-09-18T11:57:10.322870+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:10.322870+0000","last_undegraded":"2024-09-18T11:57:10.322870+0000","last_fullsized":"2024-09-18T11:57:10.322870+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12718,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.319141+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:10.319141+0000","last_peered":"2024-09-18T11:57:10.319141+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:10.319141+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17250,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322950+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:10.322950+0000","last_peered":"2024-09-18T11:57:10.322950+0000","last_clean":"2024-09-18T11:57:10.322950+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:10.322950+0000","last_undegraded":"2024-09-18T11:57:10.322950+0000","last_fullsized":"2024-09-18T11:57:10.322950+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13135,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323699+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:10.323699+0000","last_peered":"2024-09-18T11:57:10.323699+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:10.323699+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13081,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.388493+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:11.388493+0000","last_peered":"2024-09-18T11:57:11.388493+0000","last_clean":"2024-09-18T11:57:11.388493+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:11.388493+0000","last_undegraded":"2024-09-18T11:57:11.388493+0000","last_fullsized":"2024-09-18T11:57:11.388493+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13225,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322856+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:10.322856+0000","last_peered":"2024-09-18T11:57:10.322856+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:10.322856+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17202,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320115+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:10.320115+0000","last_peered":"2024-09-18T11:57:10.320115+0000","last_clean":"2024-09-18T11:57:10.320115+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:10.320115+0000","last_undegraded":"2024-09-18T11:57:10.320115+0000","last_fullsized":"2024-09-18T11:57:10.320115+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12818,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320224+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:10.320224+0000","last_peered":"2024-09-18T11:57:10.320224+0000","last_clean":"2024-09-18T11:57:10.320224+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:10.320224+0000","last_undegraded":"2024-09-18T11:57:10.320224+0000","last_fullsized":"2024-09-18T11:57:10.320224+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17728,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923940,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447900,"statfs":{"total":96636764160,"available":95690649600,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.26800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.24099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.28599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919268,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452572,"statfs":{"total":96636764160,"available":95695433728,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.32800000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.27100000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.30499999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.252}]}]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934584,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437256,"statfs":{"total":96636764160,"available":95679750144,"internally_reserved":0,"allocated":761856,"data_stored":526351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.222},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.317},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915556,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456284,"statfs":{"total":96636764160,"available":95699234816,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.185}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.20300000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.21199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.23799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:18.978 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:19.269 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:19.269 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:19.282 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-09-18T11:57:18.117503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17728,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693348,"kb_used_data":1956,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794012,"statfs":{"total":386547056640,"available":382765068288,"internally_reserved":0,"allocated":2002944,"data_stored":1060924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001481"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13363,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322480+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:10.322480+0000","last_peered":"2024-09-18T11:57:10.322480+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:10.322480+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17358,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320458+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:10.320458+0000","last_peered":"2024-09-18T11:57:10.320458+0000","last_clean":"2024-09-18T11:57:10.320458+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:10.320458+0000","last_undegraded":"2024-09-18T11:57:10.320458+0000","last_fullsized":"2024-09-18T11:57:10.320458+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17032,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.318863+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:10.318863+0000","last_peered":"2024-09-18T11:57:10.318863+0000","last_clean":"2024-09-18T11:57:10.318863+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:10.318863+0000","last_undegraded":"2024-09-18T11:57:10.318863+0000","last_fullsized":"2024-09-18T11:57:10.318863+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":14777,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:15.907586+0000","last_change":"2024-09-18T11:57:10.283259+0000","last_active":"2024-09-18T11:57:15.907586+0000","last_peered":"2024-09-18T11:57:15.907586+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:15.907586+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":888,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1228}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13026,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322372+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:10.322372+0000","last_peered":"2024-09-18T11:57:10.322372+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:10.322372+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12917,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323147+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:10.323147+0000","last_peered":"2024-09-18T11:57:10.323147+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:10.323147+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17382,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.316575+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:10.316575+0000","last_peered":"2024-09-18T11:57:10.316575+0000","last_clean":"2024-09-18T11:57:10.316575+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:10.316575+0000","last_undegraded":"2024-09-18T11:57:10.316575+0000","last_fullsized":"2024-09-18T11:57:10.316575+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320180+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:10.320180+0000","last_peered":"2024-09-18T11:57:10.320180+0000","last_clean":"2024-09-18T11:57:10.320180+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:10.320180+0000","last_undegraded":"2024-09-18T11:57:10.320180+0000","last_fullsized":"2024-09-18T11:57:10.320180+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17537,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322870+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:10.322870+0000","last_peered":"2024-09-18T11:57:10.322870+0000","last_clean":"2024-09-18T11:57:10.322870+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:10.322870+0000","last_undegraded":"2024-09-18T11:57:10.322870+0000","last_fullsized":"2024-09-18T11:57:10.322870+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12718,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.319141+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:10.319141+0000","last_peered":"2024-09-18T11:57:10.319141+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:10.319141+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17250,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322950+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:10.322950+0000","last_peered":"2024-09-18T11:57:10.322950+0000","last_clean":"2024-09-18T11:57:10.322950+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:10.322950+0000","last_undegraded":"2024-09-18T11:57:10.322950+0000","last_fullsized":"2024-09-18T11:57:10.322950+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13135,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323699+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:10.323699+0000","last_peered":"2024-09-18T11:57:10.323699+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:10.323699+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13081,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.388493+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:11.388493+0000","last_peered":"2024-09-18T11:57:11.388493+0000","last_clean":"2024-09-18T11:57:11.388493+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:11.388493+0000","last_undegraded":"2024-09-18T11:57:11.388493+0000","last_fullsized":"2024-09-18T11:57:11.388493+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13225,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322856+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:10.322856+0000","last_peered":"2024-09-18T11:57:10.322856+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:10.322856+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17202,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320115+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:10.320115+0000","last_peered":"2024-09-18T11:57:10.320115+0000","last_clean":"2024-09-18T11:57:10.320115+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:10.320115+0000","last_undegraded":"2024-09-18T11:57:10.320115+0000","last_fullsized":"2024-09-18T11:57:10.320115+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12818,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320224+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:10.320224+0000","last_peered":"2024-09-18T11:57:10.320224+0000","last_clean":"2024-09-18T11:57:10.320224+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:10.320224+0000","last_undegraded":"2024-09-18T11:57:10.320224+0000","last_fullsized":"2024-09-18T11:57:10.320224+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17728,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923940,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447900,"statfs":{"total":96636764160,"available":95690649600,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.26800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.24099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.28599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919268,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452572,"statfs":{"total":96636764160,"available":95695433728,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.32800000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.27100000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.30499999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.252}]}]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934584,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437256,"statfs":{"total":96636764160,"available":95679750144,"internally_reserved":0,"allocated":761856,"data_stored":526351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.222},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.317},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915556,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456284,"statfs":{"total":96636764160,"available":95699234816,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.185}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.20300000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.21199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.23799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:19.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:19.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13363, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.322480+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:10.322480+0000', 'last_peered': '2024-09-18T11:57:10.322480+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:10.322480+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:19.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:57:19.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 14777, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:15.907586+0000', 'last_change': '2024-09-18T11:57:10.283259+0000', 'last_active': '2024-09-18T11:57:15.907586+0000', 'last_peered': '2024-09-18T11:57:15.907586+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:57:15.907586+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 888, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1228}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:19.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:19.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13026, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.322372+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:10.322372+0000', 'last_peered': '2024-09-18T11:57:10.322372+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:10.322372+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:19.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:57:19.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12917, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.323147+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:57:10.323147+0000', 'last_peered': '2024-09-18T11:57:10.323147+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:57:10.323147+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:19.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:19.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12718, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.319141+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:57:10.319141+0000', 'last_peered': '2024-09-18T11:57:10.319141+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:57:10.319141+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:19.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:19.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13135, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.323699+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:10.323699+0000', 'last_peered': '2024-09-18T11:57:10.323699+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:10.323699+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:19.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:19.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13225, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.322856+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:57:10.322856+0000', 'last_peered': '2024-09-18T11:57:10.322856+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:10.322856+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:19.284 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:19.634 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:19.634 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13681/112 objects degraded (12215.179%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":837,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3781988352,"bytes_avail":382765068288,"bytes_total":386547056640,"degraded_objects":13681,"degraded_total":112,"degraded_ratio":122.15178571428571},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:19.647 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T11:57:19.647 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:19.948 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:19.949 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:19.961 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-09-18T11:57:18.117503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17728,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693348,"kb_used_data":1956,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794012,"statfs":{"total":386547056640,"available":382765068288,"internally_reserved":0,"allocated":2002944,"data_stored":1060924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001481"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13363,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322480+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:10.322480+0000","last_peered":"2024-09-18T11:57:10.322480+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:10.322480+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17358,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320458+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:10.320458+0000","last_peered":"2024-09-18T11:57:10.320458+0000","last_clean":"2024-09-18T11:57:10.320458+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:10.320458+0000","last_undegraded":"2024-09-18T11:57:10.320458+0000","last_fullsized":"2024-09-18T11:57:10.320458+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17032,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.318863+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:10.318863+0000","last_peered":"2024-09-18T11:57:10.318863+0000","last_clean":"2024-09-18T11:57:10.318863+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:10.318863+0000","last_undegraded":"2024-09-18T11:57:10.318863+0000","last_fullsized":"2024-09-18T11:57:10.318863+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":14777,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:15.907586+0000","last_change":"2024-09-18T11:57:10.283259+0000","last_active":"2024-09-18T11:57:15.907586+0000","last_peered":"2024-09-18T11:57:15.907586+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:15.907586+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":888,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1228}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13026,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322372+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:10.322372+0000","last_peered":"2024-09-18T11:57:10.322372+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:10.322372+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12917,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323147+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:10.323147+0000","last_peered":"2024-09-18T11:57:10.323147+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:10.323147+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17382,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.316575+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:10.316575+0000","last_peered":"2024-09-18T11:57:10.316575+0000","last_clean":"2024-09-18T11:57:10.316575+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:10.316575+0000","last_undegraded":"2024-09-18T11:57:10.316575+0000","last_fullsized":"2024-09-18T11:57:10.316575+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320180+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:10.320180+0000","last_peered":"2024-09-18T11:57:10.320180+0000","last_clean":"2024-09-18T11:57:10.320180+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:10.320180+0000","last_undegraded":"2024-09-18T11:57:10.320180+0000","last_fullsized":"2024-09-18T11:57:10.320180+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17537,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322870+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:10.322870+0000","last_peered":"2024-09-18T11:57:10.322870+0000","last_clean":"2024-09-18T11:57:10.322870+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:10.322870+0000","last_undegraded":"2024-09-18T11:57:10.322870+0000","last_fullsized":"2024-09-18T11:57:10.322870+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12718,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.319141+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:10.319141+0000","last_peered":"2024-09-18T11:57:10.319141+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:10.319141+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17250,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322950+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:10.322950+0000","last_peered":"2024-09-18T11:57:10.322950+0000","last_clean":"2024-09-18T11:57:10.322950+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:10.322950+0000","last_undegraded":"2024-09-18T11:57:10.322950+0000","last_fullsized":"2024-09-18T11:57:10.322950+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13135,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323699+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:10.323699+0000","last_peered":"2024-09-18T11:57:10.323699+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:10.323699+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13081,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.388493+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:11.388493+0000","last_peered":"2024-09-18T11:57:11.388493+0000","last_clean":"2024-09-18T11:57:11.388493+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:11.388493+0000","last_undegraded":"2024-09-18T11:57:11.388493+0000","last_fullsized":"2024-09-18T11:57:11.388493+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13225,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322856+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:10.322856+0000","last_peered":"2024-09-18T11:57:10.322856+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:10.322856+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17202,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320115+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:10.320115+0000","last_peered":"2024-09-18T11:57:10.320115+0000","last_clean":"2024-09-18T11:57:10.320115+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:10.320115+0000","last_undegraded":"2024-09-18T11:57:10.320115+0000","last_fullsized":"2024-09-18T11:57:10.320115+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12818,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320224+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:10.320224+0000","last_peered":"2024-09-18T11:57:10.320224+0000","last_clean":"2024-09-18T11:57:10.320224+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:10.320224+0000","last_undegraded":"2024-09-18T11:57:10.320224+0000","last_fullsized":"2024-09-18T11:57:10.320224+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17728,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923940,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447900,"statfs":{"total":96636764160,"available":95690649600,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.26800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.24099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.28599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919268,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452572,"statfs":{"total":96636764160,"available":95695433728,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.32800000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.27100000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.30499999999999999},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.252}]}]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934584,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437256,"statfs":{"total":96636764160,"available":95679750144,"internally_reserved":0,"allocated":761856,"data_stored":526351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.222},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.317},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915556,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456284,"statfs":{"total":96636764160,"available":95699234816,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.185}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.20300000000000001},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.21199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.23799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:22.963 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:23.287 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:23.288 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:23.299 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-18T11:57:22.499044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18810,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3685020,"kb_used_data":1956,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802340,"statfs":{"total":386547056640,"available":382773596160,"internally_reserved":0,"allocated":2002944,"data_stored":1060924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1880,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.382307"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13363,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322480+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:10.322480+0000","last_peered":"2024-09-18T11:57:10.322480+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:10.322480+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17358,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320458+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:10.320458+0000","last_peered":"2024-09-18T11:57:10.320458+0000","last_clean":"2024-09-18T11:57:10.320458+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:10.320458+0000","last_undegraded":"2024-09-18T11:57:10.320458+0000","last_fullsized":"2024-09-18T11:57:10.320458+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17032,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.318863+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:10.318863+0000","last_peered":"2024-09-18T11:57:10.318863+0000","last_clean":"2024-09-18T11:57:10.318863+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:10.318863+0000","last_undegraded":"2024-09-18T11:57:10.318863+0000","last_fullsized":"2024-09-18T11:57:10.318863+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":16941,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:20.903710+0000","last_change":"2024-09-18T11:57:10.283259+0000","last_active":"2024-09-18T11:57:20.903710+0000","last_peered":"2024-09-18T11:57:20.903710+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:20.903710+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1970,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13026,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322372+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:10.322372+0000","last_peered":"2024-09-18T11:57:10.322372+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:10.322372+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12917,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323147+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:10.323147+0000","last_peered":"2024-09-18T11:57:10.323147+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:10.323147+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17382,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.316575+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:10.316575+0000","last_peered":"2024-09-18T11:57:10.316575+0000","last_clean":"2024-09-18T11:57:10.316575+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:10.316575+0000","last_undegraded":"2024-09-18T11:57:10.316575+0000","last_fullsized":"2024-09-18T11:57:10.316575+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320180+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:10.320180+0000","last_peered":"2024-09-18T11:57:10.320180+0000","last_clean":"2024-09-18T11:57:10.320180+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:10.320180+0000","last_undegraded":"2024-09-18T11:57:10.320180+0000","last_fullsized":"2024-09-18T11:57:10.320180+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17537,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322870+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:10.322870+0000","last_peered":"2024-09-18T11:57:10.322870+0000","last_clean":"2024-09-18T11:57:10.322870+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:10.322870+0000","last_undegraded":"2024-09-18T11:57:10.322870+0000","last_fullsized":"2024-09-18T11:57:10.322870+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12718,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.319141+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:10.319141+0000","last_peered":"2024-09-18T11:57:10.319141+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:10.319141+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17250,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322950+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:10.322950+0000","last_peered":"2024-09-18T11:57:10.322950+0000","last_clean":"2024-09-18T11:57:10.322950+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:10.322950+0000","last_undegraded":"2024-09-18T11:57:10.322950+0000","last_fullsized":"2024-09-18T11:57:10.322950+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13135,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323699+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:10.323699+0000","last_peered":"2024-09-18T11:57:10.323699+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:10.323699+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13081,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.388493+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:11.388493+0000","last_peered":"2024-09-18T11:57:11.388493+0000","last_clean":"2024-09-18T11:57:11.388493+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:11.388493+0000","last_undegraded":"2024-09-18T11:57:11.388493+0000","last_fullsized":"2024-09-18T11:57:11.388493+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13225,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322856+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:10.322856+0000","last_peered":"2024-09-18T11:57:10.322856+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:10.322856+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17202,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320115+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:10.320115+0000","last_peered":"2024-09-18T11:57:10.320115+0000","last_clean":"2024-09-18T11:57:10.320115+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:10.320115+0000","last_undegraded":"2024-09-18T11:57:10.320115+0000","last_fullsized":"2024-09-18T11:57:10.320115+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12818,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320224+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:10.320224+0000","last_peered":"2024-09-18T11:57:10.320224+0000","last_clean":"2024-09-18T11:57:10.320224+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:10.320224+0000","last_undegraded":"2024-09-18T11:57:10.320224+0000","last_fullsized":"2024-09-18T11:57:10.320224+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18810,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923940,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447900,"statfs":{"total":96636764160,"available":95690649600,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.19800000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.248}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.27100000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919268,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452572,"statfs":{"total":96636764160,"available":95695433728,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.17599999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.248}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.21299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.27600000000000002},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26900000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926256,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445584,"statfs":{"total":96636764160,"available":95688278016,"internally_reserved":0,"allocated":761856,"data_stored":526351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.161},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.214},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.23899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915556,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456284,"statfs":{"total":96636764160,"available":95699234816,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.161},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.23699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.255},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.193},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:23.300 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:23.595 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:23.607 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-18T11:57:22.499044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18810,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3685020,"kb_used_data":1956,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802340,"statfs":{"total":386547056640,"available":382773596160,"internally_reserved":0,"allocated":2002944,"data_stored":1060924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1880,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.382307"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13363,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322480+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:10.322480+0000","last_peered":"2024-09-18T11:57:10.322480+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:10.322480+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17358,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320458+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:10.320458+0000","last_peered":"2024-09-18T11:57:10.320458+0000","last_clean":"2024-09-18T11:57:10.320458+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:10.320458+0000","last_undegraded":"2024-09-18T11:57:10.320458+0000","last_fullsized":"2024-09-18T11:57:10.320458+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17032,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.318863+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:10.318863+0000","last_peered":"2024-09-18T11:57:10.318863+0000","last_clean":"2024-09-18T11:57:10.318863+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:10.318863+0000","last_undegraded":"2024-09-18T11:57:10.318863+0000","last_fullsized":"2024-09-18T11:57:10.318863+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":16941,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:20.903710+0000","last_change":"2024-09-18T11:57:10.283259+0000","last_active":"2024-09-18T11:57:20.903710+0000","last_peered":"2024-09-18T11:57:20.903710+0000","last_clean":"2024-09-18T11:43:44.319986+0000","last_became_active":"2024-09-18T11:44:47.481955+0000","last_became_peered":"2024-09-18T11:44:47.481955+0000","last_unstale":"2024-09-18T11:57:20.903710+0000","last_undegraded":"2024-09-18T11:44:47.186043+0000","last_fullsized":"2024-09-18T11:44:47.122141+0000","mapping_epoch":30,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1970,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13026,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322372+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:10.322372+0000","last_peered":"2024-09-18T11:57:10.322372+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:10.322372+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12917,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323147+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:10.323147+0000","last_peered":"2024-09-18T11:57:10.323147+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:10.323147+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17382,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.316575+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:10.316575+0000","last_peered":"2024-09-18T11:57:10.316575+0000","last_clean":"2024-09-18T11:57:10.316575+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:10.316575+0000","last_undegraded":"2024-09-18T11:57:10.316575+0000","last_fullsized":"2024-09-18T11:57:10.316575+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320180+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:10.320180+0000","last_peered":"2024-09-18T11:57:10.320180+0000","last_clean":"2024-09-18T11:57:10.320180+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:10.320180+0000","last_undegraded":"2024-09-18T11:57:10.320180+0000","last_fullsized":"2024-09-18T11:57:10.320180+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17537,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322870+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:10.322870+0000","last_peered":"2024-09-18T11:57:10.322870+0000","last_clean":"2024-09-18T11:57:10.322870+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:10.322870+0000","last_undegraded":"2024-09-18T11:57:10.322870+0000","last_fullsized":"2024-09-18T11:57:10.322870+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12718,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.319141+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:10.319141+0000","last_peered":"2024-09-18T11:57:10.319141+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:10.319141+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17250,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322950+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:10.322950+0000","last_peered":"2024-09-18T11:57:10.322950+0000","last_clean":"2024-09-18T11:57:10.322950+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:10.322950+0000","last_undegraded":"2024-09-18T11:57:10.322950+0000","last_fullsized":"2024-09-18T11:57:10.322950+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13135,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323699+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:10.323699+0000","last_peered":"2024-09-18T11:57:10.323699+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:10.323699+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13081,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.388493+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:11.388493+0000","last_peered":"2024-09-18T11:57:11.388493+0000","last_clean":"2024-09-18T11:57:11.388493+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:11.388493+0000","last_undegraded":"2024-09-18T11:57:11.388493+0000","last_fullsized":"2024-09-18T11:57:11.388493+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13225,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322856+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:10.322856+0000","last_peered":"2024-09-18T11:57:10.322856+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:10.322856+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17202,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320115+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:10.320115+0000","last_peered":"2024-09-18T11:57:10.320115+0000","last_clean":"2024-09-18T11:57:10.320115+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:10.320115+0000","last_undegraded":"2024-09-18T11:57:10.320115+0000","last_fullsized":"2024-09-18T11:57:10.320115+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12818,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.320224+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:10.320224+0000","last_peered":"2024-09-18T11:57:10.320224+0000","last_clean":"2024-09-18T11:57:10.320224+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:10.320224+0000","last_undegraded":"2024-09-18T11:57:10.320224+0000","last_fullsized":"2024-09-18T11:57:10.320224+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18810,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923940,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447900,"statfs":{"total":96636764160,"available":95690649600,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.19800000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.248}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.27100000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919268,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452572,"statfs":{"total":96636764160,"available":95695433728,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.17599999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.248}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.21299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.27600000000000002},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26900000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926256,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445584,"statfs":{"total":96636764160,"available":95688278016,"internally_reserved":0,"allocated":761856,"data_stored":526351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.161},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.214},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.23899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915556,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456284,"statfs":{"total":96636764160,"available":95699234816,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.161},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.23699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.255},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.193},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:23.609 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:23.609 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13363, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.322480+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:10.322480+0000', 'last_peered': '2024-09-18T11:57:10.322480+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:10.322480+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:23.609 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-18T11:57:23.609 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "39'12924", 'reported_seq': 16941, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:20.903710+0000', 'last_change': '2024-09-18T11:57:10.283259+0000', 'last_active': '2024-09-18T11:57:20.903710+0000', 'last_peered': '2024-09-18T11:57:20.903710+0000', 'last_clean': '2024-09-18T11:43:44.319986+0000', 'last_became_active': '2024-09-18T11:44:47.481955+0000', 'last_became_peered': '2024-09-18T11:44:47.481955+0000', 'last_unstale': '2024-09-18T11:57:20.903710+0000', 'last_undegraded': '2024-09-18T11:44:47.186043+0000', 'last_fullsized': '2024-09-18T11:44:47.122141+0000', 'mapping_epoch': 30, 'log_start': "31'2900", 'ondisk_log_start': "31'2900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 148, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12922, 'num_write_kb': 51693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1970, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -146}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:23.609 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:23.609 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13026, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.322372+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:10.322372+0000', 'last_peered': '2024-09-18T11:57:10.322372+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:10.322372+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:23.609 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:57:23.609 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 12917, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.323147+0000', 'last_change': '2024-09-18T11:44:47.401029+0000', 'last_active': '2024-09-18T11:57:10.323147+0000', 'last_peered': '2024-09-18T11:57:10.323147+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:57:10.323147+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:23.609 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:23.609 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12718, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.319141+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:57:10.319141+0000', 'last_peered': '2024-09-18T11:57:10.319141+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:57:10.319141+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:23.610 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:23.610 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13135, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.323699+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:10.323699+0000', 'last_peered': '2024-09-18T11:57:10.323699+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:10.323699+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:23.610 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:23.610 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13225, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:10.322856+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:57:10.322856+0000', 'last_peered': '2024-09-18T11:57:10.322856+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:10.322856+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:23.610 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:23.956 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12599/112 objects degraded (11249.107%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":842,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3773460480,"bytes_avail":382773596160,"bytes_total":386547056640,"degraded_objects":12599,"degraded_total":112,"degraded_ratio":112.49107142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:23.969 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T11:57:23.970 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:24.270 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:24.271 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:24.283 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-18T11:57:24.118516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18958,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693360,"kb_used_data":1972,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373794000,"statfs":{"total":386547056640,"available":382765056000,"internally_reserved":0,"allocated":2019328,"data_stored":1068435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001372"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13363,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322480+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:10.322480+0000","last_peered":"2024-09-18T11:57:10.322480+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:10.322480+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17360,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497538+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:21.497538+0000","last_peered":"2024-09-18T11:57:21.497538+0000","last_clean":"2024-09-18T11:57:21.497538+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:21.497538+0000","last_undegraded":"2024-09-18T11:57:21.497538+0000","last_fullsized":"2024-09-18T11:57:21.497538+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17032,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.318863+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:10.318863+0000","last_peered":"2024-09-18T11:57:10.318863+0000","last_clean":"2024-09-18T11:57:10.318863+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:10.318863+0000","last_undegraded":"2024-09-18T11:57:10.318863+0000","last_fullsized":"2024-09-18T11:57:10.318863+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13035,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.573687+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:22.573687+0000","last_peered":"2024-09-18T11:57:22.573687+0000","last_clean":"2024-09-18T11:57:22.573687+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:22.573687+0000","last_undegraded":"2024-09-18T11:57:22.573687+0000","last_fullsized":"2024-09-18T11:57:22.573687+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13026,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322372+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:10.322372+0000","last_peered":"2024-09-18T11:57:10.322372+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:10.322372+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":12917,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323147+0000","last_change":"2024-09-18T11:44:47.401029+0000","last_active":"2024-09-18T11:57:10.323147+0000","last_peered":"2024-09-18T11:57:10.323147+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:10.323147+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17382,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.316575+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:10.316575+0000","last_peered":"2024-09-18T11:57:10.316575+0000","last_clean":"2024-09-18T11:57:10.316575+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:10.316575+0000","last_undegraded":"2024-09-18T11:57:10.316575+0000","last_fullsized":"2024-09-18T11:57:10.316575+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17041,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497310+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:21.497310+0000","last_peered":"2024-09-18T11:57:21.497310+0000","last_clean":"2024-09-18T11:57:21.497310+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:21.497310+0000","last_undegraded":"2024-09-18T11:57:21.497310+0000","last_fullsized":"2024-09-18T11:57:21.497310+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17537,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322870+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:10.322870+0000","last_peered":"2024-09-18T11:57:10.322870+0000","last_clean":"2024-09-18T11:57:10.322870+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:10.322870+0000","last_undegraded":"2024-09-18T11:57:10.322870+0000","last_fullsized":"2024-09-18T11:57:10.322870+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12718,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.319141+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:10.319141+0000","last_peered":"2024-09-18T11:57:10.319141+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:10.319141+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17250,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T11:57:10.322950+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:10.322950+0000","last_peered":"2024-09-18T11:57:10.322950+0000","last_clean":"2024-09-18T11:57:10.322950+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:10.322950+0000","last_undegraded":"2024-09-18T11:57:10.322950+0000","last_fullsized":"2024-09-18T11:57:10.322950+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13135,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.323699+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:10.323699+0000","last_peered":"2024-09-18T11:57:10.323699+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:10.323699+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13083,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497148+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:21.497148+0000","last_peered":"2024-09-18T11:57:21.497148+0000","last_clean":"2024-09-18T11:57:21.497148+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:21.497148+0000","last_undegraded":"2024-09-18T11:57:21.497148+0000","last_fullsized":"2024-09-18T11:57:21.497148+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13225,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:10.322856+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:10.322856+0000","last_peered":"2024-09-18T11:57:10.322856+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:10.322856+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17204,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497284+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:21.497284+0000","last_peered":"2024-09-18T11:57:21.497284+0000","last_clean":"2024-09-18T11:57:21.497284+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:21.497284+0000","last_undegraded":"2024-09-18T11:57:21.497284+0000","last_fullsized":"2024-09-18T11:57:21.497284+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12820,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497365+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:21.497365+0000","last_peered":"2024-09-18T11:57:21.497365+0000","last_clean":"2024-09-18T11:57:21.497365+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:21.497365+0000","last_undegraded":"2024-09-18T11:57:21.497365+0000","last_fullsized":"2024-09-18T11:57:21.497365+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18958,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923940,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447900,"statfs":{"total":96636764160,"available":95690649600,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.19800000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.248}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.27100000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919268,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452572,"statfs":{"total":96636764160,"available":95695433728,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:19 2024","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":5.7750000000000004,"15min":4.335},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.153},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.17599999999999999},{"interface":"front","average":{"1min":5.6139999999999999,"5min":5.9749999999999996,"15min":4.0030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.17699999999999999},"max":{"1min":33.259,"5min":33.259,"15min":33.259},"last":0.248}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.21299999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.27600000000000002},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26900000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934596,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437244,"statfs":{"total":96636764160,"available":95679737856,"internally_reserved":0,"allocated":778240,"data_stored":533862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.253}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.223},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.23599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915556,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456284,"statfs":{"total":96636764160,"available":95699234816,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.161},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.23699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.255},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.193},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:27.284 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:27.559 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:27.559 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:27.571 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-18T11:57:26.118882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19599,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693400,"kb_used_data":2020,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793960,"statfs":{"total":386547056640,"available":382765015040,"internally_reserved":0,"allocated":2068480,"data_stored":1090968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1871,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001378"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499874+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:21.499874+0000","last_peered":"2024-09-18T11:57:21.499874+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:21.499874+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17360,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497538+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:21.497538+0000","last_peered":"2024-09-18T11:57:21.497538+0000","last_clean":"2024-09-18T11:57:21.497538+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:21.497538+0000","last_undegraded":"2024-09-18T11:57:21.497538+0000","last_fullsized":"2024-09-18T11:57:21.497538+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17034,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499651+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:21.499651+0000","last_peered":"2024-09-18T11:57:21.499651+0000","last_clean":"2024-09-18T11:57:21.499651+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:21.499651+0000","last_undegraded":"2024-09-18T11:57:21.499651+0000","last_fullsized":"2024-09-18T11:57:21.499651+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13035,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.573687+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:22.573687+0000","last_peered":"2024-09-18T11:57:22.573687+0000","last_clean":"2024-09-18T11:57:22.573687+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:22.573687+0000","last_undegraded":"2024-09-18T11:57:22.573687+0000","last_fullsized":"2024-09-18T11:57:22.573687+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13028,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499894+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:21.499894+0000","last_peered":"2024-09-18T11:57:21.499894+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:21.499894+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":14202,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:25.328185+0000","last_change":"2024-09-18T11:57:21.442179+0000","last_active":"2024-09-18T11:57:25.328185+0000","last_peered":"2024-09-18T11:57:25.328185+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:25.328185+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1478,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17384,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.501542+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:21.501542+0000","last_peered":"2024-09-18T11:57:21.501542+0000","last_clean":"2024-09-18T11:57:21.501542+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:21.501542+0000","last_undegraded":"2024-09-18T11:57:21.501542+0000","last_fullsized":"2024-09-18T11:57:21.501542+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17041,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497310+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:21.497310+0000","last_peered":"2024-09-18T11:57:21.497310+0000","last_clean":"2024-09-18T11:57:21.497310+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:21.497310+0000","last_undegraded":"2024-09-18T11:57:21.497310+0000","last_fullsized":"2024-09-18T11:57:21.497310+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17539,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499368+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:21.499368+0000","last_peered":"2024-09-18T11:57:21.499368+0000","last_clean":"2024-09-18T11:57:21.499368+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:21.499368+0000","last_undegraded":"2024-09-18T11:57:21.499368+0000","last_fullsized":"2024-09-18T11:57:21.499368+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12720,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499923+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:21.499923+0000","last_peered":"2024-09-18T11:57:21.499923+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:21.499923+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17252,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499889+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:21.499889+0000","last_peered":"2024-09-18T11:57:21.499889+0000","last_clean":"2024-09-18T11:57:21.499889+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:21.499889+0000","last_undegraded":"2024-09-18T11:57:21.499889+0000","last_fullsized":"2024-09-18T11:57:21.499889+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13137,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500416+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:21.500416+0000","last_peered":"2024-09-18T11:57:21.500416+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:21.500416+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13083,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497148+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:21.497148+0000","last_peered":"2024-09-18T11:57:21.497148+0000","last_clean":"2024-09-18T11:57:21.497148+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:21.497148+0000","last_undegraded":"2024-09-18T11:57:21.497148+0000","last_fullsized":"2024-09-18T11:57:21.497148+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13227,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500272+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:21.500272+0000","last_peered":"2024-09-18T11:57:21.500272+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:21.500272+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17204,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497284+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:21.497284+0000","last_peered":"2024-09-18T11:57:21.497284+0000","last_clean":"2024-09-18T11:57:21.497284+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:21.497284+0000","last_undegraded":"2024-09-18T11:57:21.497284+0000","last_fullsized":"2024-09-18T11:57:21.497284+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12820,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497365+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:21.497365+0000","last_peered":"2024-09-18T11:57:21.497365+0000","last_clean":"2024-09-18T11:57:21.497365+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:21.497365+0000","last_undegraded":"2024-09-18T11:57:21.497365+0000","last_fullsized":"2024-09-18T11:57:21.497365+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19599,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923956,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447884,"statfs":{"total":96636764160,"available":95690633216,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.187},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.222}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.253},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.23599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.27300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.28000000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.252},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26200000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934596,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437244,"statfs":{"total":96636764160,"available":95679737856,"internally_reserved":0,"allocated":778240,"data_stored":533862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.253}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.223},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.23599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915568,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456272,"statfs":{"total":96636764160,"available":95699222528,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.161},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.23699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.255},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.193},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:27.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:27.889 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:27.889 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:27.901 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-18T11:57:26.118882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19599,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693400,"kb_used_data":2020,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793960,"statfs":{"total":386547056640,"available":382765015040,"internally_reserved":0,"allocated":2068480,"data_stored":1090968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1871,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001378"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499874+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:21.499874+0000","last_peered":"2024-09-18T11:57:21.499874+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:21.499874+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17360,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497538+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:21.497538+0000","last_peered":"2024-09-18T11:57:21.497538+0000","last_clean":"2024-09-18T11:57:21.497538+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:21.497538+0000","last_undegraded":"2024-09-18T11:57:21.497538+0000","last_fullsized":"2024-09-18T11:57:21.497538+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17034,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499651+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:21.499651+0000","last_peered":"2024-09-18T11:57:21.499651+0000","last_clean":"2024-09-18T11:57:21.499651+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:21.499651+0000","last_undegraded":"2024-09-18T11:57:21.499651+0000","last_fullsized":"2024-09-18T11:57:21.499651+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13035,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.573687+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:22.573687+0000","last_peered":"2024-09-18T11:57:22.573687+0000","last_clean":"2024-09-18T11:57:22.573687+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:22.573687+0000","last_undegraded":"2024-09-18T11:57:22.573687+0000","last_fullsized":"2024-09-18T11:57:22.573687+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13028,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499894+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:21.499894+0000","last_peered":"2024-09-18T11:57:21.499894+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:21.499894+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":14202,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:25.328185+0000","last_change":"2024-09-18T11:57:21.442179+0000","last_active":"2024-09-18T11:57:25.328185+0000","last_peered":"2024-09-18T11:57:25.328185+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:25.328185+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1478,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17384,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.501542+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:21.501542+0000","last_peered":"2024-09-18T11:57:21.501542+0000","last_clean":"2024-09-18T11:57:21.501542+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:21.501542+0000","last_undegraded":"2024-09-18T11:57:21.501542+0000","last_fullsized":"2024-09-18T11:57:21.501542+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17041,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497310+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:21.497310+0000","last_peered":"2024-09-18T11:57:21.497310+0000","last_clean":"2024-09-18T11:57:21.497310+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:21.497310+0000","last_undegraded":"2024-09-18T11:57:21.497310+0000","last_fullsized":"2024-09-18T11:57:21.497310+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17539,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499368+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:21.499368+0000","last_peered":"2024-09-18T11:57:21.499368+0000","last_clean":"2024-09-18T11:57:21.499368+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:21.499368+0000","last_undegraded":"2024-09-18T11:57:21.499368+0000","last_fullsized":"2024-09-18T11:57:21.499368+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12720,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499923+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:21.499923+0000","last_peered":"2024-09-18T11:57:21.499923+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:21.499923+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17252,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499889+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:21.499889+0000","last_peered":"2024-09-18T11:57:21.499889+0000","last_clean":"2024-09-18T11:57:21.499889+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:21.499889+0000","last_undegraded":"2024-09-18T11:57:21.499889+0000","last_fullsized":"2024-09-18T11:57:21.499889+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13137,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500416+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:21.500416+0000","last_peered":"2024-09-18T11:57:21.500416+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:21.500416+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13083,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497148+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:21.497148+0000","last_peered":"2024-09-18T11:57:21.497148+0000","last_clean":"2024-09-18T11:57:21.497148+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:21.497148+0000","last_undegraded":"2024-09-18T11:57:21.497148+0000","last_fullsized":"2024-09-18T11:57:21.497148+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13227,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500272+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:21.500272+0000","last_peered":"2024-09-18T11:57:21.500272+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:21.500272+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17204,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497284+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:21.497284+0000","last_peered":"2024-09-18T11:57:21.497284+0000","last_clean":"2024-09-18T11:57:21.497284+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:21.497284+0000","last_undegraded":"2024-09-18T11:57:21.497284+0000","last_fullsized":"2024-09-18T11:57:21.497284+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12820,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497365+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:21.497365+0000","last_peered":"2024-09-18T11:57:21.497365+0000","last_clean":"2024-09-18T11:57:21.497365+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:21.497365+0000","last_undegraded":"2024-09-18T11:57:21.497365+0000","last_fullsized":"2024-09-18T11:57:21.497365+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19599,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923956,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447884,"statfs":{"total":96636764160,"available":95690633216,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.187},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.222}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.253},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.23599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.27300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.28000000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.252},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26200000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934596,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437244,"statfs":{"total":96636764160,"available":95679737856,"internally_reserved":0,"allocated":778240,"data_stored":533862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.253}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.223},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.23599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915568,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456272,"statfs":{"total":96636764160,"available":95699222528,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.161},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.23699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.255},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.193},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:27.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:27.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13365, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.499874+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:21.499874+0000', 'last_peered': '2024-09-18T11:57:21.499874+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:21.499874+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:27.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:27.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13028, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.499894+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:21.499894+0000', 'last_peered': '2024-09-18T11:57:21.499894+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:21.499894+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:27.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:57:27.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 14202, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:25.328185+0000', 'last_change': '2024-09-18T11:57:21.442179+0000', 'last_active': '2024-09-18T11:57:25.328185+0000', 'last_peered': '2024-09-18T11:57:25.328185+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:57:25.328185+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1478, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 676, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1478}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:27.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:27.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12720, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.499923+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:57:21.499923+0000', 'last_peered': '2024-09-18T11:57:21.499923+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:57:21.499923+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:27.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:27.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13137, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.500416+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:21.500416+0000', 'last_peered': '2024-09-18T11:57:21.500416+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:21.500416+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:27.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:27.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13227, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.500272+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:57:21.500272+0000', 'last_peered': '2024-09-18T11:57:21.500272+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:21.500272+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:27.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:28.252 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:28.252 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11810/112 objects degraded (10544.643%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":846,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3782041600,"bytes_avail":382765015040,"bytes_total":386547056640,"degraded_objects":11810,"degraded_total":112,"degraded_ratio":105.44642857142857,"recovering_objects_per_sec":233,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1871,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:28.265 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:28.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:28.552 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:28.553 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:28.565 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-09-18T11:57:28.119320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19599,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693400,"kb_used_data":2020,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793960,"statfs":{"total":386547056640,"available":382765015040,"internally_reserved":0,"allocated":2068480,"data_stored":1090968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1871,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001525"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499874+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:21.499874+0000","last_peered":"2024-09-18T11:57:21.499874+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:21.499874+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17360,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497538+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:21.497538+0000","last_peered":"2024-09-18T11:57:21.497538+0000","last_clean":"2024-09-18T11:57:21.497538+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:21.497538+0000","last_undegraded":"2024-09-18T11:57:21.497538+0000","last_fullsized":"2024-09-18T11:57:21.497538+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17034,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499651+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:21.499651+0000","last_peered":"2024-09-18T11:57:21.499651+0000","last_clean":"2024-09-18T11:57:21.499651+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:21.499651+0000","last_undegraded":"2024-09-18T11:57:21.499651+0000","last_fullsized":"2024-09-18T11:57:21.499651+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13035,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.573687+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:22.573687+0000","last_peered":"2024-09-18T11:57:22.573687+0000","last_clean":"2024-09-18T11:57:22.573687+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:22.573687+0000","last_undegraded":"2024-09-18T11:57:22.573687+0000","last_fullsized":"2024-09-18T11:57:22.573687+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13028,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499894+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:21.499894+0000","last_peered":"2024-09-18T11:57:21.499894+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:21.499894+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":14202,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:25.328185+0000","last_change":"2024-09-18T11:57:21.442179+0000","last_active":"2024-09-18T11:57:25.328185+0000","last_peered":"2024-09-18T11:57:25.328185+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:25.328185+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1478,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17384,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.501542+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:21.501542+0000","last_peered":"2024-09-18T11:57:21.501542+0000","last_clean":"2024-09-18T11:57:21.501542+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:21.501542+0000","last_undegraded":"2024-09-18T11:57:21.501542+0000","last_fullsized":"2024-09-18T11:57:21.501542+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17041,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497310+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:21.497310+0000","last_peered":"2024-09-18T11:57:21.497310+0000","last_clean":"2024-09-18T11:57:21.497310+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:21.497310+0000","last_undegraded":"2024-09-18T11:57:21.497310+0000","last_fullsized":"2024-09-18T11:57:21.497310+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17539,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499368+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:21.499368+0000","last_peered":"2024-09-18T11:57:21.499368+0000","last_clean":"2024-09-18T11:57:21.499368+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:21.499368+0000","last_undegraded":"2024-09-18T11:57:21.499368+0000","last_fullsized":"2024-09-18T11:57:21.499368+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12720,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499923+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:21.499923+0000","last_peered":"2024-09-18T11:57:21.499923+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:21.499923+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17252,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499889+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:21.499889+0000","last_peered":"2024-09-18T11:57:21.499889+0000","last_clean":"2024-09-18T11:57:21.499889+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:21.499889+0000","last_undegraded":"2024-09-18T11:57:21.499889+0000","last_fullsized":"2024-09-18T11:57:21.499889+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13137,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500416+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:21.500416+0000","last_peered":"2024-09-18T11:57:21.500416+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:21.500416+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13083,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497148+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:21.497148+0000","last_peered":"2024-09-18T11:57:21.497148+0000","last_clean":"2024-09-18T11:57:21.497148+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:21.497148+0000","last_undegraded":"2024-09-18T11:57:21.497148+0000","last_fullsized":"2024-09-18T11:57:21.497148+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13227,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500272+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:21.500272+0000","last_peered":"2024-09-18T11:57:21.500272+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:21.500272+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17204,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497284+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:21.497284+0000","last_peered":"2024-09-18T11:57:21.497284+0000","last_clean":"2024-09-18T11:57:21.497284+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:21.497284+0000","last_undegraded":"2024-09-18T11:57:21.497284+0000","last_fullsized":"2024-09-18T11:57:21.497284+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12820,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497365+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:21.497365+0000","last_peered":"2024-09-18T11:57:21.497365+0000","last_clean":"2024-09-18T11:57:21.497365+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:21.497365+0000","last_undegraded":"2024-09-18T11:57:21.497365+0000","last_fullsized":"2024-09-18T11:57:21.497365+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19599,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923956,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447884,"statfs":{"total":96636764160,"available":95690633216,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.187},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.222}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.253},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.23599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.27300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29899999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.28000000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.252},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26200000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934596,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437244,"statfs":{"total":96636764160,"available":95679737856,"internally_reserved":0,"allocated":778240,"data_stored":533862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.253}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.223},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.23599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.26500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915568,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456272,"statfs":{"total":96636764160,"available":95699222528,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.161},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.23699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.255},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.193},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:31.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:31.844 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:31.844 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:31.855 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-18T11:57:30.119643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19599,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3685076,"kb_used_data":2020,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802284,"statfs":{"total":386547056640,"available":382773538816,"internally_reserved":0,"allocated":2068480,"data_stored":1090968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":789,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.623593"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499874+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:21.499874+0000","last_peered":"2024-09-18T11:57:21.499874+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:21.499874+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17360,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497538+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:21.497538+0000","last_peered":"2024-09-18T11:57:21.497538+0000","last_clean":"2024-09-18T11:57:21.497538+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:21.497538+0000","last_undegraded":"2024-09-18T11:57:21.497538+0000","last_fullsized":"2024-09-18T11:57:21.497538+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17034,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499651+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:21.499651+0000","last_peered":"2024-09-18T11:57:21.499651+0000","last_clean":"2024-09-18T11:57:21.499651+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:21.499651+0000","last_undegraded":"2024-09-18T11:57:21.499651+0000","last_fullsized":"2024-09-18T11:57:21.499651+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13035,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.573687+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:22.573687+0000","last_peered":"2024-09-18T11:57:22.573687+0000","last_clean":"2024-09-18T11:57:22.573687+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:22.573687+0000","last_undegraded":"2024-09-18T11:57:22.573687+0000","last_fullsized":"2024-09-18T11:57:22.573687+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13028,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499894+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:21.499894+0000","last_peered":"2024-09-18T11:57:21.499894+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:21.499894+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":14202,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:25.328185+0000","last_change":"2024-09-18T11:57:21.442179+0000","last_active":"2024-09-18T11:57:25.328185+0000","last_peered":"2024-09-18T11:57:25.328185+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:25.328185+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1478,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17384,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.501542+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:21.501542+0000","last_peered":"2024-09-18T11:57:21.501542+0000","last_clean":"2024-09-18T11:57:21.501542+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:21.501542+0000","last_undegraded":"2024-09-18T11:57:21.501542+0000","last_fullsized":"2024-09-18T11:57:21.501542+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17041,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497310+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:21.497310+0000","last_peered":"2024-09-18T11:57:21.497310+0000","last_clean":"2024-09-18T11:57:21.497310+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:21.497310+0000","last_undegraded":"2024-09-18T11:57:21.497310+0000","last_fullsized":"2024-09-18T11:57:21.497310+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17539,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499368+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:21.499368+0000","last_peered":"2024-09-18T11:57:21.499368+0000","last_clean":"2024-09-18T11:57:21.499368+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:21.499368+0000","last_undegraded":"2024-09-18T11:57:21.499368+0000","last_fullsized":"2024-09-18T11:57:21.499368+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12720,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499923+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:21.499923+0000","last_peered":"2024-09-18T11:57:21.499923+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:21.499923+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17252,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499889+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:21.499889+0000","last_peered":"2024-09-18T11:57:21.499889+0000","last_clean":"2024-09-18T11:57:21.499889+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:21.499889+0000","last_undegraded":"2024-09-18T11:57:21.499889+0000","last_fullsized":"2024-09-18T11:57:21.499889+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13137,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500416+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:21.500416+0000","last_peered":"2024-09-18T11:57:21.500416+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:21.500416+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13083,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497148+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:21.497148+0000","last_peered":"2024-09-18T11:57:21.497148+0000","last_clean":"2024-09-18T11:57:21.497148+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:21.497148+0000","last_undegraded":"2024-09-18T11:57:21.497148+0000","last_fullsized":"2024-09-18T11:57:21.497148+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13227,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500272+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:21.500272+0000","last_peered":"2024-09-18T11:57:21.500272+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:21.500272+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17204,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497284+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:21.497284+0000","last_peered":"2024-09-18T11:57:21.497284+0000","last_clean":"2024-09-18T11:57:21.497284+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:21.497284+0000","last_undegraded":"2024-09-18T11:57:21.497284+0000","last_fullsized":"2024-09-18T11:57:21.497284+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12820,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497365+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:21.497365+0000","last_peered":"2024-09-18T11:57:21.497365+0000","last_clean":"2024-09-18T11:57:21.497365+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:21.497365+0000","last_undegraded":"2024-09-18T11:57:21.497365+0000","last_fullsized":"2024-09-18T11:57:21.497365+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19599,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923956,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447884,"statfs":{"total":96636764160,"available":95690633216,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.187},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.222}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.253},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.23599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.27300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.27600000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.23200000000000001},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926272,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445568,"statfs":{"total":96636764160,"available":95688261632,"internally_reserved":0,"allocated":778240,"data_stored":533862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.252}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.219},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.28199999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915568,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456272,"statfs":{"total":96636764160,"available":95699222528,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.161},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.23699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:22 2024","interfaces":[{"interface":"back","average":{"1min":4.4349999999999996,"5min":5.8899999999999997,"15min":4.5670000000000002},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.14299999999999999},"max":{"1min":13.397,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.255},{"interface":"front","average":{"1min":4.6639999999999997,"5min":5.9740000000000002,"15min":4.1340000000000003},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":11.403,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.26800000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:26 2024","interfaces":[{"interface":"back","average":{"1min":5.9109999999999996,"5min":7.1139999999999999,"15min":4.6029999999999998},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":25.166,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.193},{"interface":"front","average":{"1min":4.6799999999999997,"5min":8.7530000000000001,"15min":5.4969999999999999},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":18.625,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:31.857 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:32.138 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:32.150 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2024-09-18T11:57:32.120035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20572,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3685076,"kb_used_data":2020,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802284,"statfs":{"total":386547056640,"available":382773538816,"internally_reserved":0,"allocated":2068480,"data_stored":1090968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001916"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499874+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:21.499874+0000","last_peered":"2024-09-18T11:57:21.499874+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:21.499874+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17360,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497538+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:21.497538+0000","last_peered":"2024-09-18T11:57:21.497538+0000","last_clean":"2024-09-18T11:57:21.497538+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:21.497538+0000","last_undegraded":"2024-09-18T11:57:21.497538+0000","last_fullsized":"2024-09-18T11:57:21.497538+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17034,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499651+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:21.499651+0000","last_peered":"2024-09-18T11:57:21.499651+0000","last_clean":"2024-09-18T11:57:21.499651+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:21.499651+0000","last_undegraded":"2024-09-18T11:57:21.499651+0000","last_fullsized":"2024-09-18T11:57:21.499651+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13035,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.573687+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:22.573687+0000","last_peered":"2024-09-18T11:57:22.573687+0000","last_clean":"2024-09-18T11:57:22.573687+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:22.573687+0000","last_undegraded":"2024-09-18T11:57:22.573687+0000","last_fullsized":"2024-09-18T11:57:22.573687+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13028,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499894+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:21.499894+0000","last_peered":"2024-09-18T11:57:21.499894+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:21.499894+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":16148,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:30.330048+0000","last_change":"2024-09-18T11:57:21.442179+0000","last_active":"2024-09-18T11:57:30.330048+0000","last_peered":"2024-09-18T11:57:30.330048+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:30.330048+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17384,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.501542+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:21.501542+0000","last_peered":"2024-09-18T11:57:21.501542+0000","last_clean":"2024-09-18T11:57:21.501542+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:21.501542+0000","last_undegraded":"2024-09-18T11:57:21.501542+0000","last_fullsized":"2024-09-18T11:57:21.501542+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17041,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497310+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:21.497310+0000","last_peered":"2024-09-18T11:57:21.497310+0000","last_clean":"2024-09-18T11:57:21.497310+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:21.497310+0000","last_undegraded":"2024-09-18T11:57:21.497310+0000","last_fullsized":"2024-09-18T11:57:21.497310+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17539,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499368+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:21.499368+0000","last_peered":"2024-09-18T11:57:21.499368+0000","last_clean":"2024-09-18T11:57:21.499368+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:21.499368+0000","last_undegraded":"2024-09-18T11:57:21.499368+0000","last_fullsized":"2024-09-18T11:57:21.499368+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12720,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499923+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:21.499923+0000","last_peered":"2024-09-18T11:57:21.499923+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:21.499923+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17252,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499889+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:21.499889+0000","last_peered":"2024-09-18T11:57:21.499889+0000","last_clean":"2024-09-18T11:57:21.499889+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:21.499889+0000","last_undegraded":"2024-09-18T11:57:21.499889+0000","last_fullsized":"2024-09-18T11:57:21.499889+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13137,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500416+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:21.500416+0000","last_peered":"2024-09-18T11:57:21.500416+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:21.500416+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13083,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497148+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:21.497148+0000","last_peered":"2024-09-18T11:57:21.497148+0000","last_clean":"2024-09-18T11:57:21.497148+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:21.497148+0000","last_undegraded":"2024-09-18T11:57:21.497148+0000","last_fullsized":"2024-09-18T11:57:21.497148+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13227,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500272+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:21.500272+0000","last_peered":"2024-09-18T11:57:21.500272+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:21.500272+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17204,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497284+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:21.497284+0000","last_peered":"2024-09-18T11:57:21.497284+0000","last_clean":"2024-09-18T11:57:21.497284+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:21.497284+0000","last_undegraded":"2024-09-18T11:57:21.497284+0000","last_fullsized":"2024-09-18T11:57:21.497284+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12820,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497365+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:21.497365+0000","last_peered":"2024-09-18T11:57:21.497365+0000","last_clean":"2024-09-18T11:57:21.497365+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:21.497365+0000","last_undegraded":"2024-09-18T11:57:21.497365+0000","last_fullsized":"2024-09-18T11:57:21.497365+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20572,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923956,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447884,"statfs":{"total":96636764160,"available":95690633216,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.255}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.29199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.27600000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.23200000000000001},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926272,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445568,"statfs":{"total":96636764160,"available":95688261632,"internally_reserved":0,"allocated":778240,"data_stored":533862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.252}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.219},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.28199999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915568,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456272,"statfs":{"total":96636764160,"available":95699222528,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.183}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.223}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:32.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:32.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13365, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.499874+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:21.499874+0000', 'last_peered': '2024-09-18T11:57:21.499874+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:21.499874+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:32.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:32.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13028, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.499894+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:21.499894+0000', 'last_peered': '2024-09-18T11:57:21.499894+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:21.499894+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:32.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-18T11:57:32.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12786", 'reported_seq': 16148, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:30.330048+0000', 'last_change': '2024-09-18T11:57:21.442179+0000', 'last_active': '2024-09-18T11:57:30.330048+0000', 'last_peered': '2024-09-18T11:57:30.330048+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.389641+0000', 'last_became_peered': '2024-09-18T11:44:47.389641+0000', 'last_unstale': '2024-09-18T11:57:30.330048+0000', 'last_undegraded': '2024-09-18T11:44:47.186195+0000', 'last_fullsized': '2024-09-18T11:44:47.118509+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 505, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 51144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1649, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -505}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:32.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:32.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 12720, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.499923+0000', 'last_change': '2024-09-18T11:44:47.474535+0000', 'last_active': '2024-09-18T11:57:21.499923+0000', 'last_peered': '2024-09-18T11:57:21.499923+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:57:21.499923+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:32.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:32.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13137, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.500416+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:21.500416+0000', 'last_peered': '2024-09-18T11:57:21.500416+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:21.500416+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:32.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:32.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13227, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.500272+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:57:21.500272+0000', 'last_peered': '2024-09-18T11:57:21.500272+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:21.500272+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:32.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:32.501 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:32.502 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10837/112 objects degraded (9675.893%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":850,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3773517824,"bytes_avail":382773538816,"bytes_total":386547056640,"degraded_objects":10837,"degraded_total":112,"degraded_ratio":96.758928571428569,"recovering_objects_per_sec":176,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1762,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:32.515 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:32.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:32.794 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:32.795 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:32.806 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2024-09-18T11:57:32.120035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20572,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3685076,"kb_used_data":2020,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802284,"statfs":{"total":386547056640,"available":382773538816,"internally_reserved":0,"allocated":2068480,"data_stored":1090968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001916"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499874+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:21.499874+0000","last_peered":"2024-09-18T11:57:21.499874+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:21.499874+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17360,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497538+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:21.497538+0000","last_peered":"2024-09-18T11:57:21.497538+0000","last_clean":"2024-09-18T11:57:21.497538+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:21.497538+0000","last_undegraded":"2024-09-18T11:57:21.497538+0000","last_fullsized":"2024-09-18T11:57:21.497538+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17034,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499651+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:21.499651+0000","last_peered":"2024-09-18T11:57:21.499651+0000","last_clean":"2024-09-18T11:57:21.499651+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:21.499651+0000","last_undegraded":"2024-09-18T11:57:21.499651+0000","last_fullsized":"2024-09-18T11:57:21.499651+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13035,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.573687+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:22.573687+0000","last_peered":"2024-09-18T11:57:22.573687+0000","last_clean":"2024-09-18T11:57:22.573687+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:22.573687+0000","last_undegraded":"2024-09-18T11:57:22.573687+0000","last_fullsized":"2024-09-18T11:57:22.573687+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13028,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499894+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:21.499894+0000","last_peered":"2024-09-18T11:57:21.499894+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:21.499894+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":16148,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:30.330048+0000","last_change":"2024-09-18T11:57:21.442179+0000","last_active":"2024-09-18T11:57:30.330048+0000","last_peered":"2024-09-18T11:57:30.330048+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:30.330048+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17384,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.501542+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:21.501542+0000","last_peered":"2024-09-18T11:57:21.501542+0000","last_clean":"2024-09-18T11:57:21.501542+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:21.501542+0000","last_undegraded":"2024-09-18T11:57:21.501542+0000","last_fullsized":"2024-09-18T11:57:21.501542+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17041,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497310+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:21.497310+0000","last_peered":"2024-09-18T11:57:21.497310+0000","last_clean":"2024-09-18T11:57:21.497310+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:21.497310+0000","last_undegraded":"2024-09-18T11:57:21.497310+0000","last_fullsized":"2024-09-18T11:57:21.497310+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17539,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499368+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:21.499368+0000","last_peered":"2024-09-18T11:57:21.499368+0000","last_clean":"2024-09-18T11:57:21.499368+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:21.499368+0000","last_undegraded":"2024-09-18T11:57:21.499368+0000","last_fullsized":"2024-09-18T11:57:21.499368+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12720,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499923+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:21.499923+0000","last_peered":"2024-09-18T11:57:21.499923+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:21.499923+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17252,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499889+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:21.499889+0000","last_peered":"2024-09-18T11:57:21.499889+0000","last_clean":"2024-09-18T11:57:21.499889+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:21.499889+0000","last_undegraded":"2024-09-18T11:57:21.499889+0000","last_fullsized":"2024-09-18T11:57:21.499889+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13137,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500416+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:21.500416+0000","last_peered":"2024-09-18T11:57:21.500416+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:21.500416+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13083,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497148+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:21.497148+0000","last_peered":"2024-09-18T11:57:21.497148+0000","last_clean":"2024-09-18T11:57:21.497148+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:21.497148+0000","last_undegraded":"2024-09-18T11:57:21.497148+0000","last_fullsized":"2024-09-18T11:57:21.497148+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13227,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500272+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:21.500272+0000","last_peered":"2024-09-18T11:57:21.500272+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:21.500272+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17204,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497284+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:21.497284+0000","last_peered":"2024-09-18T11:57:21.497284+0000","last_clean":"2024-09-18T11:57:21.497284+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:21.497284+0000","last_undegraded":"2024-09-18T11:57:21.497284+0000","last_fullsized":"2024-09-18T11:57:21.497284+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12820,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.497365+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:21.497365+0000","last_peered":"2024-09-18T11:57:21.497365+0000","last_clean":"2024-09-18T11:57:21.497365+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:21.497365+0000","last_undegraded":"2024-09-18T11:57:21.497365+0000","last_fullsized":"2024-09-18T11:57:21.497365+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20572,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923956,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447884,"statfs":{"total":96636764160,"available":95690633216,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.255}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.29199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.27600000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.23200000000000001},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926272,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445568,"statfs":{"total":96636764160,"available":95688261632,"internally_reserved":0,"allocated":778240,"data_stored":533862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.252}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.219},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.28199999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915568,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456272,"statfs":{"total":96636764160,"available":95699222528,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.183}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.223}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":352256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:35.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:36.124 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:36.124 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:36.137 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-18T11:57:34.200683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20572,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693388,"kb_used_data":2012,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793972,"statfs":{"total":386547056640,"available":382765027328,"internally_reserved":0,"allocated":2060288,"data_stored":1082776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":973,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.081801"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13365,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499874+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:21.499874+0000","last_peered":"2024-09-18T11:57:21.499874+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:21.499874+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17361,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500927+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:22.500927+0000","last_peered":"2024-09-18T11:57:22.500927+0000","last_clean":"2024-09-18T11:57:22.500927+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:22.500927+0000","last_undegraded":"2024-09-18T11:57:22.500927+0000","last_fullsized":"2024-09-18T11:57:22.500927+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17034,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499651+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:21.499651+0000","last_peered":"2024-09-18T11:57:21.499651+0000","last_clean":"2024-09-18T11:57:21.499651+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:21.499651+0000","last_undegraded":"2024-09-18T11:57:21.499651+0000","last_fullsized":"2024-09-18T11:57:21.499651+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13036,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.574553+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:22.574553+0000","last_peered":"2024-09-18T11:57:22.574553+0000","last_clean":"2024-09-18T11:57:22.574553+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:22.574553+0000","last_undegraded":"2024-09-18T11:57:22.574553+0000","last_fullsized":"2024-09-18T11:57:22.574553+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13028,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499894+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:21.499894+0000","last_peered":"2024-09-18T11:57:21.499894+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:21.499894+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":16148,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:30.330048+0000","last_change":"2024-09-18T11:57:21.442179+0000","last_active":"2024-09-18T11:57:30.330048+0000","last_peered":"2024-09-18T11:57:30.330048+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.389641+0000","last_became_peered":"2024-09-18T11:44:47.389641+0000","last_unstale":"2024-09-18T11:57:30.330048+0000","last_undegraded":"2024-09-18T11:44:47.186195+0000","last_fullsized":"2024-09-18T11:44:47.118509+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:23:05.545216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17384,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.501542+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:21.501542+0000","last_peered":"2024-09-18T11:57:21.501542+0000","last_clean":"2024-09-18T11:57:21.501542+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:21.501542+0000","last_undegraded":"2024-09-18T11:57:21.501542+0000","last_fullsized":"2024-09-18T11:57:21.501542+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17042,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500719+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:22.500719+0000","last_peered":"2024-09-18T11:57:22.500719+0000","last_clean":"2024-09-18T11:57:22.500719+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:22.500719+0000","last_undegraded":"2024-09-18T11:57:22.500719+0000","last_fullsized":"2024-09-18T11:57:22.500719+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17539,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499368+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:21.499368+0000","last_peered":"2024-09-18T11:57:21.499368+0000","last_clean":"2024-09-18T11:57:21.499368+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:21.499368+0000","last_undegraded":"2024-09-18T11:57:21.499368+0000","last_fullsized":"2024-09-18T11:57:21.499368+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":12720,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.499923+0000","last_change":"2024-09-18T11:44:47.474535+0000","last_active":"2024-09-18T11:57:21.499923+0000","last_peered":"2024-09-18T11:57:21.499923+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:21.499923+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17252,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.499889+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:21.499889+0000","last_peered":"2024-09-18T11:57:21.499889+0000","last_clean":"2024-09-18T11:57:21.499889+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:21.499889+0000","last_undegraded":"2024-09-18T11:57:21.499889+0000","last_fullsized":"2024-09-18T11:57:21.499889+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13137,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500416+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:21.500416+0000","last_peered":"2024-09-18T11:57:21.500416+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:21.500416+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13084,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.501172+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:22.501172+0000","last_peered":"2024-09-18T11:57:22.501172+0000","last_clean":"2024-09-18T11:57:22.501172+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:22.501172+0000","last_undegraded":"2024-09-18T11:57:22.501172+0000","last_fullsized":"2024-09-18T11:57:22.501172+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13227,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.500272+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:21.500272+0000","last_peered":"2024-09-18T11:57:21.500272+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:21.500272+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17205,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500694+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:22.500694+0000","last_peered":"2024-09-18T11:57:22.500694+0000","last_clean":"2024-09-18T11:57:22.500694+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:22.500694+0000","last_undegraded":"2024-09-18T11:57:22.500694+0000","last_fullsized":"2024-09-18T11:57:22.500694+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12821,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500997+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:22.500997+0000","last_peered":"2024-09-18T11:57:22.500997+0000","last_clean":"2024-09-18T11:57:22.500997+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:22.500997+0000","last_undegraded":"2024-09-18T11:57:22.500997+0000","last_fullsized":"2024-09-18T11:57:22.500997+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20572,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":356352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923956,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447884,"statfs":{"total":96636764160,"available":95690633216,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7389999999999999,"5min":7.2750000000000004,"15min":5.298},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.155},"max":{"1min":21.800999999999998,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":6.7309999999999999,"5min":6.7169999999999996,"15min":5.0579999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":21.968,"5min":22.969000000000001,"15min":31.675000000000001},"last":0.255}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:56:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8959999999999999,"5min":7.1989999999999998,"15min":4.9550000000000001},"min":{"1min":0.219,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":28.202999999999999,"5min":28.827000000000002,"15min":42.752000000000002},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.798,"15min":3.8450000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.20200000000000001},"max":{"1min":13.087999999999999,"5min":24.974,"15min":24.974},"last":0.29199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.27600000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:56:29 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":5.9420000000000002,"15min":4.3140000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":9.9749999999999996,"5min":22.73,"15min":22.73},"last":0.23200000000000001},{"interface":"front","average":{"1min":3.3420000000000001,"5min":5.8099999999999996,"15min":3.7890000000000001},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":13.493,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.26300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934584,"kb_used_data":752,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437256,"statfs":{"total":96636764160,"available":95679750144,"internally_reserved":0,"allocated":770048,"data_stored":525670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.185},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.23799999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.19900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915568,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456272,"statfs":{"total":96636764160,"available":95699222528,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.183}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.223}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":344064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:36.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:36.431 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:36.443 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-09-18T11:57:36.122399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21481,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693432,"kb_used_data":2060,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793928,"statfs":{"total":386547056640,"available":382764982272,"internally_reserved":0,"allocated":2109440,"data_stored":1105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1882,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003079"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13367,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195347+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:33.195347+0000","last_peered":"2024-09-18T11:57:33.195347+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:33.195347+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17361,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500927+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:22.500927+0000","last_peered":"2024-09-18T11:57:22.500927+0000","last_clean":"2024-09-18T11:57:22.500927+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:22.500927+0000","last_undegraded":"2024-09-18T11:57:22.500927+0000","last_fullsized":"2024-09-18T11:57:22.500927+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17036,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190564+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:33.190564+0000","last_peered":"2024-09-18T11:57:33.190564+0000","last_clean":"2024-09-18T11:57:33.190564+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:33.190564+0000","last_undegraded":"2024-09-18T11:57:33.190564+0000","last_fullsized":"2024-09-18T11:57:33.190564+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13036,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.574553+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:22.574553+0000","last_peered":"2024-09-18T11:57:22.574553+0000","last_clean":"2024-09-18T11:57:22.574553+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:22.574553+0000","last_undegraded":"2024-09-18T11:57:22.574553+0000","last_fullsized":"2024-09-18T11:57:22.574553+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13030,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195135+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:33.195135+0000","last_peered":"2024-09-18T11:57:33.195135+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:33.195135+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17172,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.281895+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:34.281895+0000","last_peered":"2024-09-18T11:57:34.281895+0000","last_clean":"2024-09-18T11:57:34.281895+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:34.281895+0000","last_undegraded":"2024-09-18T11:57:34.281895+0000","last_fullsized":"2024-09-18T11:57:34.281895+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17386,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190791+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:33.190791+0000","last_peered":"2024-09-18T11:57:33.190791+0000","last_clean":"2024-09-18T11:57:33.190791+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:33.190791+0000","last_undegraded":"2024-09-18T11:57:33.190791+0000","last_fullsized":"2024-09-18T11:57:33.190791+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17042,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500719+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:22.500719+0000","last_peered":"2024-09-18T11:57:22.500719+0000","last_clean":"2024-09-18T11:57:22.500719+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:22.500719+0000","last_undegraded":"2024-09-18T11:57:22.500719+0000","last_fullsized":"2024-09-18T11:57:22.500719+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17541,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.199985+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:33.199985+0000","last_peered":"2024-09-18T11:57:33.199985+0000","last_clean":"2024-09-18T11:57:33.199985+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:33.199985+0000","last_undegraded":"2024-09-18T11:57:33.199985+0000","last_fullsized":"2024-09-18T11:57:33.199985+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":13531,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:34.846347+0000","last_change":"2024-09-18T11:57:32.403893+0000","last_active":"2024-09-18T11:57:34.846347+0000","last_peered":"2024-09-18T11:57:34.846347+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:34.846347+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.195027+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:33.195027+0000","last_peered":"2024-09-18T11:57:33.195027+0000","last_clean":"2024-09-18T11:57:33.195027+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:33.195027+0000","last_undegraded":"2024-09-18T11:57:33.195027+0000","last_fullsized":"2024-09-18T11:57:33.195027+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13139,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195402+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:33.195402+0000","last_peered":"2024-09-18T11:57:33.195402+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:33.195402+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13084,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.501172+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:22.501172+0000","last_peered":"2024-09-18T11:57:22.501172+0000","last_clean":"2024-09-18T11:57:22.501172+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:22.501172+0000","last_undegraded":"2024-09-18T11:57:22.501172+0000","last_fullsized":"2024-09-18T11:57:22.501172+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13229,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195180+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:33.195180+0000","last_peered":"2024-09-18T11:57:33.195180+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:33.195180+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17205,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500694+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:22.500694+0000","last_peered":"2024-09-18T11:57:22.500694+0000","last_clean":"2024-09-18T11:57:22.500694+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:22.500694+0000","last_undegraded":"2024-09-18T11:57:22.500694+0000","last_fullsized":"2024-09-18T11:57:22.500694+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12821,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500997+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:22.500997+0000","last_peered":"2024-09-18T11:57:22.500997+0000","last_clean":"2024-09-18T11:57:22.500997+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:22.500997+0000","last_undegraded":"2024-09-18T11:57:22.500997+0000","last_fullsized":"2024-09-18T11:57:22.500997+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21481,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":356352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923968,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447872,"statfs":{"total":96636764160,"available":95690620928,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.219},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.26900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.19}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919296,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452544,"statfs":{"total":96636764160,"available":95695405056,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.184},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.27300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.22}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.25800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934584,"kb_used_data":752,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437256,"statfs":{"total":96636764160,"available":95679750144,"internally_reserved":0,"allocated":770048,"data_stored":525670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.185},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.23799999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.19900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915584,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456256,"statfs":{"total":96636764160,"available":95699206144,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.251},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.215}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28899999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":344064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:36.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:36.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13367, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195347+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:33.195347+0000', 'last_peered': '2024-09-18T11:57:33.195347+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:33.195347+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:36.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:36.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13030, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195135+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:33.195135+0000', 'last_peered': '2024-09-18T11:57:33.195135+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:33.195135+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:36.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:36.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 13531, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:34.846347+0000', 'last_change': '2024-09-18T11:57:32.403893+0000', 'last_active': '2024-09-18T11:57:34.846347+0000', 'last_peered': '2024-09-18T11:57:34.846347+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:57:34.846347+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 418, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1559}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:36.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:36.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13139, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195402+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:33.195402+0000', 'last_peered': '2024-09-18T11:57:33.195402+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:33.195402+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:36.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:36.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13229, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195180+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:57:33.195180+0000', 'last_peered': '2024-09-18T11:57:33.195180+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:33.195180+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:36.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:36.786 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9928/112 objects degraded (8864.286%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":854,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3782074368,"bytes_avail":382764982272,"bytes_total":386547056640,"degraded_objects":9928,"degraded_total":112,"degraded_ratio":88.642857142857139,"recovering_objects_per_sec":235,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1882,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:36.799 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:36.799 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:37.084 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:37.084 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:37.096 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-09-18T11:57:36.122399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21481,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693432,"kb_used_data":2060,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793928,"statfs":{"total":386547056640,"available":382764982272,"internally_reserved":0,"allocated":2109440,"data_stored":1105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1882,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003079"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13367,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195347+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:33.195347+0000","last_peered":"2024-09-18T11:57:33.195347+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:33.195347+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17361,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500927+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:22.500927+0000","last_peered":"2024-09-18T11:57:22.500927+0000","last_clean":"2024-09-18T11:57:22.500927+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:22.500927+0000","last_undegraded":"2024-09-18T11:57:22.500927+0000","last_fullsized":"2024-09-18T11:57:22.500927+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17036,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190564+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:33.190564+0000","last_peered":"2024-09-18T11:57:33.190564+0000","last_clean":"2024-09-18T11:57:33.190564+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:33.190564+0000","last_undegraded":"2024-09-18T11:57:33.190564+0000","last_fullsized":"2024-09-18T11:57:33.190564+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13036,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.574553+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:22.574553+0000","last_peered":"2024-09-18T11:57:22.574553+0000","last_clean":"2024-09-18T11:57:22.574553+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:22.574553+0000","last_undegraded":"2024-09-18T11:57:22.574553+0000","last_fullsized":"2024-09-18T11:57:22.574553+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13030,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195135+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:33.195135+0000","last_peered":"2024-09-18T11:57:33.195135+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:33.195135+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17172,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.281895+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:34.281895+0000","last_peered":"2024-09-18T11:57:34.281895+0000","last_clean":"2024-09-18T11:57:34.281895+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:34.281895+0000","last_undegraded":"2024-09-18T11:57:34.281895+0000","last_fullsized":"2024-09-18T11:57:34.281895+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17386,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190791+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:33.190791+0000","last_peered":"2024-09-18T11:57:33.190791+0000","last_clean":"2024-09-18T11:57:33.190791+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:33.190791+0000","last_undegraded":"2024-09-18T11:57:33.190791+0000","last_fullsized":"2024-09-18T11:57:33.190791+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17042,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500719+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:22.500719+0000","last_peered":"2024-09-18T11:57:22.500719+0000","last_clean":"2024-09-18T11:57:22.500719+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:22.500719+0000","last_undegraded":"2024-09-18T11:57:22.500719+0000","last_fullsized":"2024-09-18T11:57:22.500719+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17541,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.199985+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:33.199985+0000","last_peered":"2024-09-18T11:57:33.199985+0000","last_clean":"2024-09-18T11:57:33.199985+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:33.199985+0000","last_undegraded":"2024-09-18T11:57:33.199985+0000","last_fullsized":"2024-09-18T11:57:33.199985+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":13531,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:34.846347+0000","last_change":"2024-09-18T11:57:32.403893+0000","last_active":"2024-09-18T11:57:34.846347+0000","last_peered":"2024-09-18T11:57:34.846347+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:34.846347+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.195027+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:33.195027+0000","last_peered":"2024-09-18T11:57:33.195027+0000","last_clean":"2024-09-18T11:57:33.195027+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:33.195027+0000","last_undegraded":"2024-09-18T11:57:33.195027+0000","last_fullsized":"2024-09-18T11:57:33.195027+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13139,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195402+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:33.195402+0000","last_peered":"2024-09-18T11:57:33.195402+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:33.195402+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13084,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.501172+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:22.501172+0000","last_peered":"2024-09-18T11:57:22.501172+0000","last_clean":"2024-09-18T11:57:22.501172+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:22.501172+0000","last_undegraded":"2024-09-18T11:57:22.501172+0000","last_fullsized":"2024-09-18T11:57:22.501172+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13229,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195180+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:33.195180+0000","last_peered":"2024-09-18T11:57:33.195180+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:33.195180+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17205,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500694+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:22.500694+0000","last_peered":"2024-09-18T11:57:22.500694+0000","last_clean":"2024-09-18T11:57:22.500694+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:22.500694+0000","last_undegraded":"2024-09-18T11:57:22.500694+0000","last_fullsized":"2024-09-18T11:57:22.500694+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12821,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.500997+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:22.500997+0000","last_peered":"2024-09-18T11:57:22.500997+0000","last_clean":"2024-09-18T11:57:22.500997+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:22.500997+0000","last_undegraded":"2024-09-18T11:57:22.500997+0000","last_fullsized":"2024-09-18T11:57:22.500997+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21481,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":356352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923968,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447872,"statfs":{"total":96636764160,"available":95690620928,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.219},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.26900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.19}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919296,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452544,"statfs":{"total":96636764160,"available":95695405056,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.184},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.27300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.22}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.25800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934584,"kb_used_data":752,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437256,"statfs":{"total":96636764160,"available":95679750144,"internally_reserved":0,"allocated":770048,"data_stored":525670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.185},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.23799999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.19900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915584,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456256,"statfs":{"total":96636764160,"available":95699206144,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.251},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.215}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28899999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":344064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:40.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:40.392 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:40.392 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:40.404 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-18T11:57:40.123022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3685128,"kb_used_data":2076,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802232,"statfs":{"total":386547056640,"available":382773485568,"internally_reserved":0,"allocated":2125824,"data_stored":1112524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002986"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13367,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195347+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:33.195347+0000","last_peered":"2024-09-18T11:57:33.195347+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:33.195347+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17362,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201738+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:33.201738+0000","last_peered":"2024-09-18T11:57:33.201738+0000","last_clean":"2024-09-18T11:57:33.201738+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:33.201738+0000","last_undegraded":"2024-09-18T11:57:33.201738+0000","last_fullsized":"2024-09-18T11:57:33.201738+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17036,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190564+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:33.190564+0000","last_peered":"2024-09-18T11:57:33.190564+0000","last_clean":"2024-09-18T11:57:33.190564+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:33.190564+0000","last_undegraded":"2024-09-18T11:57:33.190564+0000","last_fullsized":"2024-09-18T11:57:33.190564+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13037,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201052+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:33.201052+0000","last_peered":"2024-09-18T11:57:33.201052+0000","last_clean":"2024-09-18T11:57:33.201052+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:33.201052+0000","last_undegraded":"2024-09-18T11:57:33.201052+0000","last_fullsized":"2024-09-18T11:57:33.201052+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13030,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195135+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:33.195135+0000","last_peered":"2024-09-18T11:57:33.195135+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:33.195135+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17172,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.281895+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:34.281895+0000","last_peered":"2024-09-18T11:57:34.281895+0000","last_clean":"2024-09-18T11:57:34.281895+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:34.281895+0000","last_undegraded":"2024-09-18T11:57:34.281895+0000","last_fullsized":"2024-09-18T11:57:34.281895+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17386,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190791+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:33.190791+0000","last_peered":"2024-09-18T11:57:33.190791+0000","last_clean":"2024-09-18T11:57:33.190791+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:33.190791+0000","last_undegraded":"2024-09-18T11:57:33.190791+0000","last_fullsized":"2024-09-18T11:57:33.190791+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17043,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:33.201653+0000","last_peered":"2024-09-18T11:57:33.201653+0000","last_clean":"2024-09-18T11:57:33.201653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:33.201653+0000","last_undegraded":"2024-09-18T11:57:33.201653+0000","last_fullsized":"2024-09-18T11:57:33.201653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17541,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.199985+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:33.199985+0000","last_peered":"2024-09-18T11:57:33.199985+0000","last_clean":"2024-09-18T11:57:33.199985+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:33.199985+0000","last_undegraded":"2024-09-18T11:57:33.199985+0000","last_fullsized":"2024-09-18T11:57:33.199985+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":15457,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:39.844875+0000","last_change":"2024-09-18T11:57:32.403893+0000","last_active":"2024-09-18T11:57:39.844875+0000","last_peered":"2024-09-18T11:57:39.844875+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:39.844875+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1381,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.195027+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:33.195027+0000","last_peered":"2024-09-18T11:57:33.195027+0000","last_clean":"2024-09-18T11:57:33.195027+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:33.195027+0000","last_undegraded":"2024-09-18T11:57:33.195027+0000","last_fullsized":"2024-09-18T11:57:33.195027+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13139,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195402+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:33.195402+0000","last_peered":"2024-09-18T11:57:33.195402+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:33.195402+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13085,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201177+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:33.201177+0000","last_peered":"2024-09-18T11:57:33.201177+0000","last_clean":"2024-09-18T11:57:33.201177+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:33.201177+0000","last_undegraded":"2024-09-18T11:57:33.201177+0000","last_fullsized":"2024-09-18T11:57:33.201177+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13229,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195180+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:33.195180+0000","last_peered":"2024-09-18T11:57:33.195180+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:33.195180+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17206,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:33.201637+0000","last_peered":"2024-09-18T11:57:33.201637+0000","last_clean":"2024-09-18T11:57:33.201637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:33.201637+0000","last_undegraded":"2024-09-18T11:57:33.201637+0000","last_fullsized":"2024-09-18T11:57:33.201637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12822,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201655+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:33.201655+0000","last_peered":"2024-09-18T11:57:33.201655+0000","last_clean":"2024-09-18T11:57:33.201655+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:33.201655+0000","last_undegraded":"2024-09-18T11:57:33.201655+0000","last_fullsized":"2024-09-18T11:57:33.201655+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":356352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923968,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447872,"statfs":{"total":96636764160,"available":95690620928,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.219},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.26900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.19}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919296,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452544,"statfs":{"total":96636764160,"available":95695405056,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.223},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.28499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926280,"kb_used_data":768,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445560,"statfs":{"total":96636764160,"available":95688253440,"internally_reserved":0,"allocated":786432,"data_stored":533107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.191}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.20999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.28399999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915584,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456256,"statfs":{"total":96636764160,"available":95699206144,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.251},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.215}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28899999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":344064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:40.406 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:40.679 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:40.680 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:40.692 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-18T11:57:40.123022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3685128,"kb_used_data":2076,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802232,"statfs":{"total":386547056640,"available":382773485568,"internally_reserved":0,"allocated":2125824,"data_stored":1112524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002986"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13367,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195347+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:33.195347+0000","last_peered":"2024-09-18T11:57:33.195347+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:33.195347+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17362,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201738+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:33.201738+0000","last_peered":"2024-09-18T11:57:33.201738+0000","last_clean":"2024-09-18T11:57:33.201738+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:33.201738+0000","last_undegraded":"2024-09-18T11:57:33.201738+0000","last_fullsized":"2024-09-18T11:57:33.201738+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17036,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190564+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:33.190564+0000","last_peered":"2024-09-18T11:57:33.190564+0000","last_clean":"2024-09-18T11:57:33.190564+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:33.190564+0000","last_undegraded":"2024-09-18T11:57:33.190564+0000","last_fullsized":"2024-09-18T11:57:33.190564+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13037,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201052+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:33.201052+0000","last_peered":"2024-09-18T11:57:33.201052+0000","last_clean":"2024-09-18T11:57:33.201052+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:33.201052+0000","last_undegraded":"2024-09-18T11:57:33.201052+0000","last_fullsized":"2024-09-18T11:57:33.201052+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13030,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195135+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:33.195135+0000","last_peered":"2024-09-18T11:57:33.195135+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:33.195135+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17172,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.281895+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:34.281895+0000","last_peered":"2024-09-18T11:57:34.281895+0000","last_clean":"2024-09-18T11:57:34.281895+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:34.281895+0000","last_undegraded":"2024-09-18T11:57:34.281895+0000","last_fullsized":"2024-09-18T11:57:34.281895+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17386,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190791+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:33.190791+0000","last_peered":"2024-09-18T11:57:33.190791+0000","last_clean":"2024-09-18T11:57:33.190791+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:33.190791+0000","last_undegraded":"2024-09-18T11:57:33.190791+0000","last_fullsized":"2024-09-18T11:57:33.190791+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17043,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:33.201653+0000","last_peered":"2024-09-18T11:57:33.201653+0000","last_clean":"2024-09-18T11:57:33.201653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:33.201653+0000","last_undegraded":"2024-09-18T11:57:33.201653+0000","last_fullsized":"2024-09-18T11:57:33.201653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17541,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.199985+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:33.199985+0000","last_peered":"2024-09-18T11:57:33.199985+0000","last_clean":"2024-09-18T11:57:33.199985+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:33.199985+0000","last_undegraded":"2024-09-18T11:57:33.199985+0000","last_fullsized":"2024-09-18T11:57:33.199985+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":15457,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:39.844875+0000","last_change":"2024-09-18T11:57:32.403893+0000","last_active":"2024-09-18T11:57:39.844875+0000","last_peered":"2024-09-18T11:57:39.844875+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:39.844875+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1381,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.195027+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:33.195027+0000","last_peered":"2024-09-18T11:57:33.195027+0000","last_clean":"2024-09-18T11:57:33.195027+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:33.195027+0000","last_undegraded":"2024-09-18T11:57:33.195027+0000","last_fullsized":"2024-09-18T11:57:33.195027+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13139,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195402+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:33.195402+0000","last_peered":"2024-09-18T11:57:33.195402+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:33.195402+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13085,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201177+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:33.201177+0000","last_peered":"2024-09-18T11:57:33.201177+0000","last_clean":"2024-09-18T11:57:33.201177+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:33.201177+0000","last_undegraded":"2024-09-18T11:57:33.201177+0000","last_fullsized":"2024-09-18T11:57:33.201177+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13229,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195180+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:33.195180+0000","last_peered":"2024-09-18T11:57:33.195180+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:33.195180+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17206,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:33.201637+0000","last_peered":"2024-09-18T11:57:33.201637+0000","last_clean":"2024-09-18T11:57:33.201637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:33.201637+0000","last_undegraded":"2024-09-18T11:57:33.201637+0000","last_fullsized":"2024-09-18T11:57:33.201637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12822,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201655+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:33.201655+0000","last_peered":"2024-09-18T11:57:33.201655+0000","last_clean":"2024-09-18T11:57:33.201655+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:33.201655+0000","last_undegraded":"2024-09-18T11:57:33.201655+0000","last_fullsized":"2024-09-18T11:57:33.201655+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":356352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923968,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447872,"statfs":{"total":96636764160,"available":95690620928,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.219},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.26900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.19}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919296,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452544,"statfs":{"total":96636764160,"available":95695405056,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.223},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.28499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926280,"kb_used_data":768,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445560,"statfs":{"total":96636764160,"available":95688253440,"internally_reserved":0,"allocated":786432,"data_stored":533107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.191}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.20999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.28399999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915584,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456256,"statfs":{"total":96636764160,"available":95699206144,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.251},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.215}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28899999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":344064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:40.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:40.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13367, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195347+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:33.195347+0000', 'last_peered': '2024-09-18T11:57:33.195347+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:33.195347+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:40.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:40.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13030, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195135+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:33.195135+0000', 'last_peered': '2024-09-18T11:57:33.195135+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:33.195135+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:40.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:40.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 15457, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:39.844875+0000', 'last_change': '2024-09-18T11:57:32.403893+0000', 'last_active': '2024-09-18T11:57:39.844875+0000', 'last_peered': '2024-09-18T11:57:39.844875+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:57:39.844875+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 598, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1381, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -596}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:40.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:40.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13139, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195402+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:33.195402+0000', 'last_peered': '2024-09-18T11:57:33.195402+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:33.195402+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:40.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:40.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13229, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195180+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:57:33.195180+0000', 'last_peered': '2024-09-18T11:57:33.195180+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:33.195180+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:40.694 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:41.107 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:41.107 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8965/112 objects degraded (8004.464%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":859,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3773571072,"bytes_avail":382773485568,"bytes_total":386547056640,"degraded_objects":8965,"degraded_total":112,"degraded_ratio":80.044642857142861,"recovering_objects_per_sec":233,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1872,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:41.119 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:41.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:41.401 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:41.401 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:41.413 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-18T11:57:40.123022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3685128,"kb_used_data":2076,"kb_used_omap":14,"kb_used_meta":3682993,"kb_avail":373802232,"statfs":{"total":386547056640,"available":382773485568,"internally_reserved":0,"allocated":2125824,"data_stored":1112524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771384859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002986"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13367,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195347+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:33.195347+0000","last_peered":"2024-09-18T11:57:33.195347+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:33.195347+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17362,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201738+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:33.201738+0000","last_peered":"2024-09-18T11:57:33.201738+0000","last_clean":"2024-09-18T11:57:33.201738+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:33.201738+0000","last_undegraded":"2024-09-18T11:57:33.201738+0000","last_fullsized":"2024-09-18T11:57:33.201738+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17036,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190564+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:33.190564+0000","last_peered":"2024-09-18T11:57:33.190564+0000","last_clean":"2024-09-18T11:57:33.190564+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:33.190564+0000","last_undegraded":"2024-09-18T11:57:33.190564+0000","last_fullsized":"2024-09-18T11:57:33.190564+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13037,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201052+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:33.201052+0000","last_peered":"2024-09-18T11:57:33.201052+0000","last_clean":"2024-09-18T11:57:33.201052+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:33.201052+0000","last_undegraded":"2024-09-18T11:57:33.201052+0000","last_fullsized":"2024-09-18T11:57:33.201052+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13030,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195135+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:33.195135+0000","last_peered":"2024-09-18T11:57:33.195135+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:33.195135+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17172,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.281895+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:34.281895+0000","last_peered":"2024-09-18T11:57:34.281895+0000","last_clean":"2024-09-18T11:57:34.281895+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:34.281895+0000","last_undegraded":"2024-09-18T11:57:34.281895+0000","last_fullsized":"2024-09-18T11:57:34.281895+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17386,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190791+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:33.190791+0000","last_peered":"2024-09-18T11:57:33.190791+0000","last_clean":"2024-09-18T11:57:33.190791+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:33.190791+0000","last_undegraded":"2024-09-18T11:57:33.190791+0000","last_fullsized":"2024-09-18T11:57:33.190791+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17043,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201653+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:33.201653+0000","last_peered":"2024-09-18T11:57:33.201653+0000","last_clean":"2024-09-18T11:57:33.201653+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:33.201653+0000","last_undegraded":"2024-09-18T11:57:33.201653+0000","last_fullsized":"2024-09-18T11:57:33.201653+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17541,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.199985+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:33.199985+0000","last_peered":"2024-09-18T11:57:33.199985+0000","last_clean":"2024-09-18T11:57:33.199985+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:33.199985+0000","last_undegraded":"2024-09-18T11:57:33.199985+0000","last_fullsized":"2024-09-18T11:57:33.199985+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":15457,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:39.844875+0000","last_change":"2024-09-18T11:57:32.403893+0000","last_active":"2024-09-18T11:57:39.844875+0000","last_peered":"2024-09-18T11:57:39.844875+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:39.844875+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1381,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.195027+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:33.195027+0000","last_peered":"2024-09-18T11:57:33.195027+0000","last_clean":"2024-09-18T11:57:33.195027+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:33.195027+0000","last_undegraded":"2024-09-18T11:57:33.195027+0000","last_fullsized":"2024-09-18T11:57:33.195027+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13139,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195402+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:33.195402+0000","last_peered":"2024-09-18T11:57:33.195402+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:33.195402+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13085,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201177+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:33.201177+0000","last_peered":"2024-09-18T11:57:33.201177+0000","last_clean":"2024-09-18T11:57:33.201177+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:33.201177+0000","last_undegraded":"2024-09-18T11:57:33.201177+0000","last_fullsized":"2024-09-18T11:57:33.201177+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13229,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195180+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:33.195180+0000","last_peered":"2024-09-18T11:57:33.195180+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:33.195180+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17206,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201637+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:33.201637+0000","last_peered":"2024-09-18T11:57:33.201637+0000","last_clean":"2024-09-18T11:57:33.201637+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:33.201637+0000","last_undegraded":"2024-09-18T11:57:33.201637+0000","last_fullsized":"2024-09-18T11:57:33.201637+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12822,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.201655+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:33.201655+0000","last_peered":"2024-09-18T11:57:33.201655+0000","last_clean":"2024-09-18T11:57:33.201655+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:33.201655+0000","last_undegraded":"2024-09-18T11:57:33.201655+0000","last_fullsized":"2024-09-18T11:57:33.201655+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":356352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923968,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447872,"statfs":{"total":96636764160,"available":95690620928,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.219},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.26900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.19}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919296,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452544,"statfs":{"total":96636764160,"available":95695405056,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.223},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.28499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926280,"kb_used_data":768,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445560,"statfs":{"total":96636764160,"available":95688253440,"internally_reserved":0,"allocated":786432,"data_stored":533107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.191}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.20999999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.28399999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915584,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456256,"statfs":{"total":96636764160,"available":95699206144,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.251},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.215}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28899999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":344064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:44.415 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:44.694 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:44.695 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:44.707 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-18T11:57:44.225015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693440,"kb_used_data":2068,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793920,"statfs":{"total":386547056640,"available":382764974080,"internally_reserved":0,"allocated":2117632,"data_stored":1104332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.102615"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13367,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195347+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:33.195347+0000","last_peered":"2024-09-18T11:57:33.195347+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:33.195347+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17363,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207265+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:34.207265+0000","last_peered":"2024-09-18T11:57:34.207265+0000","last_clean":"2024-09-18T11:57:34.207265+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:34.207265+0000","last_undegraded":"2024-09-18T11:57:34.207265+0000","last_fullsized":"2024-09-18T11:57:34.207265+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17036,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190564+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:33.190564+0000","last_peered":"2024-09-18T11:57:33.190564+0000","last_clean":"2024-09-18T11:57:33.190564+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:33.190564+0000","last_undegraded":"2024-09-18T11:57:33.190564+0000","last_fullsized":"2024-09-18T11:57:33.190564+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13038,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206939+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:34.206939+0000","last_peered":"2024-09-18T11:57:34.206939+0000","last_clean":"2024-09-18T11:57:34.206939+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:34.206939+0000","last_undegraded":"2024-09-18T11:57:34.206939+0000","last_fullsized":"2024-09-18T11:57:34.206939+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13030,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195135+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:33.195135+0000","last_peered":"2024-09-18T11:57:33.195135+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:33.195135+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17172,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.281895+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:34.281895+0000","last_peered":"2024-09-18T11:57:34.281895+0000","last_clean":"2024-09-18T11:57:34.281895+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:34.281895+0000","last_undegraded":"2024-09-18T11:57:34.281895+0000","last_fullsized":"2024-09-18T11:57:34.281895+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17386,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190791+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:33.190791+0000","last_peered":"2024-09-18T11:57:33.190791+0000","last_clean":"2024-09-18T11:57:33.190791+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:33.190791+0000","last_undegraded":"2024-09-18T11:57:33.190791+0000","last_fullsized":"2024-09-18T11:57:33.190791+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17044,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206969+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:34.206969+0000","last_peered":"2024-09-18T11:57:34.206969+0000","last_clean":"2024-09-18T11:57:34.206969+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:34.206969+0000","last_undegraded":"2024-09-18T11:57:34.206969+0000","last_fullsized":"2024-09-18T11:57:34.206969+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17541,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.199985+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:33.199985+0000","last_peered":"2024-09-18T11:57:33.199985+0000","last_clean":"2024-09-18T11:57:33.199985+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:33.199985+0000","last_undegraded":"2024-09-18T11:57:33.199985+0000","last_fullsized":"2024-09-18T11:57:33.199985+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":15457,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:39.844875+0000","last_change":"2024-09-18T11:57:32.403893+0000","last_active":"2024-09-18T11:57:39.844875+0000","last_peered":"2024-09-18T11:57:39.844875+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:39.844875+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1381,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.195027+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:33.195027+0000","last_peered":"2024-09-18T11:57:33.195027+0000","last_clean":"2024-09-18T11:57:33.195027+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:33.195027+0000","last_undegraded":"2024-09-18T11:57:33.195027+0000","last_fullsized":"2024-09-18T11:57:33.195027+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13139,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195402+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:33.195402+0000","last_peered":"2024-09-18T11:57:33.195402+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:33.195402+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207022+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:34.207022+0000","last_peered":"2024-09-18T11:57:34.207022+0000","last_clean":"2024-09-18T11:57:34.207022+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:34.207022+0000","last_undegraded":"2024-09-18T11:57:34.207022+0000","last_fullsized":"2024-09-18T11:57:34.207022+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13229,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195180+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:33.195180+0000","last_peered":"2024-09-18T11:57:33.195180+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:33.195180+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17207,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206973+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:34.206973+0000","last_peered":"2024-09-18T11:57:34.206973+0000","last_clean":"2024-09-18T11:57:34.206973+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:34.206973+0000","last_undegraded":"2024-09-18T11:57:34.206973+0000","last_fullsized":"2024-09-18T11:57:34.206973+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12823,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207045+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:34.207045+0000","last_peered":"2024-09-18T11:57:34.207045+0000","last_clean":"2024-09-18T11:57:34.207045+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:34.207045+0000","last_undegraded":"2024-09-18T11:57:34.207045+0000","last_fullsized":"2024-09-18T11:57:34.207045+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923968,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447872,"statfs":{"total":96636764160,"available":95690620928,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.224}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.28100000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.254},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919296,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452544,"statfs":{"total":96636764160,"available":95695405056,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.223},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.28499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934592,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437248,"statfs":{"total":96636764160,"available":95679741952,"internally_reserved":0,"allocated":778240,"data_stored":524915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.19600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.22800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.311}]}]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915584,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456256,"statfs":{"total":96636764160,"available":95699206144,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.36699999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:44.708 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:44.985 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:44.985 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:44.997 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-18T11:57:44.225015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693440,"kb_used_data":2068,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793920,"statfs":{"total":386547056640,"available":382764974080,"internally_reserved":0,"allocated":2117632,"data_stored":1104332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.102615"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13367,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195347+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:33.195347+0000","last_peered":"2024-09-18T11:57:33.195347+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:33.195347+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17363,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207265+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:34.207265+0000","last_peered":"2024-09-18T11:57:34.207265+0000","last_clean":"2024-09-18T11:57:34.207265+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:34.207265+0000","last_undegraded":"2024-09-18T11:57:34.207265+0000","last_fullsized":"2024-09-18T11:57:34.207265+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17036,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190564+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:33.190564+0000","last_peered":"2024-09-18T11:57:33.190564+0000","last_clean":"2024-09-18T11:57:33.190564+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:33.190564+0000","last_undegraded":"2024-09-18T11:57:33.190564+0000","last_fullsized":"2024-09-18T11:57:33.190564+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13038,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206939+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:34.206939+0000","last_peered":"2024-09-18T11:57:34.206939+0000","last_clean":"2024-09-18T11:57:34.206939+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:34.206939+0000","last_undegraded":"2024-09-18T11:57:34.206939+0000","last_fullsized":"2024-09-18T11:57:34.206939+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13030,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195135+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:33.195135+0000","last_peered":"2024-09-18T11:57:33.195135+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:33.195135+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17172,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.281895+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:34.281895+0000","last_peered":"2024-09-18T11:57:34.281895+0000","last_clean":"2024-09-18T11:57:34.281895+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:34.281895+0000","last_undegraded":"2024-09-18T11:57:34.281895+0000","last_fullsized":"2024-09-18T11:57:34.281895+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17386,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190791+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:33.190791+0000","last_peered":"2024-09-18T11:57:33.190791+0000","last_clean":"2024-09-18T11:57:33.190791+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:33.190791+0000","last_undegraded":"2024-09-18T11:57:33.190791+0000","last_fullsized":"2024-09-18T11:57:33.190791+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17044,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206969+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:34.206969+0000","last_peered":"2024-09-18T11:57:34.206969+0000","last_clean":"2024-09-18T11:57:34.206969+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:34.206969+0000","last_undegraded":"2024-09-18T11:57:34.206969+0000","last_fullsized":"2024-09-18T11:57:34.206969+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17541,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.199985+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:33.199985+0000","last_peered":"2024-09-18T11:57:33.199985+0000","last_clean":"2024-09-18T11:57:33.199985+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:33.199985+0000","last_undegraded":"2024-09-18T11:57:33.199985+0000","last_fullsized":"2024-09-18T11:57:33.199985+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":15457,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:39.844875+0000","last_change":"2024-09-18T11:57:32.403893+0000","last_active":"2024-09-18T11:57:39.844875+0000","last_peered":"2024-09-18T11:57:39.844875+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:39.844875+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1381,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.195027+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:33.195027+0000","last_peered":"2024-09-18T11:57:33.195027+0000","last_clean":"2024-09-18T11:57:33.195027+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:33.195027+0000","last_undegraded":"2024-09-18T11:57:33.195027+0000","last_fullsized":"2024-09-18T11:57:33.195027+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13139,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195402+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:33.195402+0000","last_peered":"2024-09-18T11:57:33.195402+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:33.195402+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207022+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:34.207022+0000","last_peered":"2024-09-18T11:57:34.207022+0000","last_clean":"2024-09-18T11:57:34.207022+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:34.207022+0000","last_undegraded":"2024-09-18T11:57:34.207022+0000","last_fullsized":"2024-09-18T11:57:34.207022+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13229,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195180+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:33.195180+0000","last_peered":"2024-09-18T11:57:33.195180+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:33.195180+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17207,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206973+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:34.206973+0000","last_peered":"2024-09-18T11:57:34.206973+0000","last_clean":"2024-09-18T11:57:34.206973+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:34.206973+0000","last_undegraded":"2024-09-18T11:57:34.206973+0000","last_fullsized":"2024-09-18T11:57:34.206973+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12823,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207045+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:34.207045+0000","last_peered":"2024-09-18T11:57:34.207045+0000","last_clean":"2024-09-18T11:57:34.207045+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:34.207045+0000","last_undegraded":"2024-09-18T11:57:34.207045+0000","last_fullsized":"2024-09-18T11:57:34.207045+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923968,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447872,"statfs":{"total":96636764160,"available":95690620928,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.224}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.28100000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.254},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919296,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452544,"statfs":{"total":96636764160,"available":95695405056,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.223},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.28499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934592,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437248,"statfs":{"total":96636764160,"available":95679741952,"internally_reserved":0,"allocated":778240,"data_stored":524915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.19600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.22800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.311}]}]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915584,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456256,"statfs":{"total":96636764160,"available":95699206144,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.36699999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:44.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:44.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13367, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195347+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:33.195347+0000', 'last_peered': '2024-09-18T11:57:33.195347+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:33.195347+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:44.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:44.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13030, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195135+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:33.195135+0000', 'last_peered': '2024-09-18T11:57:33.195135+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:33.195135+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:44.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-18T11:57:44.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12592", 'reported_seq': 15457, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:39.844875+0000', 'last_change': '2024-09-18T11:57:32.403893+0000', 'last_active': '2024-09-18T11:57:39.844875+0000', 'last_peered': '2024-09-18T11:57:39.844875+0000', 'last_clean': '2024-09-18T11:43:39.896963+0000', 'last_became_active': '2024-09-18T11:44:47.462638+0000', 'last_became_peered': '2024-09-18T11:44:47.462638+0000', 'last_unstale': '2024-09-18T11:57:39.844875+0000', 'last_undegraded': '2024-09-18T11:44:47.161105+0000', 'last_fullsized': '2024-09-18T11:44:47.122722+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 598, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12590, 'num_write_kb': 50366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1381, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -596}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:44.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:44.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13139, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195402+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:33.195402+0000', 'last_peered': '2024-09-18T11:57:33.195402+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:33.195402+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:44.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:44.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 13229, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:33.195180+0000', 'last_change': '2024-09-18T11:44:47.468342+0000', 'last_active': '2024-09-18T11:57:33.195180+0000', 'last_peered': '2024-09-18T11:57:33.195180+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:33.195180+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:45.000 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:45.366 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:45.367 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8965/112 objects degraded (8004.464%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":863,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3782082560,"bytes_avail":382764974080,"bytes_total":386547056640,"degraded_objects":8965,"degraded_total":112,"degraded_ratio":80.044642857142861,"recovering_objects_per_sec":188,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1872,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:45.380 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:45.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:45.683 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:45.684 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:45.696 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-18T11:57:44.225015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693440,"kb_used_data":2068,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793920,"statfs":{"total":386547056640,"available":382764974080,"internally_reserved":0,"allocated":2117632,"data_stored":1104332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.102615"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13367,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195347+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:33.195347+0000","last_peered":"2024-09-18T11:57:33.195347+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:33.195347+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17363,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207265+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:34.207265+0000","last_peered":"2024-09-18T11:57:34.207265+0000","last_clean":"2024-09-18T11:57:34.207265+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:34.207265+0000","last_undegraded":"2024-09-18T11:57:34.207265+0000","last_fullsized":"2024-09-18T11:57:34.207265+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17036,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190564+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:33.190564+0000","last_peered":"2024-09-18T11:57:33.190564+0000","last_clean":"2024-09-18T11:57:33.190564+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:33.190564+0000","last_undegraded":"2024-09-18T11:57:33.190564+0000","last_fullsized":"2024-09-18T11:57:33.190564+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13038,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206939+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:34.206939+0000","last_peered":"2024-09-18T11:57:34.206939+0000","last_clean":"2024-09-18T11:57:34.206939+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:34.206939+0000","last_undegraded":"2024-09-18T11:57:34.206939+0000","last_fullsized":"2024-09-18T11:57:34.206939+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13030,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195135+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:33.195135+0000","last_peered":"2024-09-18T11:57:33.195135+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:33.195135+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17172,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.281895+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:34.281895+0000","last_peered":"2024-09-18T11:57:34.281895+0000","last_clean":"2024-09-18T11:57:34.281895+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:34.281895+0000","last_undegraded":"2024-09-18T11:57:34.281895+0000","last_fullsized":"2024-09-18T11:57:34.281895+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17386,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.190791+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:33.190791+0000","last_peered":"2024-09-18T11:57:33.190791+0000","last_clean":"2024-09-18T11:57:33.190791+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:33.190791+0000","last_undegraded":"2024-09-18T11:57:33.190791+0000","last_fullsized":"2024-09-18T11:57:33.190791+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17044,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206969+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:34.206969+0000","last_peered":"2024-09-18T11:57:34.206969+0000","last_clean":"2024-09-18T11:57:34.206969+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:34.206969+0000","last_undegraded":"2024-09-18T11:57:34.206969+0000","last_fullsized":"2024-09-18T11:57:34.206969+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17541,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.199985+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:33.199985+0000","last_peered":"2024-09-18T11:57:33.199985+0000","last_clean":"2024-09-18T11:57:33.199985+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:33.199985+0000","last_undegraded":"2024-09-18T11:57:33.199985+0000","last_fullsized":"2024-09-18T11:57:33.199985+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":15457,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:39.844875+0000","last_change":"2024-09-18T11:57:32.403893+0000","last_active":"2024-09-18T11:57:39.844875+0000","last_peered":"2024-09-18T11:57:39.844875+0000","last_clean":"2024-09-18T11:43:39.896963+0000","last_became_active":"2024-09-18T11:44:47.462638+0000","last_became_peered":"2024-09-18T11:44:47.462638+0000","last_unstale":"2024-09-18T11:57:39.844875+0000","last_undegraded":"2024-09-18T11:44:47.161105+0000","last_fullsized":"2024-09-18T11:44:47.122722+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1381,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T11:57:33.195027+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:33.195027+0000","last_peered":"2024-09-18T11:57:33.195027+0000","last_clean":"2024-09-18T11:57:33.195027+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:33.195027+0000","last_undegraded":"2024-09-18T11:57:33.195027+0000","last_fullsized":"2024-09-18T11:57:33.195027+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13139,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195402+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:33.195402+0000","last_peered":"2024-09-18T11:57:33.195402+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:33.195402+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207022+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:34.207022+0000","last_peered":"2024-09-18T11:57:34.207022+0000","last_clean":"2024-09-18T11:57:34.207022+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:34.207022+0000","last_undegraded":"2024-09-18T11:57:34.207022+0000","last_fullsized":"2024-09-18T11:57:34.207022+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13229,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:33.195180+0000","last_change":"2024-09-18T11:44:47.468342+0000","last_active":"2024-09-18T11:57:33.195180+0000","last_peered":"2024-09-18T11:57:33.195180+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:33.195180+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17207,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206973+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:34.206973+0000","last_peered":"2024-09-18T11:57:34.206973+0000","last_clean":"2024-09-18T11:57:34.206973+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:34.206973+0000","last_undegraded":"2024-09-18T11:57:34.206973+0000","last_fullsized":"2024-09-18T11:57:34.206973+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12823,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207045+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:34.207045+0000","last_peered":"2024-09-18T11:57:34.207045+0000","last_clean":"2024-09-18T11:57:34.207045+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:34.207045+0000","last_undegraded":"2024-09-18T11:57:34.207045+0000","last_fullsized":"2024-09-18T11:57:34.207045+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22444,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923968,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447872,"statfs":{"total":96636764160,"available":95690620928,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.224}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.28100000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.254},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919296,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452544,"statfs":{"total":96636764160,"available":95695405056,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.223},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.28499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934592,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437248,"statfs":{"total":96636764160,"available":95679741952,"internally_reserved":0,"allocated":778240,"data_stored":524915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.19600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.22800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.311}]}]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915584,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456256,"statfs":{"total":96636764160,"available":95699206144,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.36699999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:48.700 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:48.982 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:48.983 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:48.995 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-18T11:57:48.124457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23476,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693500,"kb_used_data":2116,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793860,"statfs":{"total":386547056640,"available":382764912640,"internally_reserved":0,"allocated":2166784,"data_stored":1126421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001434"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13369,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225976+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:43.225976+0000","last_peered":"2024-09-18T11:57:43.225976+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:43.225976+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17363,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207265+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:34.207265+0000","last_peered":"2024-09-18T11:57:34.207265+0000","last_clean":"2024-09-18T11:57:34.207265+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:34.207265+0000","last_undegraded":"2024-09-18T11:57:34.207265+0000","last_fullsized":"2024-09-18T11:57:34.207265+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17038,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.218506+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:43.218506+0000","last_peered":"2024-09-18T11:57:43.218506+0000","last_clean":"2024-09-18T11:57:43.218506+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:43.218506+0000","last_undegraded":"2024-09-18T11:57:43.218506+0000","last_fullsized":"2024-09-18T11:57:43.218506+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13038,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206939+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:34.206939+0000","last_peered":"2024-09-18T11:57:34.206939+0000","last_clean":"2024-09-18T11:57:34.206939+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:34.206939+0000","last_undegraded":"2024-09-18T11:57:34.206939+0000","last_fullsized":"2024-09-18T11:57:34.206939+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13032,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225264+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:43.225264+0000","last_peered":"2024-09-18T11:57:43.225264+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:43.225264+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17174,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225338+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:43.225338+0000","last_peered":"2024-09-18T11:57:43.225338+0000","last_clean":"2024-09-18T11:57:43.225338+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:43.225338+0000","last_undegraded":"2024-09-18T11:57:43.225338+0000","last_fullsized":"2024-09-18T11:57:43.225338+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17388,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.224741+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:43.224741+0000","last_peered":"2024-09-18T11:57:43.224741+0000","last_clean":"2024-09-18T11:57:43.224741+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:43.224741+0000","last_undegraded":"2024-09-18T11:57:43.224741+0000","last_fullsized":"2024-09-18T11:57:43.224741+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17044,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206969+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:34.206969+0000","last_peered":"2024-09-18T11:57:34.206969+0000","last_clean":"2024-09-18T11:57:34.206969+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:34.206969+0000","last_undegraded":"2024-09-18T11:57:34.206969+0000","last_fullsized":"2024-09-18T11:57:34.206969+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17543,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225076+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:43.225076+0000","last_peered":"2024-09-18T11:57:43.225076+0000","last_clean":"2024-09-18T11:57:43.225076+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:43.225076+0000","last_undegraded":"2024-09-18T11:57:43.225076+0000","last_fullsized":"2024-09-18T11:57:43.225076+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16667,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.298755+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:44.298755+0000","last_peered":"2024-09-18T11:57:44.298755+0000","last_clean":"2024-09-18T11:57:44.298755+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:44.298755+0000","last_undegraded":"2024-09-18T11:57:44.298755+0000","last_fullsized":"2024-09-18T11:57:44.298755+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17256,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225417+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:43.225417+0000","last_peered":"2024-09-18T11:57:43.225417+0000","last_clean":"2024-09-18T11:57:43.225417+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:43.225417+0000","last_undegraded":"2024-09-18T11:57:43.225417+0000","last_fullsized":"2024-09-18T11:57:43.225417+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13141,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225951+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:43.225951+0000","last_peered":"2024-09-18T11:57:43.225951+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:43.225951+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207022+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:34.207022+0000","last_peered":"2024-09-18T11:57:34.207022+0000","last_clean":"2024-09-18T11:57:34.207022+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:34.207022+0000","last_undegraded":"2024-09-18T11:57:34.207022+0000","last_fullsized":"2024-09-18T11:57:34.207022+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":14100,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:45.331774+0000","last_change":"2024-09-18T11:57:42.502387+0000","last_active":"2024-09-18T11:57:45.331774+0000","last_peered":"2024-09-18T11:57:45.331774+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:45.331774+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":471,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1689}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17207,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206973+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:34.206973+0000","last_peered":"2024-09-18T11:57:34.206973+0000","last_clean":"2024-09-18T11:57:34.206973+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:34.206973+0000","last_undegraded":"2024-09-18T11:57:34.206973+0000","last_fullsized":"2024-09-18T11:57:34.206973+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12823,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207045+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:34.207045+0000","last_peered":"2024-09-18T11:57:34.207045+0000","last_clean":"2024-09-18T11:57:34.207045+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:34.207045+0000","last_undegraded":"2024-09-18T11:57:34.207045+0000","last_fullsized":"2024-09-18T11:57:34.207045+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23476,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923988,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447852,"statfs":{"total":96636764160,"available":95690600448,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.187},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.25600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.223},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.308}]}]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919316,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452524,"statfs":{"total":96636764160,"available":95695384576,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29799999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.33100000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934592,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437248,"statfs":{"total":96636764160,"available":95679741952,"internally_reserved":0,"allocated":778240,"data_stored":524915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.19600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.22800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.311}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915604,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456236,"statfs":{"total":96636764160,"available":95699185664,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.247}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.29599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:48.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:49.320 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:49.321 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:49.333 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-18T11:57:48.124457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23476,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693500,"kb_used_data":2116,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793860,"statfs":{"total":386547056640,"available":382764912640,"internally_reserved":0,"allocated":2166784,"data_stored":1126421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001434"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13369,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225976+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:43.225976+0000","last_peered":"2024-09-18T11:57:43.225976+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:43.225976+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17363,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207265+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:34.207265+0000","last_peered":"2024-09-18T11:57:34.207265+0000","last_clean":"2024-09-18T11:57:34.207265+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:34.207265+0000","last_undegraded":"2024-09-18T11:57:34.207265+0000","last_fullsized":"2024-09-18T11:57:34.207265+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17038,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.218506+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:43.218506+0000","last_peered":"2024-09-18T11:57:43.218506+0000","last_clean":"2024-09-18T11:57:43.218506+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:43.218506+0000","last_undegraded":"2024-09-18T11:57:43.218506+0000","last_fullsized":"2024-09-18T11:57:43.218506+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13038,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206939+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:34.206939+0000","last_peered":"2024-09-18T11:57:34.206939+0000","last_clean":"2024-09-18T11:57:34.206939+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:34.206939+0000","last_undegraded":"2024-09-18T11:57:34.206939+0000","last_fullsized":"2024-09-18T11:57:34.206939+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13032,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225264+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:43.225264+0000","last_peered":"2024-09-18T11:57:43.225264+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:43.225264+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17174,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225338+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:43.225338+0000","last_peered":"2024-09-18T11:57:43.225338+0000","last_clean":"2024-09-18T11:57:43.225338+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:43.225338+0000","last_undegraded":"2024-09-18T11:57:43.225338+0000","last_fullsized":"2024-09-18T11:57:43.225338+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17388,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.224741+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:43.224741+0000","last_peered":"2024-09-18T11:57:43.224741+0000","last_clean":"2024-09-18T11:57:43.224741+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:43.224741+0000","last_undegraded":"2024-09-18T11:57:43.224741+0000","last_fullsized":"2024-09-18T11:57:43.224741+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17044,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206969+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:34.206969+0000","last_peered":"2024-09-18T11:57:34.206969+0000","last_clean":"2024-09-18T11:57:34.206969+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:34.206969+0000","last_undegraded":"2024-09-18T11:57:34.206969+0000","last_fullsized":"2024-09-18T11:57:34.206969+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17543,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225076+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:43.225076+0000","last_peered":"2024-09-18T11:57:43.225076+0000","last_clean":"2024-09-18T11:57:43.225076+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:43.225076+0000","last_undegraded":"2024-09-18T11:57:43.225076+0000","last_fullsized":"2024-09-18T11:57:43.225076+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16667,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.298755+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:44.298755+0000","last_peered":"2024-09-18T11:57:44.298755+0000","last_clean":"2024-09-18T11:57:44.298755+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:44.298755+0000","last_undegraded":"2024-09-18T11:57:44.298755+0000","last_fullsized":"2024-09-18T11:57:44.298755+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17256,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225417+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:43.225417+0000","last_peered":"2024-09-18T11:57:43.225417+0000","last_clean":"2024-09-18T11:57:43.225417+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:43.225417+0000","last_undegraded":"2024-09-18T11:57:43.225417+0000","last_fullsized":"2024-09-18T11:57:43.225417+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13141,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225951+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:43.225951+0000","last_peered":"2024-09-18T11:57:43.225951+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:43.225951+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207022+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:34.207022+0000","last_peered":"2024-09-18T11:57:34.207022+0000","last_clean":"2024-09-18T11:57:34.207022+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:34.207022+0000","last_undegraded":"2024-09-18T11:57:34.207022+0000","last_fullsized":"2024-09-18T11:57:34.207022+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":14100,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:45.331774+0000","last_change":"2024-09-18T11:57:42.502387+0000","last_active":"2024-09-18T11:57:45.331774+0000","last_peered":"2024-09-18T11:57:45.331774+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:45.331774+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":471,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1689}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17207,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206973+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:34.206973+0000","last_peered":"2024-09-18T11:57:34.206973+0000","last_clean":"2024-09-18T11:57:34.206973+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:34.206973+0000","last_undegraded":"2024-09-18T11:57:34.206973+0000","last_fullsized":"2024-09-18T11:57:34.206973+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12823,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207045+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:34.207045+0000","last_peered":"2024-09-18T11:57:34.207045+0000","last_clean":"2024-09-18T11:57:34.207045+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:34.207045+0000","last_undegraded":"2024-09-18T11:57:34.207045+0000","last_fullsized":"2024-09-18T11:57:34.207045+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23476,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923988,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447852,"statfs":{"total":96636764160,"available":95690600448,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.187},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.25600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.223},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.308}]}]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919316,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452524,"statfs":{"total":96636764160,"available":95695384576,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29799999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.33100000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934592,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437248,"statfs":{"total":96636764160,"available":95679741952,"internally_reserved":0,"allocated":778240,"data_stored":524915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.19600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.22800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.311}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915604,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456236,"statfs":{"total":96636764160,"available":95699185664,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.247}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.29599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:49.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:49.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13369, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:43.225976+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:43.225976+0000', 'last_peered': '2024-09-18T11:57:43.225976+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:43.225976+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:49.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:49.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13032, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:43.225264+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:43.225264+0000', 'last_peered': '2024-09-18T11:57:43.225264+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:43.225264+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:49.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:49.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13141, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:43.225951+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:43.225951+0000', 'last_peered': '2024-09-18T11:57:43.225951+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:43.225951+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:49.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:49.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 14100, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:45.331774+0000', 'last_change': '2024-09-18T11:57:42.502387+0000', 'last_active': '2024-09-18T11:57:45.331774+0000', 'last_peered': '2024-09-18T11:57:45.331774+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:45.331774+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1691, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 471, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1689}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:49.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:49.688 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:49.689 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7933/112 objects degraded (7083.036%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":867,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3782144000,"bytes_avail":382764912640,"bytes_total":386547056640,"degraded_objects":7933,"degraded_total":112,"degraded_ratio":70.830357142857139,"recovering_objects_per_sec":128,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1032,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:49.701 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:49.702 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:49.982 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:49.982 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:49.994 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-18T11:57:48.124457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23476,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693500,"kb_used_data":2116,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793860,"statfs":{"total":386547056640,"available":382764912640,"internally_reserved":0,"allocated":2166784,"data_stored":1126421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001434"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13369,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225976+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:43.225976+0000","last_peered":"2024-09-18T11:57:43.225976+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:43.225976+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17363,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207265+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:34.207265+0000","last_peered":"2024-09-18T11:57:34.207265+0000","last_clean":"2024-09-18T11:57:34.207265+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:34.207265+0000","last_undegraded":"2024-09-18T11:57:34.207265+0000","last_fullsized":"2024-09-18T11:57:34.207265+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17038,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.218506+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:43.218506+0000","last_peered":"2024-09-18T11:57:43.218506+0000","last_clean":"2024-09-18T11:57:43.218506+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:43.218506+0000","last_undegraded":"2024-09-18T11:57:43.218506+0000","last_fullsized":"2024-09-18T11:57:43.218506+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13038,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206939+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:34.206939+0000","last_peered":"2024-09-18T11:57:34.206939+0000","last_clean":"2024-09-18T11:57:34.206939+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:34.206939+0000","last_undegraded":"2024-09-18T11:57:34.206939+0000","last_fullsized":"2024-09-18T11:57:34.206939+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13032,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225264+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:43.225264+0000","last_peered":"2024-09-18T11:57:43.225264+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:43.225264+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17174,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225338+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:43.225338+0000","last_peered":"2024-09-18T11:57:43.225338+0000","last_clean":"2024-09-18T11:57:43.225338+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:43.225338+0000","last_undegraded":"2024-09-18T11:57:43.225338+0000","last_fullsized":"2024-09-18T11:57:43.225338+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17388,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.224741+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:43.224741+0000","last_peered":"2024-09-18T11:57:43.224741+0000","last_clean":"2024-09-18T11:57:43.224741+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:43.224741+0000","last_undegraded":"2024-09-18T11:57:43.224741+0000","last_fullsized":"2024-09-18T11:57:43.224741+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17044,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206969+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:34.206969+0000","last_peered":"2024-09-18T11:57:34.206969+0000","last_clean":"2024-09-18T11:57:34.206969+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:34.206969+0000","last_undegraded":"2024-09-18T11:57:34.206969+0000","last_fullsized":"2024-09-18T11:57:34.206969+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17543,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225076+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:43.225076+0000","last_peered":"2024-09-18T11:57:43.225076+0000","last_clean":"2024-09-18T11:57:43.225076+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:43.225076+0000","last_undegraded":"2024-09-18T11:57:43.225076+0000","last_fullsized":"2024-09-18T11:57:43.225076+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16667,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.298755+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:44.298755+0000","last_peered":"2024-09-18T11:57:44.298755+0000","last_clean":"2024-09-18T11:57:44.298755+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:44.298755+0000","last_undegraded":"2024-09-18T11:57:44.298755+0000","last_fullsized":"2024-09-18T11:57:44.298755+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17256,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225417+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:43.225417+0000","last_peered":"2024-09-18T11:57:43.225417+0000","last_clean":"2024-09-18T11:57:43.225417+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:43.225417+0000","last_undegraded":"2024-09-18T11:57:43.225417+0000","last_fullsized":"2024-09-18T11:57:43.225417+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13141,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225951+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:43.225951+0000","last_peered":"2024-09-18T11:57:43.225951+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:43.225951+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207022+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:34.207022+0000","last_peered":"2024-09-18T11:57:34.207022+0000","last_clean":"2024-09-18T11:57:34.207022+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:34.207022+0000","last_undegraded":"2024-09-18T11:57:34.207022+0000","last_fullsized":"2024-09-18T11:57:34.207022+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":14100,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:45.331774+0000","last_change":"2024-09-18T11:57:42.502387+0000","last_active":"2024-09-18T11:57:45.331774+0000","last_peered":"2024-09-18T11:57:45.331774+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:45.331774+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":471,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1689}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17207,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.206973+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:34.206973+0000","last_peered":"2024-09-18T11:57:34.206973+0000","last_clean":"2024-09-18T11:57:34.206973+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:34.206973+0000","last_undegraded":"2024-09-18T11:57:34.206973+0000","last_fullsized":"2024-09-18T11:57:34.206973+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12823,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T11:57:34.207045+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:34.207045+0000","last_peered":"2024-09-18T11:57:34.207045+0000","last_clean":"2024-09-18T11:57:34.207045+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:34.207045+0000","last_undegraded":"2024-09-18T11:57:34.207045+0000","last_fullsized":"2024-09-18T11:57:34.207045+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23476,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923988,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447852,"statfs":{"total":96636764160,"available":95690600448,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.187},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.25600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.223},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.308}]}]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919316,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452524,"statfs":{"total":96636764160,"available":95695384576,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29799999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26400000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.33100000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934592,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437248,"statfs":{"total":96636764160,"available":95679741952,"internally_reserved":0,"allocated":778240,"data_stored":524915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.19600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.22800000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.311}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915604,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456236,"statfs":{"total":96636764160,"available":95699185664,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.247}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.29599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:52.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:53.282 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:53.283 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:53.294 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-18T11:57:52.125070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24367,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681036,"kb_used_data":2132,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806324,"statfs":{"total":386547056640,"available":382777675776,"internally_reserved":0,"allocated":2183168,"data_stored":1133784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1923,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.909171"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13369,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225976+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:43.225976+0000","last_peered":"2024-09-18T11:57:43.225976+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:43.225976+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17364,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217277+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:43.217277+0000","last_peered":"2024-09-18T11:57:43.217277+0000","last_clean":"2024-09-18T11:57:43.217277+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:43.217277+0000","last_undegraded":"2024-09-18T11:57:43.217277+0000","last_fullsized":"2024-09-18T11:57:43.217277+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17038,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.218506+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:43.218506+0000","last_peered":"2024-09-18T11:57:43.218506+0000","last_clean":"2024-09-18T11:57:43.218506+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:43.218506+0000","last_undegraded":"2024-09-18T11:57:43.218506+0000","last_fullsized":"2024-09-18T11:57:43.218506+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13039,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217226+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:43.217226+0000","last_peered":"2024-09-18T11:57:43.217226+0000","last_clean":"2024-09-18T11:57:43.217226+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:43.217226+0000","last_undegraded":"2024-09-18T11:57:43.217226+0000","last_fullsized":"2024-09-18T11:57:43.217226+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13032,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225264+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:43.225264+0000","last_peered":"2024-09-18T11:57:43.225264+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:43.225264+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17174,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225338+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:43.225338+0000","last_peered":"2024-09-18T11:57:43.225338+0000","last_clean":"2024-09-18T11:57:43.225338+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:43.225338+0000","last_undegraded":"2024-09-18T11:57:43.225338+0000","last_fullsized":"2024-09-18T11:57:43.225338+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17388,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.224741+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:43.224741+0000","last_peered":"2024-09-18T11:57:43.224741+0000","last_clean":"2024-09-18T11:57:43.224741+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:43.224741+0000","last_undegraded":"2024-09-18T11:57:43.224741+0000","last_fullsized":"2024-09-18T11:57:43.224741+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17045,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217399+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:43.217399+0000","last_peered":"2024-09-18T11:57:43.217399+0000","last_clean":"2024-09-18T11:57:43.217399+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:43.217399+0000","last_undegraded":"2024-09-18T11:57:43.217399+0000","last_fullsized":"2024-09-18T11:57:43.217399+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17543,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225076+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:43.225076+0000","last_peered":"2024-09-18T11:57:43.225076+0000","last_clean":"2024-09-18T11:57:43.225076+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:43.225076+0000","last_undegraded":"2024-09-18T11:57:43.225076+0000","last_fullsized":"2024-09-18T11:57:43.225076+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16667,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.298755+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:44.298755+0000","last_peered":"2024-09-18T11:57:44.298755+0000","last_clean":"2024-09-18T11:57:44.298755+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:44.298755+0000","last_undegraded":"2024-09-18T11:57:44.298755+0000","last_fullsized":"2024-09-18T11:57:44.298755+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17256,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225417+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:43.225417+0000","last_peered":"2024-09-18T11:57:43.225417+0000","last_clean":"2024-09-18T11:57:43.225417+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:43.225417+0000","last_undegraded":"2024-09-18T11:57:43.225417+0000","last_fullsized":"2024-09-18T11:57:43.225417+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13141,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225951+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:43.225951+0000","last_peered":"2024-09-18T11:57:43.225951+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:43.225951+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13087,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217309+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:43.217309+0000","last_peered":"2024-09-18T11:57:43.217309+0000","last_clean":"2024-09-18T11:57:43.217309+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:43.217309+0000","last_undegraded":"2024-09-18T11:57:43.217309+0000","last_fullsized":"2024-09-18T11:57:43.217309+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":15882,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:50.332997+0000","last_change":"2024-09-18T11:57:42.502387+0000","last_active":"2024-09-18T11:57:50.332997+0000","last_peered":"2024-09-18T11:57:50.332997+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:50.332997+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1362,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17208,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217439+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:43.217439+0000","last_peered":"2024-09-18T11:57:43.217439+0000","last_clean":"2024-09-18T11:57:43.217439+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:43.217439+0000","last_undegraded":"2024-09-18T11:57:43.217439+0000","last_fullsized":"2024-09-18T11:57:43.217439+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12824,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217475+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:43.217475+0000","last_peered":"2024-09-18T11:57:43.217475+0000","last_clean":"2024-09-18T11:57:43.217475+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:43.217475+0000","last_undegraded":"2024-09-18T11:57:43.217475+0000","last_fullsized":"2024-09-18T11:57:43.217475+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24367,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923988,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447852,"statfs":{"total":96636764160,"available":95690600448,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.187},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.223}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.27800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919316,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452524,"statfs":{"total":96636764160,"available":95695384576,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.245},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.318}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29599999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922128,"kb_used_data":776,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449712,"statfs":{"total":96636764160,"available":95692505088,"internally_reserved":0,"allocated":794624,"data_stored":532278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.214},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915604,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456236,"statfs":{"total":96636764160,"available":95699185664,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.247}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.29599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:53.295 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:53.649 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:53.659 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:53.661 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-18T11:57:52.125070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24367,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681036,"kb_used_data":2132,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806324,"statfs":{"total":386547056640,"available":382777675776,"internally_reserved":0,"allocated":2183168,"data_stored":1133784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1923,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.909171"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13369,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225976+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:43.225976+0000","last_peered":"2024-09-18T11:57:43.225976+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:43.225976+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17364,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217277+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:43.217277+0000","last_peered":"2024-09-18T11:57:43.217277+0000","last_clean":"2024-09-18T11:57:43.217277+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:43.217277+0000","last_undegraded":"2024-09-18T11:57:43.217277+0000","last_fullsized":"2024-09-18T11:57:43.217277+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17038,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.218506+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:43.218506+0000","last_peered":"2024-09-18T11:57:43.218506+0000","last_clean":"2024-09-18T11:57:43.218506+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:43.218506+0000","last_undegraded":"2024-09-18T11:57:43.218506+0000","last_fullsized":"2024-09-18T11:57:43.218506+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13039,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217226+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:43.217226+0000","last_peered":"2024-09-18T11:57:43.217226+0000","last_clean":"2024-09-18T11:57:43.217226+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:43.217226+0000","last_undegraded":"2024-09-18T11:57:43.217226+0000","last_fullsized":"2024-09-18T11:57:43.217226+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13032,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225264+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:43.225264+0000","last_peered":"2024-09-18T11:57:43.225264+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:43.225264+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17174,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225338+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:43.225338+0000","last_peered":"2024-09-18T11:57:43.225338+0000","last_clean":"2024-09-18T11:57:43.225338+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:43.225338+0000","last_undegraded":"2024-09-18T11:57:43.225338+0000","last_fullsized":"2024-09-18T11:57:43.225338+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17388,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.224741+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:43.224741+0000","last_peered":"2024-09-18T11:57:43.224741+0000","last_clean":"2024-09-18T11:57:43.224741+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:43.224741+0000","last_undegraded":"2024-09-18T11:57:43.224741+0000","last_fullsized":"2024-09-18T11:57:43.224741+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17045,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217399+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:43.217399+0000","last_peered":"2024-09-18T11:57:43.217399+0000","last_clean":"2024-09-18T11:57:43.217399+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:43.217399+0000","last_undegraded":"2024-09-18T11:57:43.217399+0000","last_fullsized":"2024-09-18T11:57:43.217399+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17543,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225076+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:43.225076+0000","last_peered":"2024-09-18T11:57:43.225076+0000","last_clean":"2024-09-18T11:57:43.225076+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:43.225076+0000","last_undegraded":"2024-09-18T11:57:43.225076+0000","last_fullsized":"2024-09-18T11:57:43.225076+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16667,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.298755+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:44.298755+0000","last_peered":"2024-09-18T11:57:44.298755+0000","last_clean":"2024-09-18T11:57:44.298755+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:44.298755+0000","last_undegraded":"2024-09-18T11:57:44.298755+0000","last_fullsized":"2024-09-18T11:57:44.298755+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17256,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225417+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:43.225417+0000","last_peered":"2024-09-18T11:57:43.225417+0000","last_clean":"2024-09-18T11:57:43.225417+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:43.225417+0000","last_undegraded":"2024-09-18T11:57:43.225417+0000","last_fullsized":"2024-09-18T11:57:43.225417+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13141,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225951+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:43.225951+0000","last_peered":"2024-09-18T11:57:43.225951+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:43.225951+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13087,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217309+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:43.217309+0000","last_peered":"2024-09-18T11:57:43.217309+0000","last_clean":"2024-09-18T11:57:43.217309+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:43.217309+0000","last_undegraded":"2024-09-18T11:57:43.217309+0000","last_fullsized":"2024-09-18T11:57:43.217309+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":15882,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:50.332997+0000","last_change":"2024-09-18T11:57:42.502387+0000","last_active":"2024-09-18T11:57:50.332997+0000","last_peered":"2024-09-18T11:57:50.332997+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:50.332997+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1362,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17208,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217439+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:43.217439+0000","last_peered":"2024-09-18T11:57:43.217439+0000","last_clean":"2024-09-18T11:57:43.217439+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:43.217439+0000","last_undegraded":"2024-09-18T11:57:43.217439+0000","last_fullsized":"2024-09-18T11:57:43.217439+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12824,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217475+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:43.217475+0000","last_peered":"2024-09-18T11:57:43.217475+0000","last_clean":"2024-09-18T11:57:43.217475+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:43.217475+0000","last_undegraded":"2024-09-18T11:57:43.217475+0000","last_fullsized":"2024-09-18T11:57:43.217475+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24367,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923988,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447852,"statfs":{"total":96636764160,"available":95690600448,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.187},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.223}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.27800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919316,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452524,"statfs":{"total":96636764160,"available":95695384576,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.245},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.318}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29599999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922128,"kb_used_data":776,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449712,"statfs":{"total":96636764160,"available":95692505088,"internally_reserved":0,"allocated":794624,"data_stored":532278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.214},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915604,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456236,"statfs":{"total":96636764160,"available":95699185664,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.247}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.29599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:53.662 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:53.662 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13369, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:43.225976+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:43.225976+0000', 'last_peered': '2024-09-18T11:57:43.225976+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:43.225976+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:53.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:53.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13032, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:43.225264+0000', 'last_change': '2024-09-18T11:44:47.491291+0000', 'last_active': '2024-09-18T11:57:43.225264+0000', 'last_peered': '2024-09-18T11:57:43.225264+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:43.225264+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:53.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:53.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13141, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:43.225951+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:43.225951+0000', 'last_peered': '2024-09-18T11:57:43.225951+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:43.225951+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:53.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:53.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 15882, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:50.332997+0000', 'last_change': '2024-09-18T11:57:42.502387+0000', 'last_active': '2024-09-18T11:57:50.332997+0000', 'last_peered': '2024-09-18T11:57:50.332997+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:50.332997+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 800, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1362, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -798}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:53.663 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:54.035 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:54.035 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7042/112 objects degraded (6287.500%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":872,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3769380864,"bytes_avail":382777675776,"bytes_total":386547056640,"degraded_objects":7042,"degraded_total":112,"degraded_ratio":62.875,"recovering_objects_per_sec":215,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1923,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:54.047 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:54.047 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:54.354 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:54.354 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:54.366 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-18T11:57:54.303876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24367,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689356,"kb_used_data":2132,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373798004,"statfs":{"total":386547056640,"available":382769156096,"internally_reserved":0,"allocated":2183168,"data_stored":1133784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1923,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.078861"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13369,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225976+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:43.225976+0000","last_peered":"2024-09-18T11:57:43.225976+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:43.225976+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17364,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217277+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:43.217277+0000","last_peered":"2024-09-18T11:57:43.217277+0000","last_clean":"2024-09-18T11:57:43.217277+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:43.217277+0000","last_undegraded":"2024-09-18T11:57:43.217277+0000","last_fullsized":"2024-09-18T11:57:43.217277+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17038,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.218506+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:43.218506+0000","last_peered":"2024-09-18T11:57:43.218506+0000","last_clean":"2024-09-18T11:57:43.218506+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:43.218506+0000","last_undegraded":"2024-09-18T11:57:43.218506+0000","last_fullsized":"2024-09-18T11:57:43.218506+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13039,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217226+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:43.217226+0000","last_peered":"2024-09-18T11:57:43.217226+0000","last_clean":"2024-09-18T11:57:43.217226+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:43.217226+0000","last_undegraded":"2024-09-18T11:57:43.217226+0000","last_fullsized":"2024-09-18T11:57:43.217226+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13032,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225264+0000","last_change":"2024-09-18T11:44:47.491291+0000","last_active":"2024-09-18T11:57:43.225264+0000","last_peered":"2024-09-18T11:57:43.225264+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:43.225264+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17174,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225338+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:43.225338+0000","last_peered":"2024-09-18T11:57:43.225338+0000","last_clean":"2024-09-18T11:57:43.225338+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:43.225338+0000","last_undegraded":"2024-09-18T11:57:43.225338+0000","last_fullsized":"2024-09-18T11:57:43.225338+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17388,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.224741+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:43.224741+0000","last_peered":"2024-09-18T11:57:43.224741+0000","last_clean":"2024-09-18T11:57:43.224741+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:43.224741+0000","last_undegraded":"2024-09-18T11:57:43.224741+0000","last_fullsized":"2024-09-18T11:57:43.224741+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17045,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217399+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:43.217399+0000","last_peered":"2024-09-18T11:57:43.217399+0000","last_clean":"2024-09-18T11:57:43.217399+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:43.217399+0000","last_undegraded":"2024-09-18T11:57:43.217399+0000","last_fullsized":"2024-09-18T11:57:43.217399+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17543,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225076+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:43.225076+0000","last_peered":"2024-09-18T11:57:43.225076+0000","last_clean":"2024-09-18T11:57:43.225076+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:43.225076+0000","last_undegraded":"2024-09-18T11:57:43.225076+0000","last_fullsized":"2024-09-18T11:57:43.225076+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16667,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.298755+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:44.298755+0000","last_peered":"2024-09-18T11:57:44.298755+0000","last_clean":"2024-09-18T11:57:44.298755+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:44.298755+0000","last_undegraded":"2024-09-18T11:57:44.298755+0000","last_fullsized":"2024-09-18T11:57:44.298755+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17256,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.225417+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:43.225417+0000","last_peered":"2024-09-18T11:57:43.225417+0000","last_clean":"2024-09-18T11:57:43.225417+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:43.225417+0000","last_undegraded":"2024-09-18T11:57:43.225417+0000","last_fullsized":"2024-09-18T11:57:43.225417+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13141,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:43.225951+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:43.225951+0000","last_peered":"2024-09-18T11:57:43.225951+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:43.225951+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13087,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217309+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:43.217309+0000","last_peered":"2024-09-18T11:57:43.217309+0000","last_clean":"2024-09-18T11:57:43.217309+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:43.217309+0000","last_undegraded":"2024-09-18T11:57:43.217309+0000","last_fullsized":"2024-09-18T11:57:43.217309+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":15882,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:50.332997+0000","last_change":"2024-09-18T11:57:42.502387+0000","last_active":"2024-09-18T11:57:50.332997+0000","last_peered":"2024-09-18T11:57:50.332997+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:50.332997+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1362,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17208,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217439+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:43.217439+0000","last_peered":"2024-09-18T11:57:43.217439+0000","last_clean":"2024-09-18T11:57:43.217439+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:43.217439+0000","last_undegraded":"2024-09-18T11:57:43.217439+0000","last_fullsized":"2024-09-18T11:57:43.217439+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12824,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217475+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:43.217475+0000","last_peered":"2024-09-18T11:57:43.217475+0000","last_clean":"2024-09-18T11:57:43.217475+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:43.217475+0000","last_undegraded":"2024-09-18T11:57:43.217475+0000","last_fullsized":"2024-09-18T11:57:43.217475+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24367,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923988,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447852,"statfs":{"total":96636764160,"available":95690600448,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.187},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.223}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.27800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919316,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452524,"statfs":{"total":96636764160,"available":95695384576,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.245},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.318}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26900000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29599999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930448,"kb_used_data":776,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441392,"statfs":{"total":96636764160,"available":95683985408,"internally_reserved":0,"allocated":794624,"data_stored":532278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.26600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.28000000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.22},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.23499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915604,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456236,"statfs":{"total":96636764160,"available":95699185664,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.247}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.29599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:57.369 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:57.642 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:57.643 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:57.655 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-18T11:57:56.125835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689384,"kb_used_data":2172,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797976,"statfs":{"total":386547056640,"available":382769127424,"internally_reserved":0,"allocated":2224128,"data_stored":1152017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001377"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17364,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217277+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:43.217277+0000","last_peered":"2024-09-18T11:57:43.217277+0000","last_clean":"2024-09-18T11:57:43.217277+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:43.217277+0000","last_undegraded":"2024-09-18T11:57:43.217277+0000","last_fullsized":"2024-09-18T11:57:43.217277+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17039,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.226948+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:44.226948+0000","last_peered":"2024-09-18T11:57:44.226948+0000","last_clean":"2024-09-18T11:57:44.226948+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:44.226948+0000","last_undegraded":"2024-09-18T11:57:44.226948+0000","last_fullsized":"2024-09-18T11:57:44.226948+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13039,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217226+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:43.217226+0000","last_peered":"2024-09-18T11:57:43.217226+0000","last_clean":"2024-09-18T11:57:43.217226+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:43.217226+0000","last_undegraded":"2024-09-18T11:57:43.217226+0000","last_fullsized":"2024-09-18T11:57:43.217226+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13521,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:55.305732+0000","last_change":"2024-09-18T11:57:53.631391+0000","last_active":"2024-09-18T11:57:55.305732+0000","last_peered":"2024-09-18T11:57:55.305732+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:55.305732+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1809}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17390,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308769+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:54.308769+0000","last_peered":"2024-09-18T11:57:54.308769+0000","last_clean":"2024-09-18T11:57:54.308769+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:54.308769+0000","last_undegraded":"2024-09-18T11:57:54.308769+0000","last_fullsized":"2024-09-18T11:57:54.308769+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17045,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217399+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:43.217399+0000","last_peered":"2024-09-18T11:57:43.217399+0000","last_clean":"2024-09-18T11:57:43.217399+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:43.217399+0000","last_undegraded":"2024-09-18T11:57:43.217399+0000","last_fullsized":"2024-09-18T11:57:43.217399+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16668,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.299258+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:44.299258+0000","last_peered":"2024-09-18T11:57:44.299258+0000","last_clean":"2024-09-18T11:57:44.299258+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:44.299258+0000","last_undegraded":"2024-09-18T11:57:44.299258+0000","last_fullsized":"2024-09-18T11:57:44.299258+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13143,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.309810+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:54.309810+0000","last_peered":"2024-09-18T11:57:54.309810+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:54.309810+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13087,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217309+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:43.217309+0000","last_peered":"2024-09-18T11:57:43.217309+0000","last_clean":"2024-09-18T11:57:43.217309+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:43.217309+0000","last_undegraded":"2024-09-18T11:57:43.217309+0000","last_fullsized":"2024-09-18T11:57:43.217309+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":15882,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:50.332997+0000","last_change":"2024-09-18T11:57:42.502387+0000","last_active":"2024-09-18T11:57:50.332997+0000","last_peered":"2024-09-18T11:57:50.332997+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:50.332997+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1362,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17208,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217439+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:43.217439+0000","last_peered":"2024-09-18T11:57:43.217439+0000","last_clean":"2024-09-18T11:57:43.217439+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:43.217439+0000","last_undegraded":"2024-09-18T11:57:43.217439+0000","last_fullsized":"2024-09-18T11:57:43.217439+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12824,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217475+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:43.217475+0000","last_peered":"2024-09-18T11:57:43.217475+0000","last_clean":"2024-09-18T11:57:43.217475+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:43.217475+0000","last_undegraded":"2024-09-18T11:57:43.217475+0000","last_fullsized":"2024-09-18T11:57:43.217475+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.25}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23699999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919320,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452520,"statfs":{"total":96636764160,"available":95695380480,"internally_reserved":0,"allocated":471040,"data_stored":204157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.28999999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.23000000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.253},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.307}]}]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930448,"kb_used_data":776,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441392,"statfs":{"total":96636764160,"available":95683985408,"internally_reserved":0,"allocated":794624,"data_stored":532278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.26600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.28000000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.22},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.23499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915616,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456224,"statfs":{"total":96636764160,"available":95699173376,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.17399999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.19600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.215}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:57.657 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:57.944 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:57.945 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:57.957 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-18T11:57:56.125835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689384,"kb_used_data":2172,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797976,"statfs":{"total":386547056640,"available":382769127424,"internally_reserved":0,"allocated":2224128,"data_stored":1152017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001377"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17364,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217277+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:43.217277+0000","last_peered":"2024-09-18T11:57:43.217277+0000","last_clean":"2024-09-18T11:57:43.217277+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:43.217277+0000","last_undegraded":"2024-09-18T11:57:43.217277+0000","last_fullsized":"2024-09-18T11:57:43.217277+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17039,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.226948+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:44.226948+0000","last_peered":"2024-09-18T11:57:44.226948+0000","last_clean":"2024-09-18T11:57:44.226948+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:44.226948+0000","last_undegraded":"2024-09-18T11:57:44.226948+0000","last_fullsized":"2024-09-18T11:57:44.226948+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13039,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217226+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:43.217226+0000","last_peered":"2024-09-18T11:57:43.217226+0000","last_clean":"2024-09-18T11:57:43.217226+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:43.217226+0000","last_undegraded":"2024-09-18T11:57:43.217226+0000","last_fullsized":"2024-09-18T11:57:43.217226+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13521,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:55.305732+0000","last_change":"2024-09-18T11:57:53.631391+0000","last_active":"2024-09-18T11:57:55.305732+0000","last_peered":"2024-09-18T11:57:55.305732+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:55.305732+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1809}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17390,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308769+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:54.308769+0000","last_peered":"2024-09-18T11:57:54.308769+0000","last_clean":"2024-09-18T11:57:54.308769+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:54.308769+0000","last_undegraded":"2024-09-18T11:57:54.308769+0000","last_fullsized":"2024-09-18T11:57:54.308769+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17045,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217399+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:43.217399+0000","last_peered":"2024-09-18T11:57:43.217399+0000","last_clean":"2024-09-18T11:57:43.217399+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:43.217399+0000","last_undegraded":"2024-09-18T11:57:43.217399+0000","last_fullsized":"2024-09-18T11:57:43.217399+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16668,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.299258+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:44.299258+0000","last_peered":"2024-09-18T11:57:44.299258+0000","last_clean":"2024-09-18T11:57:44.299258+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:44.299258+0000","last_undegraded":"2024-09-18T11:57:44.299258+0000","last_fullsized":"2024-09-18T11:57:44.299258+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13143,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.309810+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:54.309810+0000","last_peered":"2024-09-18T11:57:54.309810+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:54.309810+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13087,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217309+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:43.217309+0000","last_peered":"2024-09-18T11:57:43.217309+0000","last_clean":"2024-09-18T11:57:43.217309+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:43.217309+0000","last_undegraded":"2024-09-18T11:57:43.217309+0000","last_fullsized":"2024-09-18T11:57:43.217309+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":15882,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:50.332997+0000","last_change":"2024-09-18T11:57:42.502387+0000","last_active":"2024-09-18T11:57:50.332997+0000","last_peered":"2024-09-18T11:57:50.332997+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:50.332997+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1362,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17208,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217439+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:43.217439+0000","last_peered":"2024-09-18T11:57:43.217439+0000","last_clean":"2024-09-18T11:57:43.217439+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:43.217439+0000","last_undegraded":"2024-09-18T11:57:43.217439+0000","last_fullsized":"2024-09-18T11:57:43.217439+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12824,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217475+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:43.217475+0000","last_peered":"2024-09-18T11:57:43.217475+0000","last_clean":"2024-09-18T11:57:43.217475+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:43.217475+0000","last_undegraded":"2024-09-18T11:57:43.217475+0000","last_fullsized":"2024-09-18T11:57:43.217475+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.25}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23699999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919320,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452520,"statfs":{"total":96636764160,"available":95695380480,"internally_reserved":0,"allocated":471040,"data_stored":204157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.28999999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.23000000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.253},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.307}]}]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930448,"kb_used_data":776,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441392,"statfs":{"total":96636764160,"available":95683985408,"internally_reserved":0,"allocated":794624,"data_stored":532278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.26600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.28000000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.22},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.23499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915616,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456224,"statfs":{"total":96636764160,"available":95699173376,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.17399999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.19600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.215}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:57.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:57:57.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13371, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:54.308528+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:54.308528+0000', 'last_peered': '2024-09-18T11:57:54.308528+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:54.308528+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:57.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:57:57.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 13521, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:55.305732+0000', 'last_change': '2024-09-18T11:57:53.631391+0000', 'last_active': '2024-09-18T11:57:55.305732+0000', 'last_peered': '2024-09-18T11:57:55.305732+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:57:55.305732+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1811, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 275, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1809}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:57.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:57:57.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13143, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:54.309810+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:54.309810+0000', 'last_peered': '2024-09-18T11:57:54.309810+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:54.309810+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:57.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-18T11:57:57.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13088", 'reported_seq': 15882, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:50.332997+0000', 'last_change': '2024-09-18T11:57:42.502387+0000', 'last_active': '2024-09-18T11:57:50.332997+0000', 'last_peered': '2024-09-18T11:57:50.332997+0000', 'last_clean': '2024-09-18T11:43:44.230010+0000', 'last_became_active': '2024-09-18T11:44:47.457451+0000', 'last_became_peered': '2024-09-18T11:44:47.457451+0000', 'last_unstale': '2024-09-18T11:57:50.332997+0000', 'last_undegraded': '2024-09-18T11:44:47.189734+0000', 'last_fullsized': '2024-09-18T11:44:47.116957+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 800, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13086, 'num_write_kb': 52356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1362, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -798}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:57:57.959 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:58.309 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:58.309 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6799/112 objects degraded (6070.536%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":876,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3777929216,"bytes_avail":382769127424,"bytes_total":386547056640,"degraded_objects":6799,"degraded_total":112,"degraded_ratio":60.705357142857146,"recovering_objects_per_sec":141,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1134,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:58.323 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:58.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:58.606 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:57:58.606 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:57:58.621 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2024-09-18T11:57:58.126183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689384,"kb_used_data":2172,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797976,"statfs":{"total":386547056640,"available":382769127424,"internally_reserved":0,"allocated":2224128,"data_stored":1152017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001420"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17364,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217277+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:43.217277+0000","last_peered":"2024-09-18T11:57:43.217277+0000","last_clean":"2024-09-18T11:57:43.217277+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:43.217277+0000","last_undegraded":"2024-09-18T11:57:43.217277+0000","last_fullsized":"2024-09-18T11:57:43.217277+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17039,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.226948+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:44.226948+0000","last_peered":"2024-09-18T11:57:44.226948+0000","last_clean":"2024-09-18T11:57:44.226948+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:44.226948+0000","last_undegraded":"2024-09-18T11:57:44.226948+0000","last_fullsized":"2024-09-18T11:57:44.226948+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13039,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217226+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:43.217226+0000","last_peered":"2024-09-18T11:57:43.217226+0000","last_clean":"2024-09-18T11:57:43.217226+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:43.217226+0000","last_undegraded":"2024-09-18T11:57:43.217226+0000","last_fullsized":"2024-09-18T11:57:43.217226+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13521,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:55.305732+0000","last_change":"2024-09-18T11:57:53.631391+0000","last_active":"2024-09-18T11:57:55.305732+0000","last_peered":"2024-09-18T11:57:55.305732+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:55.305732+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1809}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17390,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308769+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:54.308769+0000","last_peered":"2024-09-18T11:57:54.308769+0000","last_clean":"2024-09-18T11:57:54.308769+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:54.308769+0000","last_undegraded":"2024-09-18T11:57:54.308769+0000","last_fullsized":"2024-09-18T11:57:54.308769+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17045,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217399+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:43.217399+0000","last_peered":"2024-09-18T11:57:43.217399+0000","last_clean":"2024-09-18T11:57:43.217399+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:43.217399+0000","last_undegraded":"2024-09-18T11:57:43.217399+0000","last_fullsized":"2024-09-18T11:57:43.217399+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16668,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T11:57:44.299258+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:44.299258+0000","last_peered":"2024-09-18T11:57:44.299258+0000","last_clean":"2024-09-18T11:57:44.299258+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:44.299258+0000","last_undegraded":"2024-09-18T11:57:44.299258+0000","last_fullsized":"2024-09-18T11:57:44.299258+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13143,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.309810+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:54.309810+0000","last_peered":"2024-09-18T11:57:54.309810+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:54.309810+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13087,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217309+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:43.217309+0000","last_peered":"2024-09-18T11:57:43.217309+0000","last_clean":"2024-09-18T11:57:43.217309+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:43.217309+0000","last_undegraded":"2024-09-18T11:57:43.217309+0000","last_fullsized":"2024-09-18T11:57:43.217309+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":15882,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:50.332997+0000","last_change":"2024-09-18T11:57:42.502387+0000","last_active":"2024-09-18T11:57:50.332997+0000","last_peered":"2024-09-18T11:57:50.332997+0000","last_clean":"2024-09-18T11:43:44.230010+0000","last_became_active":"2024-09-18T11:44:47.457451+0000","last_became_peered":"2024-09-18T11:44:47.457451+0000","last_unstale":"2024-09-18T11:57:50.332997+0000","last_undegraded":"2024-09-18T11:44:47.189734+0000","last_fullsized":"2024-09-18T11:44:47.116957+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1362,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17208,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217439+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:43.217439+0000","last_peered":"2024-09-18T11:57:43.217439+0000","last_clean":"2024-09-18T11:57:43.217439+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:43.217439+0000","last_undegraded":"2024-09-18T11:57:43.217439+0000","last_fullsized":"2024-09-18T11:57:43.217439+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12824,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T11:57:43.217475+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:43.217475+0000","last_peered":"2024-09-18T11:57:43.217475+0000","last_clean":"2024-09-18T11:57:43.217475+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:43.217475+0000","last_undegraded":"2024-09-18T11:57:43.217475+0000","last_fullsized":"2024-09-18T11:57:43.217475+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24610,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.25}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23699999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919320,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452520,"statfs":{"total":96636764160,"available":95695380480,"internally_reserved":0,"allocated":471040,"data_stored":204157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.28999999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.23000000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.253},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.307}]}]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930448,"kb_used_data":776,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441392,"statfs":{"total":96636764160,"available":95683985408,"internally_reserved":0,"allocated":794624,"data_stored":532278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.26600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.28000000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.22},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.23499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915616,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456224,"statfs":{"total":96636764160,"available":95699173376,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.17399999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.19600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.215}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:01.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:01.961 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:01.962 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:01.973 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-18T11:58:00.126464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23285,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681084,"kb_used_data":2188,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806276,"statfs":{"total":386547056640,"available":382777626624,"internally_reserved":0,"allocated":2240512,"data_stored":1154748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001394"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17366,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305642+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:54.305642+0000","last_peered":"2024-09-18T11:57:54.305642+0000","last_clean":"2024-09-18T11:57:54.305642+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:54.305642+0000","last_undegraded":"2024-09-18T11:57:54.305642+0000","last_fullsized":"2024-09-18T11:57:54.305642+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17040,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306810+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:54.306810+0000","last_peered":"2024-09-18T11:57:54.306810+0000","last_clean":"2024-09-18T11:57:54.306810+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:54.306810+0000","last_undegraded":"2024-09-18T11:57:54.306810+0000","last_fullsized":"2024-09-18T11:57:54.306810+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13041,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306121+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:54.306121+0000","last_peered":"2024-09-18T11:57:54.306121+0000","last_clean":"2024-09-18T11:57:54.306121+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:54.306121+0000","last_undegraded":"2024-09-18T11:57:54.306121+0000","last_fullsized":"2024-09-18T11:57:54.306121+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":13521,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:55.305732+0000","last_change":"2024-09-18T11:57:53.631391+0000","last_active":"2024-09-18T11:57:55.305732+0000","last_peered":"2024-09-18T11:57:55.305732+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:57:55.305732+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1809}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17390,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308769+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:54.308769+0000","last_peered":"2024-09-18T11:57:54.308769+0000","last_clean":"2024-09-18T11:57:54.308769+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:54.308769+0000","last_undegraded":"2024-09-18T11:57:54.308769+0000","last_fullsized":"2024-09-18T11:57:54.308769+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305336+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:54.305336+0000","last_peered":"2024-09-18T11:57:54.305336+0000","last_clean":"2024-09-18T11:57:54.305336+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:54.305336+0000","last_undegraded":"2024-09-18T11:57:54.305336+0000","last_fullsized":"2024-09-18T11:57:54.305336+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16669,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306889+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:54.306889+0000","last_peered":"2024-09-18T11:57:54.306889+0000","last_clean":"2024-09-18T11:57:54.306889+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:54.306889+0000","last_undegraded":"2024-09-18T11:57:54.306889+0000","last_fullsized":"2024-09-18T11:57:54.306889+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13143,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.309810+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:54.309810+0000","last_peered":"2024-09-18T11:57:54.309810+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:54.309810+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13089,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306209+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:54.306209+0000","last_peered":"2024-09-18T11:57:54.306209+0000","last_clean":"2024-09-18T11:57:54.306209+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:54.306209+0000","last_undegraded":"2024-09-18T11:57:54.306209+0000","last_fullsized":"2024-09-18T11:57:54.306209+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13230,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.368006+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:57:55.368006+0000","last_peered":"2024-09-18T11:57:55.368006+0000","last_clean":"2024-09-18T11:57:55.368006+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:57:55.368006+0000","last_undegraded":"2024-09-18T11:57:55.368006+0000","last_fullsized":"2024-09-18T11:57:55.368006+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17210,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305353+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:54.305353+0000","last_peered":"2024-09-18T11:57:54.305353+0000","last_clean":"2024-09-18T11:57:54.305353+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:54.305353+0000","last_undegraded":"2024-09-18T11:57:54.305353+0000","last_fullsized":"2024-09-18T11:57:54.305353+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12826,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305422+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:54.305422+0000","last_peered":"2024-09-18T11:57:54.305422+0000","last_clean":"2024-09-18T11:57:54.305422+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:54.305422+0000","last_undegraded":"2024-09-18T11:57:54.305422+0000","last_fullsized":"2024-09-18T11:57:54.305422+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23285,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":339968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.25}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.26400000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23699999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919328,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452512,"statfs":{"total":96636764160,"available":95695372288,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29199999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.221},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.24099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922140,"kb_used_data":784,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449700,"statfs":{"total":96636764160,"available":95692492800,"internally_reserved":0,"allocated":802816,"data_stored":531375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.313},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.20200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.215}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915616,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456224,"statfs":{"total":96636764160,"available":95699173376,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.17399999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.19600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.25}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.215}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:01.975 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:02.251 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:02.251 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:02.265 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-18T11:58:02.126799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24170,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681084,"kb_used_data":2188,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806276,"statfs":{"total":386547056640,"available":382777626624,"internally_reserved":0,"allocated":2240512,"data_stored":1154748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001308"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17366,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305642+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:54.305642+0000","last_peered":"2024-09-18T11:57:54.305642+0000","last_clean":"2024-09-18T11:57:54.305642+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:54.305642+0000","last_undegraded":"2024-09-18T11:57:54.305642+0000","last_fullsized":"2024-09-18T11:57:54.305642+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17040,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306810+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:54.306810+0000","last_peered":"2024-09-18T11:57:54.306810+0000","last_clean":"2024-09-18T11:57:54.306810+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:54.306810+0000","last_undegraded":"2024-09-18T11:57:54.306810+0000","last_fullsized":"2024-09-18T11:57:54.306810+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13041,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306121+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:54.306121+0000","last_peered":"2024-09-18T11:57:54.306121+0000","last_clean":"2024-09-18T11:57:54.306121+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:54.306121+0000","last_undegraded":"2024-09-18T11:57:54.306121+0000","last_fullsized":"2024-09-18T11:57:54.306121+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":15291,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:00.332652+0000","last_change":"2024-09-18T11:57:53.631391+0000","last_active":"2024-09-18T11:58:00.332652+0000","last_peered":"2024-09-18T11:58:00.332652+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:58:00.332652+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-924}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17390,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308769+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:54.308769+0000","last_peered":"2024-09-18T11:57:54.308769+0000","last_clean":"2024-09-18T11:57:54.308769+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:54.308769+0000","last_undegraded":"2024-09-18T11:57:54.308769+0000","last_fullsized":"2024-09-18T11:57:54.308769+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305336+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:54.305336+0000","last_peered":"2024-09-18T11:57:54.305336+0000","last_clean":"2024-09-18T11:57:54.305336+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:54.305336+0000","last_undegraded":"2024-09-18T11:57:54.305336+0000","last_fullsized":"2024-09-18T11:57:54.305336+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16669,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306889+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:54.306889+0000","last_peered":"2024-09-18T11:57:54.306889+0000","last_clean":"2024-09-18T11:57:54.306889+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:54.306889+0000","last_undegraded":"2024-09-18T11:57:54.306889+0000","last_fullsized":"2024-09-18T11:57:54.306889+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13143,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.309810+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:54.309810+0000","last_peered":"2024-09-18T11:57:54.309810+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:54.309810+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13089,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306209+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:54.306209+0000","last_peered":"2024-09-18T11:57:54.306209+0000","last_clean":"2024-09-18T11:57:54.306209+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:54.306209+0000","last_undegraded":"2024-09-18T11:57:54.306209+0000","last_fullsized":"2024-09-18T11:57:54.306209+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13230,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.368006+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:57:55.368006+0000","last_peered":"2024-09-18T11:57:55.368006+0000","last_clean":"2024-09-18T11:57:55.368006+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:57:55.368006+0000","last_undegraded":"2024-09-18T11:57:55.368006+0000","last_fullsized":"2024-09-18T11:57:55.368006+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17210,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305353+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:54.305353+0000","last_peered":"2024-09-18T11:57:54.305353+0000","last_clean":"2024-09-18T11:57:54.305353+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:54.305353+0000","last_undegraded":"2024-09-18T11:57:54.305353+0000","last_fullsized":"2024-09-18T11:57:54.305353+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12826,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305422+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:54.305422+0000","last_peered":"2024-09-18T11:57:54.305422+0000","last_clean":"2024-09-18T11:57:54.305422+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:54.305422+0000","last_undegraded":"2024-09-18T11:57:54.305422+0000","last_fullsized":"2024-09-18T11:57:54.305422+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24170,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":339968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.23999999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.224},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.214}]}]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919328,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452512,"statfs":{"total":96636764160,"available":95695372288,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29199999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.221},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.24099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922140,"kb_used_data":784,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449700,"statfs":{"total":96636764160,"available":95692492800,"internally_reserved":0,"allocated":802816,"data_stored":531375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.313},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.20200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.215}]}]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915616,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456224,"statfs":{"total":96636764160,"available":95699173376,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.18099999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.30499999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.436},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:02.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:58:02.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13371, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:54.308528+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:54.308528+0000', 'last_peered': '2024-09-18T11:57:54.308528+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:54.308528+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:02.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:58:02.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 15291, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:00.332652+0000', 'last_change': '2024-09-18T11:57:53.631391+0000', 'last_active': '2024-09-18T11:58:00.332652+0000', 'last_peered': '2024-09-18T11:58:00.332652+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:58:00.332652+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 926, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1160, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -924}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:02.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:58:02.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13143, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:54.309810+0000', 'last_change': '2024-09-18T11:44:47.505710+0000', 'last_active': '2024-09-18T11:57:54.309810+0000', 'last_peered': '2024-09-18T11:57:54.309810+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:57:54.309810+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:02.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:02.692 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:02.692 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5114/112 objects degraded (4566.071%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":880,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3769430016,"bytes_avail":382777626624,"bytes_total":386547056640,"degraded_objects":5114,"degraded_total":112,"degraded_ratio":45.660714285714285},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:02.705 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T11:58:02.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:02.985 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:02.986 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:02.998 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-18T11:58:02.126799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24170,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681084,"kb_used_data":2188,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806276,"statfs":{"total":386547056640,"available":382777626624,"internally_reserved":0,"allocated":2240512,"data_stored":1154748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001308"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17366,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305642+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:54.305642+0000","last_peered":"2024-09-18T11:57:54.305642+0000","last_clean":"2024-09-18T11:57:54.305642+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:54.305642+0000","last_undegraded":"2024-09-18T11:57:54.305642+0000","last_fullsized":"2024-09-18T11:57:54.305642+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17040,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306810+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:54.306810+0000","last_peered":"2024-09-18T11:57:54.306810+0000","last_clean":"2024-09-18T11:57:54.306810+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:54.306810+0000","last_undegraded":"2024-09-18T11:57:54.306810+0000","last_fullsized":"2024-09-18T11:57:54.306810+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13041,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306121+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:54.306121+0000","last_peered":"2024-09-18T11:57:54.306121+0000","last_clean":"2024-09-18T11:57:54.306121+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:54.306121+0000","last_undegraded":"2024-09-18T11:57:54.306121+0000","last_fullsized":"2024-09-18T11:57:54.306121+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":15291,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:00.332652+0000","last_change":"2024-09-18T11:57:53.631391+0000","last_active":"2024-09-18T11:58:00.332652+0000","last_peered":"2024-09-18T11:58:00.332652+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:58:00.332652+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-924}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17390,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308769+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:54.308769+0000","last_peered":"2024-09-18T11:57:54.308769+0000","last_clean":"2024-09-18T11:57:54.308769+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:54.308769+0000","last_undegraded":"2024-09-18T11:57:54.308769+0000","last_fullsized":"2024-09-18T11:57:54.308769+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305336+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:54.305336+0000","last_peered":"2024-09-18T11:57:54.305336+0000","last_clean":"2024-09-18T11:57:54.305336+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:54.305336+0000","last_undegraded":"2024-09-18T11:57:54.305336+0000","last_fullsized":"2024-09-18T11:57:54.305336+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16669,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306889+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:54.306889+0000","last_peered":"2024-09-18T11:57:54.306889+0000","last_clean":"2024-09-18T11:57:54.306889+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:54.306889+0000","last_undegraded":"2024-09-18T11:57:54.306889+0000","last_fullsized":"2024-09-18T11:57:54.306889+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13143,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.309810+0000","last_change":"2024-09-18T11:44:47.505710+0000","last_active":"2024-09-18T11:57:54.309810+0000","last_peered":"2024-09-18T11:57:54.309810+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:57:54.309810+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13089,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306209+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:54.306209+0000","last_peered":"2024-09-18T11:57:54.306209+0000","last_clean":"2024-09-18T11:57:54.306209+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:54.306209+0000","last_undegraded":"2024-09-18T11:57:54.306209+0000","last_fullsized":"2024-09-18T11:57:54.306209+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13230,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.368006+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:57:55.368006+0000","last_peered":"2024-09-18T11:57:55.368006+0000","last_clean":"2024-09-18T11:57:55.368006+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:57:55.368006+0000","last_undegraded":"2024-09-18T11:57:55.368006+0000","last_fullsized":"2024-09-18T11:57:55.368006+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17210,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305353+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:54.305353+0000","last_peered":"2024-09-18T11:57:54.305353+0000","last_clean":"2024-09-18T11:57:54.305353+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:54.305353+0000","last_undegraded":"2024-09-18T11:57:54.305353+0000","last_fullsized":"2024-09-18T11:57:54.305353+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12826,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305422+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:54.305422+0000","last_peered":"2024-09-18T11:57:54.305422+0000","last_clean":"2024-09-18T11:57:54.305422+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:54.305422+0000","last_undegraded":"2024-09-18T11:57:54.305422+0000","last_fullsized":"2024-09-18T11:57:54.305422+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24170,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":339968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.23999999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":6.4169999999999998,"15min":6.1799999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.033999999999999,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.6200000000000001,"5min":4.9740000000000002,"15min":5.0369999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.6660000000000004,"5min":26.242999999999999,"15min":57.372},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.224},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.214}]}]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919328,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452512,"statfs":{"total":96636764160,"available":95695372288,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29199999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.221},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.24099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922140,"kb_used_data":784,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449700,"statfs":{"total":96636764160,"available":95692492800,"internally_reserved":0,"allocated":802816,"data_stored":531375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.313},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.20200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.215}]}]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915616,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456224,"statfs":{"total":96636764160,"available":95699173376,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.18099999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.30499999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.436},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:06.000 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:06.338 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:06.338 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:06.351 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-18T11:58:06.127401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689408,"kb_used_data":2196,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797952,"statfs":{"total":386547056640,"available":382769102848,"internally_reserved":0,"allocated":2248704,"data_stored":1158366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001566"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17366,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305642+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:54.305642+0000","last_peered":"2024-09-18T11:57:54.305642+0000","last_clean":"2024-09-18T11:57:54.305642+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:54.305642+0000","last_undegraded":"2024-09-18T11:57:54.305642+0000","last_fullsized":"2024-09-18T11:57:54.305642+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17040,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306810+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:54.306810+0000","last_peered":"2024-09-18T11:57:54.306810+0000","last_clean":"2024-09-18T11:57:54.306810+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:54.306810+0000","last_undegraded":"2024-09-18T11:57:54.306810+0000","last_fullsized":"2024-09-18T11:57:54.306810+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13041,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306121+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:54.306121+0000","last_peered":"2024-09-18T11:57:54.306121+0000","last_clean":"2024-09-18T11:57:54.306121+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:54.306121+0000","last_undegraded":"2024-09-18T11:57:54.306121+0000","last_fullsized":"2024-09-18T11:57:54.306121+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17144,"reported_epoch":57,"state":"active+undersized+remapped","last_fresh":"2024-09-18T11:58:04.530786+0000","last_change":"2024-09-18T11:58:04.530984+0000","last_active":"2024-09-18T11:58:04.530786+0000","last_peered":"2024-09-18T11:58:04.530786+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:58:04.530786+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17391,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.307709+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:55.307709+0000","last_peered":"2024-09-18T11:57:55.307709+0000","last_clean":"2024-09-18T11:57:55.307709+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:55.307709+0000","last_undegraded":"2024-09-18T11:57:55.307709+0000","last_fullsized":"2024-09-18T11:57:55.307709+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305336+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:54.305336+0000","last_peered":"2024-09-18T11:57:54.305336+0000","last_clean":"2024-09-18T11:57:54.305336+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:54.305336+0000","last_undegraded":"2024-09-18T11:57:54.305336+0000","last_fullsized":"2024-09-18T11:57:54.305336+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16669,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306889+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:54.306889+0000","last_peered":"2024-09-18T11:57:54.306889+0000","last_clean":"2024-09-18T11:57:54.306889+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:54.306889+0000","last_undegraded":"2024-09-18T11:57:54.306889+0000","last_fullsized":"2024-09-18T11:57:54.306889+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13400,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:05.333926+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:05.333926+0000","last_peered":"2024-09-18T11:58:05.333926+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:05.333926+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1956}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13089,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306209+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:54.306209+0000","last_peered":"2024-09-18T11:57:54.306209+0000","last_clean":"2024-09-18T11:57:54.306209+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:54.306209+0000","last_undegraded":"2024-09-18T11:57:54.306209+0000","last_fullsized":"2024-09-18T11:57:54.306209+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13230,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.368006+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:57:55.368006+0000","last_peered":"2024-09-18T11:57:55.368006+0000","last_clean":"2024-09-18T11:57:55.368006+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:57:55.368006+0000","last_undegraded":"2024-09-18T11:57:55.368006+0000","last_fullsized":"2024-09-18T11:57:55.368006+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17210,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305353+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:54.305353+0000","last_peered":"2024-09-18T11:57:54.305353+0000","last_clean":"2024-09-18T11:57:54.305353+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:54.305353+0000","last_undegraded":"2024-09-18T11:57:54.305353+0000","last_fullsized":"2024-09-18T11:57:54.305353+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12826,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305422+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:54.305422+0000","last_peered":"2024-09-18T11:57:54.305422+0000","last_clean":"2024-09-18T11:57:54.305422+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:54.305422+0000","last_undegraded":"2024-09-18T11:57:54.305422+0000","last_fullsized":"2024-09-18T11:57:54.305422+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":339968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.23100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919328,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452512,"statfs":{"total":96636764160,"available":95695372288,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29399999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.255}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.33800000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930456,"kb_used_data":784,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441384,"statfs":{"total":96636764160,"available":95683977216,"internally_reserved":0,"allocated":802816,"data_stored":531375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23899999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.254}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.28399999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.31},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915624,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456216,"statfs":{"total":96636764160,"available":95699165184,"internally_reserved":0,"allocated":487424,"data_stored":211409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28100000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.436},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:06.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:06.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-18T11:58:06.661+0000 7f3b8259c640 0 --2- 172.21.15.71:0/3576646468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b8406cdf0 0x7f3b8408c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T11:58:06.848 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:06.848 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:06.860 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-18T11:58:06.348393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689408,"kb_used_data":2196,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797952,"statfs":{"total":386547056640,"available":382769102848,"internally_reserved":0,"allocated":2248704,"data_stored":1158366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.222209"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17366,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305642+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:54.305642+0000","last_peered":"2024-09-18T11:57:54.305642+0000","last_clean":"2024-09-18T11:57:54.305642+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:54.305642+0000","last_undegraded":"2024-09-18T11:57:54.305642+0000","last_fullsized":"2024-09-18T11:57:54.305642+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17040,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306810+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:54.306810+0000","last_peered":"2024-09-18T11:57:54.306810+0000","last_clean":"2024-09-18T11:57:54.306810+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:54.306810+0000","last_undegraded":"2024-09-18T11:57:54.306810+0000","last_fullsized":"2024-09-18T11:57:54.306810+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13041,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306121+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:54.306121+0000","last_peered":"2024-09-18T11:57:54.306121+0000","last_clean":"2024-09-18T11:57:54.306121+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:54.306121+0000","last_undegraded":"2024-09-18T11:57:54.306121+0000","last_fullsized":"2024-09-18T11:57:54.306121+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17144,"reported_epoch":57,"state":"active+undersized+remapped","last_fresh":"2024-09-18T11:58:04.530786+0000","last_change":"2024-09-18T11:58:04.530984+0000","last_active":"2024-09-18T11:58:04.530786+0000","last_peered":"2024-09-18T11:58:04.530786+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:58:04.530786+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17391,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.307709+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:55.307709+0000","last_peered":"2024-09-18T11:57:55.307709+0000","last_clean":"2024-09-18T11:57:55.307709+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:55.307709+0000","last_undegraded":"2024-09-18T11:57:55.307709+0000","last_fullsized":"2024-09-18T11:57:55.307709+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305336+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:54.305336+0000","last_peered":"2024-09-18T11:57:54.305336+0000","last_clean":"2024-09-18T11:57:54.305336+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:54.305336+0000","last_undegraded":"2024-09-18T11:57:54.305336+0000","last_fullsized":"2024-09-18T11:57:54.305336+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16669,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306889+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:54.306889+0000","last_peered":"2024-09-18T11:57:54.306889+0000","last_clean":"2024-09-18T11:57:54.306889+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:54.306889+0000","last_undegraded":"2024-09-18T11:57:54.306889+0000","last_fullsized":"2024-09-18T11:57:54.306889+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13400,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:05.333926+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:05.333926+0000","last_peered":"2024-09-18T11:58:05.333926+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:05.333926+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1956}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13089,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306209+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:54.306209+0000","last_peered":"2024-09-18T11:57:54.306209+0000","last_clean":"2024-09-18T11:57:54.306209+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:54.306209+0000","last_undegraded":"2024-09-18T11:57:54.306209+0000","last_fullsized":"2024-09-18T11:57:54.306209+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13230,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.368006+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:57:55.368006+0000","last_peered":"2024-09-18T11:57:55.368006+0000","last_clean":"2024-09-18T11:57:55.368006+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:57:55.368006+0000","last_undegraded":"2024-09-18T11:57:55.368006+0000","last_fullsized":"2024-09-18T11:57:55.368006+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17210,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305353+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:54.305353+0000","last_peered":"2024-09-18T11:57:54.305353+0000","last_clean":"2024-09-18T11:57:54.305353+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:54.305353+0000","last_undegraded":"2024-09-18T11:57:54.305353+0000","last_fullsized":"2024-09-18T11:57:54.305353+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12826,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305422+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:54.305422+0000","last_peered":"2024-09-18T11:57:54.305422+0000","last_clean":"2024-09-18T11:57:54.305422+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:54.305422+0000","last_undegraded":"2024-09-18T11:57:54.305422+0000","last_fullsized":"2024-09-18T11:57:54.305422+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":339968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.23100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919328,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452512,"statfs":{"total":96636764160,"available":95695372288,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29399999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.255}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.33800000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930456,"kb_used_data":784,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441384,"statfs":{"total":96636764160,"available":95683977216,"internally_reserved":0,"allocated":802816,"data_stored":531375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23899999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.254}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.28399999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.31},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915624,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456216,"statfs":{"total":96636764160,"available":95699165184,"internally_reserved":0,"allocated":487424,"data_stored":211409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28100000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.436},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:06.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:58:06.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13371, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:54.308528+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:54.308528+0000', 'last_peered': '2024-09-18T11:57:54.308528+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:54.308528+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:06.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:58:06.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 17144, 'reported_epoch': 57, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-18T11:58:04.530786+0000', 'last_change': '2024-09-18T11:58:04.530984+0000', 'last_active': '2024-09-18T11:58:04.530786+0000', 'last_peered': '2024-09-18T11:58:04.530786+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:58:04.530786+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2086, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '1(1)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:06.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:58:06.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13400, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:05.333926+0000', 'last_change': '2024-09-18T11:58:04.532453+0000', 'last_active': '2024-09-18T11:58:05.333926+0000', 'last_peered': '2024-09-18T11:58:05.333926+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:58:05.333926+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 154, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1956}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:06.862 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:07.213 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:07.213 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4060/112 objects degraded (3625.000%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":885,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+undersized+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3777953792,"bytes_avail":382769102848,"bytes_total":386547056640,"degraded_objects":4060,"degraded_total":112,"degraded_ratio":36.25,"recovering_objects_per_sec":61,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":614,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:07.226 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:07.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:07.517 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:07.518 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:07.529 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-18T11:58:06.348393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689408,"kb_used_data":2196,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797952,"statfs":{"total":386547056640,"available":382769102848,"internally_reserved":0,"allocated":2248704,"data_stored":1158366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.222209"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17366,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305642+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:57:54.305642+0000","last_peered":"2024-09-18T11:57:54.305642+0000","last_clean":"2024-09-18T11:57:54.305642+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:57:54.305642+0000","last_undegraded":"2024-09-18T11:57:54.305642+0000","last_fullsized":"2024-09-18T11:57:54.305642+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17040,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306810+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:57:54.306810+0000","last_peered":"2024-09-18T11:57:54.306810+0000","last_clean":"2024-09-18T11:57:54.306810+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:57:54.306810+0000","last_undegraded":"2024-09-18T11:57:54.306810+0000","last_fullsized":"2024-09-18T11:57:54.306810+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13041,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306121+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:57:54.306121+0000","last_peered":"2024-09-18T11:57:54.306121+0000","last_clean":"2024-09-18T11:57:54.306121+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:57:54.306121+0000","last_undegraded":"2024-09-18T11:57:54.306121+0000","last_fullsized":"2024-09-18T11:57:54.306121+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17144,"reported_epoch":57,"state":"active+undersized+remapped","last_fresh":"2024-09-18T11:58:04.530786+0000","last_change":"2024-09-18T11:58:04.530984+0000","last_active":"2024-09-18T11:58:04.530786+0000","last_peered":"2024-09-18T11:58:04.530786+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:58:04.530786+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17391,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.307709+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:55.307709+0000","last_peered":"2024-09-18T11:57:55.307709+0000","last_clean":"2024-09-18T11:57:55.307709+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:55.307709+0000","last_undegraded":"2024-09-18T11:57:55.307709+0000","last_fullsized":"2024-09-18T11:57:55.307709+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305336+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:57:54.305336+0000","last_peered":"2024-09-18T11:57:54.305336+0000","last_clean":"2024-09-18T11:57:54.305336+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:57:54.305336+0000","last_undegraded":"2024-09-18T11:57:54.305336+0000","last_fullsized":"2024-09-18T11:57:54.305336+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16669,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306889+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:57:54.306889+0000","last_peered":"2024-09-18T11:57:54.306889+0000","last_clean":"2024-09-18T11:57:54.306889+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:57:54.306889+0000","last_undegraded":"2024-09-18T11:57:54.306889+0000","last_fullsized":"2024-09-18T11:57:54.306889+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13400,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:05.333926+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:05.333926+0000","last_peered":"2024-09-18T11:58:05.333926+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:05.333926+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1956}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13089,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.306209+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:57:54.306209+0000","last_peered":"2024-09-18T11:57:54.306209+0000","last_clean":"2024-09-18T11:57:54.306209+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:57:54.306209+0000","last_undegraded":"2024-09-18T11:57:54.306209+0000","last_fullsized":"2024-09-18T11:57:54.306209+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13230,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.368006+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:57:55.368006+0000","last_peered":"2024-09-18T11:57:55.368006+0000","last_clean":"2024-09-18T11:57:55.368006+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:57:55.368006+0000","last_undegraded":"2024-09-18T11:57:55.368006+0000","last_fullsized":"2024-09-18T11:57:55.368006+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17210,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305353+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:57:54.305353+0000","last_peered":"2024-09-18T11:57:54.305353+0000","last_clean":"2024-09-18T11:57:54.305353+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:57:54.305353+0000","last_undegraded":"2024-09-18T11:57:54.305353+0000","last_fullsized":"2024-09-18T11:57:54.305353+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12826,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.305422+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:57:54.305422+0000","last_peered":"2024-09-18T11:57:54.305422+0000","last_clean":"2024-09-18T11:57:54.305422+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:57:54.305422+0000","last_undegraded":"2024-09-18T11:57:54.305422+0000","last_fullsized":"2024-09-18T11:57:54.305422+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":339968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.23100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919328,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452512,"statfs":{"total":96636764160,"available":95695372288,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29399999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.255}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.33800000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930456,"kb_used_data":784,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441384,"statfs":{"total":96636764160,"available":95683977216,"internally_reserved":0,"allocated":802816,"data_stored":531375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23899999999999999},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.254}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.28399999999999997}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.31},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915624,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456216,"statfs":{"total":96636764160,"available":95699165184,"internally_reserved":0,"allocated":487424,"data_stored":211409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28100000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.436},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:10.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:10.833 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:10.845 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-18T11:58:10.128119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681120,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806240,"statfs":{"total":386547056640,"available":382777589760,"internally_reserved":0,"allocated":2277376,"data_stored":1168700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001320"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17368,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346587+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:05.346587+0000","last_peered":"2024-09-18T11:58:05.346587+0000","last_clean":"2024-09-18T11:58:05.346587+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:05.346587+0000","last_undegraded":"2024-09-18T11:58:05.346587+0000","last_fullsized":"2024-09-18T11:58:05.346587+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17042,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345751+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:05.345751+0000","last_peered":"2024-09-18T11:58:05.345751+0000","last_clean":"2024-09-18T11:58:05.345751+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:05.345751+0000","last_undegraded":"2024-09-18T11:58:05.345751+0000","last_fullsized":"2024-09-18T11:58:05.345751+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13043,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346143+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:05.346143+0000","last_peered":"2024-09-18T11:58:05.346143+0000","last_clean":"2024-09-18T11:58:05.346143+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:05.346143+0000","last_undegraded":"2024-09-18T11:58:05.346143+0000","last_fullsized":"2024-09-18T11:58:05.346143+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17144,"reported_epoch":57,"state":"active+undersized+remapped","last_fresh":"2024-09-18T11:58:04.530786+0000","last_change":"2024-09-18T11:58:04.530984+0000","last_active":"2024-09-18T11:58:04.530786+0000","last_peered":"2024-09-18T11:58:04.530786+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:58:04.530786+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17391,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.307709+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:55.307709+0000","last_peered":"2024-09-18T11:57:55.307709+0000","last_clean":"2024-09-18T11:57:55.307709+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:55.307709+0000","last_undegraded":"2024-09-18T11:57:55.307709+0000","last_fullsized":"2024-09-18T11:57:55.307709+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17049,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346586+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:05.346586+0000","last_peered":"2024-09-18T11:58:05.346586+0000","last_clean":"2024-09-18T11:58:05.346586+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:05.346586+0000","last_undegraded":"2024-09-18T11:58:05.346586+0000","last_fullsized":"2024-09-18T11:58:05.346586+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16671,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345841+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:05.345841+0000","last_peered":"2024-09-18T11:58:05.345841+0000","last_clean":"2024-09-18T11:58:05.345841+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:05.345841+0000","last_undegraded":"2024-09-18T11:58:05.345841+0000","last_fullsized":"2024-09-18T11:58:05.345841+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13400,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:05.333926+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:05.333926+0000","last_peered":"2024-09-18T11:58:05.333926+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:05.333926+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1956}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13091,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346732+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:05.346732+0000","last_peered":"2024-09-18T11:58:05.346732+0000","last_clean":"2024-09-18T11:58:05.346732+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:05.346732+0000","last_undegraded":"2024-09-18T11:58:05.346732+0000","last_fullsized":"2024-09-18T11:58:05.346732+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13232,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346816+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:05.346816+0000","last_peered":"2024-09-18T11:58:05.346816+0000","last_clean":"2024-09-18T11:58:05.346816+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:05.346816+0000","last_undegraded":"2024-09-18T11:58:05.346816+0000","last_fullsized":"2024-09-18T11:58:05.346816+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17212,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346813+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:05.346813+0000","last_peered":"2024-09-18T11:58:05.346813+0000","last_clean":"2024-09-18T11:58:05.346813+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:05.346813+0000","last_undegraded":"2024-09-18T11:58:05.346813+0000","last_fullsized":"2024-09-18T11:58:05.346813+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12828,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346695+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:05.346695+0000","last_peered":"2024-09-18T11:58:05.346695+0000","last_clean":"2024-09-18T11:58:05.346695+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:05.346695+0000","last_undegraded":"2024-09-18T11:58:05.346695+0000","last_fullsized":"2024-09-18T11:58:05.346695+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.23100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919348,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452492,"statfs":{"total":96636764160,"available":95695351808,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.312},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.24199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922148,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449692,"statfs":{"total":96636764160,"available":95692484608,"internally_reserved":0,"allocated":815104,"data_stored":534494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.252}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.20899999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.29099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915624,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456216,"statfs":{"total":96636764160,"available":95699165184,"internally_reserved":0,"allocated":487424,"data_stored":211409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28100000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.436},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:10.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:11.152 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:11.153 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:11.165 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-18T11:58:10.128119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681120,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806240,"statfs":{"total":386547056640,"available":382777589760,"internally_reserved":0,"allocated":2277376,"data_stored":1168700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001320"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17368,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346587+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:05.346587+0000","last_peered":"2024-09-18T11:58:05.346587+0000","last_clean":"2024-09-18T11:58:05.346587+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:05.346587+0000","last_undegraded":"2024-09-18T11:58:05.346587+0000","last_fullsized":"2024-09-18T11:58:05.346587+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17042,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345751+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:05.345751+0000","last_peered":"2024-09-18T11:58:05.345751+0000","last_clean":"2024-09-18T11:58:05.345751+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:05.345751+0000","last_undegraded":"2024-09-18T11:58:05.345751+0000","last_fullsized":"2024-09-18T11:58:05.345751+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13043,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346143+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:05.346143+0000","last_peered":"2024-09-18T11:58:05.346143+0000","last_clean":"2024-09-18T11:58:05.346143+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:05.346143+0000","last_undegraded":"2024-09-18T11:58:05.346143+0000","last_fullsized":"2024-09-18T11:58:05.346143+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17144,"reported_epoch":57,"state":"active+undersized+remapped","last_fresh":"2024-09-18T11:58:04.530786+0000","last_change":"2024-09-18T11:58:04.530984+0000","last_active":"2024-09-18T11:58:04.530786+0000","last_peered":"2024-09-18T11:58:04.530786+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:58:04.530786+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17391,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.307709+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:55.307709+0000","last_peered":"2024-09-18T11:57:55.307709+0000","last_clean":"2024-09-18T11:57:55.307709+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:55.307709+0000","last_undegraded":"2024-09-18T11:57:55.307709+0000","last_fullsized":"2024-09-18T11:57:55.307709+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17049,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346586+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:05.346586+0000","last_peered":"2024-09-18T11:58:05.346586+0000","last_clean":"2024-09-18T11:58:05.346586+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:05.346586+0000","last_undegraded":"2024-09-18T11:58:05.346586+0000","last_fullsized":"2024-09-18T11:58:05.346586+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16671,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345841+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:05.345841+0000","last_peered":"2024-09-18T11:58:05.345841+0000","last_clean":"2024-09-18T11:58:05.345841+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:05.345841+0000","last_undegraded":"2024-09-18T11:58:05.345841+0000","last_fullsized":"2024-09-18T11:58:05.345841+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13400,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:05.333926+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:05.333926+0000","last_peered":"2024-09-18T11:58:05.333926+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:05.333926+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1956}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13091,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346732+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:05.346732+0000","last_peered":"2024-09-18T11:58:05.346732+0000","last_clean":"2024-09-18T11:58:05.346732+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:05.346732+0000","last_undegraded":"2024-09-18T11:58:05.346732+0000","last_fullsized":"2024-09-18T11:58:05.346732+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13232,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346816+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:05.346816+0000","last_peered":"2024-09-18T11:58:05.346816+0000","last_clean":"2024-09-18T11:58:05.346816+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:05.346816+0000","last_undegraded":"2024-09-18T11:58:05.346816+0000","last_fullsized":"2024-09-18T11:58:05.346816+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17212,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346813+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:05.346813+0000","last_peered":"2024-09-18T11:58:05.346813+0000","last_clean":"2024-09-18T11:58:05.346813+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:05.346813+0000","last_undegraded":"2024-09-18T11:58:05.346813+0000","last_fullsized":"2024-09-18T11:58:05.346813+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12828,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346695+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:05.346695+0000","last_peered":"2024-09-18T11:58:05.346695+0000","last_clean":"2024-09-18T11:58:05.346695+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:05.346695+0000","last_undegraded":"2024-09-18T11:58:05.346695+0000","last_fullsized":"2024-09-18T11:58:05.346695+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.23100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919348,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452492,"statfs":{"total":96636764160,"available":95695351808,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.312},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.24199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922148,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449692,"statfs":{"total":96636764160,"available":95692484608,"internally_reserved":0,"allocated":815104,"data_stored":534494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.252}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.20899999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.29099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915624,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456216,"statfs":{"total":96636764160,"available":95699165184,"internally_reserved":0,"allocated":487424,"data_stored":211409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28100000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.436},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:11.166 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:58:11.166 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13371, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:54.308528+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:57:54.308528+0000', 'last_peered': '2024-09-18T11:57:54.308528+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:57:54.308528+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:11.166 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-18T11:58:11.166 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12893", 'reported_seq': 17144, 'reported_epoch': 57, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-18T11:58:04.530786+0000', 'last_change': '2024-09-18T11:58:04.530984+0000', 'last_active': '2024-09-18T11:58:04.530786+0000', 'last_peered': '2024-09-18T11:58:04.530786+0000', 'last_clean': '2024-09-18T11:43:43.917276+0000', 'last_became_active': '2024-09-18T11:44:47.481016+0000', 'last_became_peered': '2024-09-18T11:44:47.481016+0000', 'last_unstale': '2024-09-18T11:58:04.530786+0000', 'last_undegraded': '2024-09-18T11:44:47.198677+0000', 'last_fullsized': '2024-09-18T11:44:47.116717+0000', 'mapping_epoch': 30, 'log_start': "31'2800", 'ondisk_log_start': "31'2800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13558, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 51573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2086, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '1(1)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:11.167 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:58:11.167 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 13400, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:05.333926+0000', 'last_change': '2024-09-18T11:58:04.532453+0000', 'last_active': '2024-09-18T11:58:05.333926+0000', 'last_peered': '2024-09-18T11:58:05.333926+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:58:05.333926+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 154, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1956}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:11.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:11.524 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:11.524 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4060/112 objects degraded (3625.000%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":889,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+undersized+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3769466880,"bytes_avail":382777589760,"bytes_total":386547056640,"degraded_objects":4060,"degraded_total":112,"degraded_ratio":36.25,"recovering_objects_per_sec":131,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1054,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:11.537 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:11.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:11.817 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:11.818 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:11.829 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-18T11:58:10.128119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681120,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806240,"statfs":{"total":386547056640,"available":382777589760,"internally_reserved":0,"allocated":2277376,"data_stored":1168700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001320"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13371,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:54.308528+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:57:54.308528+0000","last_peered":"2024-09-18T11:57:54.308528+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:57:54.308528+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17368,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346587+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:05.346587+0000","last_peered":"2024-09-18T11:58:05.346587+0000","last_clean":"2024-09-18T11:58:05.346587+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:05.346587+0000","last_undegraded":"2024-09-18T11:58:05.346587+0000","last_fullsized":"2024-09-18T11:58:05.346587+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17042,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345751+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:05.345751+0000","last_peered":"2024-09-18T11:58:05.345751+0000","last_clean":"2024-09-18T11:58:05.345751+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:05.345751+0000","last_undegraded":"2024-09-18T11:58:05.345751+0000","last_fullsized":"2024-09-18T11:58:05.345751+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13043,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346143+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:05.346143+0000","last_peered":"2024-09-18T11:58:05.346143+0000","last_clean":"2024-09-18T11:58:05.346143+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:05.346143+0000","last_undegraded":"2024-09-18T11:58:05.346143+0000","last_fullsized":"2024-09-18T11:58:05.346143+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17144,"reported_epoch":57,"state":"active+undersized+remapped","last_fresh":"2024-09-18T11:58:04.530786+0000","last_change":"2024-09-18T11:58:04.530984+0000","last_active":"2024-09-18T11:58:04.530786+0000","last_peered":"2024-09-18T11:58:04.530786+0000","last_clean":"2024-09-18T11:43:43.917276+0000","last_became_active":"2024-09-18T11:44:47.481016+0000","last_became_peered":"2024-09-18T11:44:47.481016+0000","last_unstale":"2024-09-18T11:58:04.530786+0000","last_undegraded":"2024-09-18T11:44:47.198677+0000","last_fullsized":"2024-09-18T11:44:47.116717+0000","mapping_epoch":30,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17176,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308453+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:57:54.308453+0000","last_peered":"2024-09-18T11:57:54.308453+0000","last_clean":"2024-09-18T11:57:54.308453+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:57:54.308453+0000","last_undegraded":"2024-09-18T11:57:54.308453+0000","last_fullsized":"2024-09-18T11:57:54.308453+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17391,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T11:57:55.307709+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:57:55.307709+0000","last_peered":"2024-09-18T11:57:55.307709+0000","last_clean":"2024-09-18T11:57:55.307709+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:57:55.307709+0000","last_undegraded":"2024-09-18T11:57:55.307709+0000","last_fullsized":"2024-09-18T11:57:55.307709+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17049,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346586+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:05.346586+0000","last_peered":"2024-09-18T11:58:05.346586+0000","last_clean":"2024-09-18T11:58:05.346586+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:05.346586+0000","last_undegraded":"2024-09-18T11:58:05.346586+0000","last_fullsized":"2024-09-18T11:58:05.346586+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17545,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308364+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:57:54.308364+0000","last_peered":"2024-09-18T11:57:54.308364+0000","last_clean":"2024-09-18T11:57:54.308364+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:57:54.308364+0000","last_undegraded":"2024-09-18T11:57:54.308364+0000","last_fullsized":"2024-09-18T11:57:54.308364+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16671,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345841+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:05.345841+0000","last_peered":"2024-09-18T11:58:05.345841+0000","last_clean":"2024-09-18T11:58:05.345841+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:05.345841+0000","last_undegraded":"2024-09-18T11:58:05.345841+0000","last_fullsized":"2024-09-18T11:58:05.345841+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17258,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T11:57:54.308914+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:57:54.308914+0000","last_peered":"2024-09-18T11:57:54.308914+0000","last_clean":"2024-09-18T11:57:54.308914+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:57:54.308914+0000","last_undegraded":"2024-09-18T11:57:54.308914+0000","last_fullsized":"2024-09-18T11:57:54.308914+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13400,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:05.333926+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:05.333926+0000","last_peered":"2024-09-18T11:58:05.333926+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:05.333926+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1956}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13091,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346732+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:05.346732+0000","last_peered":"2024-09-18T11:58:05.346732+0000","last_clean":"2024-09-18T11:58:05.346732+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:05.346732+0000","last_undegraded":"2024-09-18T11:58:05.346732+0000","last_fullsized":"2024-09-18T11:58:05.346732+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13232,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346816+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:05.346816+0000","last_peered":"2024-09-18T11:58:05.346816+0000","last_clean":"2024-09-18T11:58:05.346816+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:05.346816+0000","last_undegraded":"2024-09-18T11:58:05.346816+0000","last_fullsized":"2024-09-18T11:58:05.346816+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17212,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346813+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:05.346813+0000","last_peered":"2024-09-18T11:58:05.346813+0000","last_clean":"2024-09-18T11:58:05.346813+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:05.346813+0000","last_undegraded":"2024-09-18T11:58:05.346813+0000","last_fullsized":"2024-09-18T11:58:05.346813+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12828,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346695+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:05.346695+0000","last_peered":"2024-09-18T11:58:05.346695+0000","last_clean":"2024-09-18T11:58:05.346695+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:05.346695+0000","last_undegraded":"2024-09-18T11:58:05.346695+0000","last_fullsized":"2024-09-18T11:58:05.346695+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25224,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924000,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447840,"statfs":{"total":96636764160,"available":95690588160,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.23100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919348,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452492,"statfs":{"total":96636764160,"available":95695351808,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.312},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.24199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922148,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449692,"statfs":{"total":96636764160,"available":95692484608,"internally_reserved":0,"allocated":815104,"data_stored":534494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":6.4329999999999998,"15min":5.1639999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":6.1150000000000002,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.9379999999999999,"5min":6.6769999999999996,"15min":5.6379999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":18.920000000000002,"5min":54.710999999999999,"15min":54.710999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":6.7770000000000001,"15min":5.7300000000000004},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":15.996,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":5.8949999999999996,"15min":4.7930000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19},"max":{"1min":7.0529999999999999,"5min":28.094999999999999,"15min":28.094999999999999},"last":0.252}]},{"osd":3,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":5.4749999999999996,"15min":4.9969999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":16.347999999999999,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.20899999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":6.7409999999999997,"15min":5.9960000000000004},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20200000000000001},"max":{"1min":8.4410000000000007,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.29099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915624,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456216,"statfs":{"total":96636764160,"available":95699165184,"internally_reserved":0,"allocated":487424,"data_stored":211409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.223}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.28100000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.436},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:14.833 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:15.147 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:15.148 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:15.159 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-18T11:58:14.128747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26145,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689472,"kb_used_data":2248,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797888,"statfs":{"total":386547056640,"available":382769037312,"internally_reserved":0,"allocated":2301952,"data_stored":1179512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.786834"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13373,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:05.348073+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:58:05.348073+0000","last_peered":"2024-09-18T11:58:05.348073+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:05.348073+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17368,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346587+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:05.346587+0000","last_peered":"2024-09-18T11:58:05.346587+0000","last_clean":"2024-09-18T11:58:05.346587+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:05.346587+0000","last_undegraded":"2024-09-18T11:58:05.346587+0000","last_fullsized":"2024-09-18T11:58:05.346587+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17042,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345751+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:05.345751+0000","last_peered":"2024-09-18T11:58:05.345751+0000","last_clean":"2024-09-18T11:58:05.345751+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:05.345751+0000","last_undegraded":"2024-09-18T11:58:05.345751+0000","last_fullsized":"2024-09-18T11:58:05.345751+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13043,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346143+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:05.346143+0000","last_peered":"2024-09-18T11:58:05.346143+0000","last_clean":"2024-09-18T11:58:05.346143+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:05.346143+0000","last_undegraded":"2024-09-18T11:58:05.346143+0000","last_fullsized":"2024-09-18T11:58:05.346143+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17157,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.421109+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:06.421109+0000","last_peered":"2024-09-18T11:58:06.421109+0000","last_clean":"2024-09-18T11:58:06.421109+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:06.421109+0000","last_undegraded":"2024-09-18T11:58:06.421109+0000","last_fullsized":"2024-09-18T11:58:06.421109+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17178,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.347826+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:05.347826+0000","last_peered":"2024-09-18T11:58:05.347826+0000","last_clean":"2024-09-18T11:58:05.347826+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:05.347826+0000","last_undegraded":"2024-09-18T11:58:05.347826+0000","last_fullsized":"2024-09-18T11:58:05.347826+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17392,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.353539+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:05.353539+0000","last_peered":"2024-09-18T11:58:05.353539+0000","last_clean":"2024-09-18T11:58:05.353539+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:05.353539+0000","last_undegraded":"2024-09-18T11:58:05.353539+0000","last_fullsized":"2024-09-18T11:58:05.353539+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17049,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346586+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:05.346586+0000","last_peered":"2024-09-18T11:58:05.346586+0000","last_clean":"2024-09-18T11:58:05.346586+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:05.346586+0000","last_undegraded":"2024-09-18T11:58:05.346586+0000","last_fullsized":"2024-09-18T11:58:05.346586+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17547,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.347820+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:05.347820+0000","last_peered":"2024-09-18T11:58:05.347820+0000","last_clean":"2024-09-18T11:58:05.347820+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:05.347820+0000","last_undegraded":"2024-09-18T11:58:05.347820+0000","last_fullsized":"2024-09-18T11:58:05.347820+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16671,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345841+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:05.345841+0000","last_peered":"2024-09-18T11:58:05.345841+0000","last_clean":"2024-09-18T11:58:05.345841+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:05.345841+0000","last_undegraded":"2024-09-18T11:58:05.345841+0000","last_fullsized":"2024-09-18T11:58:05.345841+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17260,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.347867+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:05.347867+0000","last_peered":"2024-09-18T11:58:05.347867+0000","last_clean":"2024-09-18T11:58:05.347867+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:05.347867+0000","last_undegraded":"2024-09-18T11:58:05.347867+0000","last_fullsized":"2024-09-18T11:58:05.347867+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":15244,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:10.332973+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:10.332973+0000","last_peered":"2024-09-18T11:58:10.332973+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:10.332973+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1075,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13091,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346732+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:05.346732+0000","last_peered":"2024-09-18T11:58:05.346732+0000","last_clean":"2024-09-18T11:58:05.346732+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:05.346732+0000","last_undegraded":"2024-09-18T11:58:05.346732+0000","last_fullsized":"2024-09-18T11:58:05.346732+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13232,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346816+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:05.346816+0000","last_peered":"2024-09-18T11:58:05.346816+0000","last_clean":"2024-09-18T11:58:05.346816+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:05.346816+0000","last_undegraded":"2024-09-18T11:58:05.346816+0000","last_fullsized":"2024-09-18T11:58:05.346816+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17212,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346813+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:05.346813+0000","last_peered":"2024-09-18T11:58:05.346813+0000","last_clean":"2024-09-18T11:58:05.346813+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:05.346813+0000","last_undegraded":"2024-09-18T11:58:05.346813+0000","last_fullsized":"2024-09-18T11:58:05.346813+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12828,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346695+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:05.346695+0000","last_peered":"2024-09-18T11:58:05.346695+0000","last_clean":"2024-09-18T11:58:05.346695+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:05.346695+0000","last_undegraded":"2024-09-18T11:58:05.346695+0000","last_fullsized":"2024-09-18T11:58:05.346695+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26145,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924020,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447820,"statfs":{"total":96636764160,"available":95690567680,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.23100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919348,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452492,"statfs":{"total":96636764160,"available":95695351808,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:57:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":7.0469999999999997,"15min":6.6660000000000004},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":19.864999999999998,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.0920000000000001,"5min":4.9900000000000002,"15min":4.4489999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.182},"max":{"1min":10.99,"5min":26.545000000000002,"15min":26.545000000000002},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.312},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.24199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930468,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441372,"statfs":{"total":96636764160,"available":95683964928,"internally_reserved":0,"allocated":815104,"data_stored":534494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.216},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.26300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.32800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915636,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456204,"statfs":{"total":96636764160,"available":95699152896,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.187},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.25900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.30199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.224},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:15.160 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:15.464 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:15.478 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-09-18T11:58:15.194306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26145,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689472,"kb_used_data":2248,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797888,"statfs":{"total":386547056640,"available":382769037312,"internally_reserved":0,"allocated":2301952,"data_stored":1179512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.066905"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13373,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:05.348073+0000","last_change":"2024-09-18T11:44:47.537286+0000","last_active":"2024-09-18T11:58:05.348073+0000","last_peered":"2024-09-18T11:58:05.348073+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:05.348073+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17368,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346587+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:05.346587+0000","last_peered":"2024-09-18T11:58:05.346587+0000","last_clean":"2024-09-18T11:58:05.346587+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:05.346587+0000","last_undegraded":"2024-09-18T11:58:05.346587+0000","last_fullsized":"2024-09-18T11:58:05.346587+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17042,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345751+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:05.345751+0000","last_peered":"2024-09-18T11:58:05.345751+0000","last_clean":"2024-09-18T11:58:05.345751+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:05.345751+0000","last_undegraded":"2024-09-18T11:58:05.345751+0000","last_fullsized":"2024-09-18T11:58:05.345751+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13043,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346143+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:05.346143+0000","last_peered":"2024-09-18T11:58:05.346143+0000","last_clean":"2024-09-18T11:58:05.346143+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:05.346143+0000","last_undegraded":"2024-09-18T11:58:05.346143+0000","last_fullsized":"2024-09-18T11:58:05.346143+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17157,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.421109+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:06.421109+0000","last_peered":"2024-09-18T11:58:06.421109+0000","last_clean":"2024-09-18T11:58:06.421109+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:06.421109+0000","last_undegraded":"2024-09-18T11:58:06.421109+0000","last_fullsized":"2024-09-18T11:58:06.421109+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17178,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.347826+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:05.347826+0000","last_peered":"2024-09-18T11:58:05.347826+0000","last_clean":"2024-09-18T11:58:05.347826+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:05.347826+0000","last_undegraded":"2024-09-18T11:58:05.347826+0000","last_fullsized":"2024-09-18T11:58:05.347826+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17392,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.353539+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:05.353539+0000","last_peered":"2024-09-18T11:58:05.353539+0000","last_clean":"2024-09-18T11:58:05.353539+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:05.353539+0000","last_undegraded":"2024-09-18T11:58:05.353539+0000","last_fullsized":"2024-09-18T11:58:05.353539+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17049,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346586+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:05.346586+0000","last_peered":"2024-09-18T11:58:05.346586+0000","last_clean":"2024-09-18T11:58:05.346586+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:05.346586+0000","last_undegraded":"2024-09-18T11:58:05.346586+0000","last_fullsized":"2024-09-18T11:58:05.346586+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17547,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.347820+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:05.347820+0000","last_peered":"2024-09-18T11:58:05.347820+0000","last_clean":"2024-09-18T11:58:05.347820+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:05.347820+0000","last_undegraded":"2024-09-18T11:58:05.347820+0000","last_fullsized":"2024-09-18T11:58:05.347820+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16671,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345841+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:05.345841+0000","last_peered":"2024-09-18T11:58:05.345841+0000","last_clean":"2024-09-18T11:58:05.345841+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:05.345841+0000","last_undegraded":"2024-09-18T11:58:05.345841+0000","last_fullsized":"2024-09-18T11:58:05.345841+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17260,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.347867+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:05.347867+0000","last_peered":"2024-09-18T11:58:05.347867+0000","last_clean":"2024-09-18T11:58:05.347867+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:05.347867+0000","last_undegraded":"2024-09-18T11:58:05.347867+0000","last_fullsized":"2024-09-18T11:58:05.347867+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":15244,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:10.332973+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:10.332973+0000","last_peered":"2024-09-18T11:58:10.332973+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:10.332973+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1075,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13091,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346732+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:05.346732+0000","last_peered":"2024-09-18T11:58:05.346732+0000","last_clean":"2024-09-18T11:58:05.346732+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:05.346732+0000","last_undegraded":"2024-09-18T11:58:05.346732+0000","last_fullsized":"2024-09-18T11:58:05.346732+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13232,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346816+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:05.346816+0000","last_peered":"2024-09-18T11:58:05.346816+0000","last_clean":"2024-09-18T11:58:05.346816+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:05.346816+0000","last_undegraded":"2024-09-18T11:58:05.346816+0000","last_fullsized":"2024-09-18T11:58:05.346816+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17212,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346813+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:05.346813+0000","last_peered":"2024-09-18T11:58:05.346813+0000","last_clean":"2024-09-18T11:58:05.346813+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:05.346813+0000","last_undegraded":"2024-09-18T11:58:05.346813+0000","last_fullsized":"2024-09-18T11:58:05.346813+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12828,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346695+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:05.346695+0000","last_peered":"2024-09-18T11:58:05.346695+0000","last_clean":"2024-09-18T11:58:05.346695+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:05.346695+0000","last_undegraded":"2024-09-18T11:58:05.346695+0000","last_fullsized":"2024-09-18T11:58:05.346695+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26145,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924020,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447820,"statfs":{"total":96636764160,"available":95690567680,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.23100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.252},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.24299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919348,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452492,"statfs":{"total":96636764160,"available":95695351808,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.16400000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.152},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.23100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.187},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29599999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930468,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441372,"statfs":{"total":96636764160,"available":95683964928,"internally_reserved":0,"allocated":815104,"data_stored":534494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.216},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.26300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.32800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915636,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456204,"statfs":{"total":96636764160,"available":95699152896,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3580000000000001,"5min":6.9889999999999999,"15min":5.4809999999999999},"min":{"1min":0.157,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":13.087999999999999,"5min":27.419,"15min":28.059000000000001},"last":0.187},{"interface":"front","average":{"1min":1.5429999999999999,"5min":5.7359999999999998,"15min":4.9889999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":10.404999999999999,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.25900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.30199999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.224},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:15.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:58:15.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13373, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:05.348073+0000', 'last_change': '2024-09-18T11:44:47.537286+0000', 'last_active': '2024-09-18T11:58:05.348073+0000', 'last_peered': '2024-09-18T11:58:05.348073+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:58:05.348073+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2103}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:15.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:58:15.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 15244, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:10.332973+0000', 'last_change': '2024-09-18T11:58:04.532453+0000', 'last_active': '2024-09-18T11:58:10.332973+0000', 'last_peered': '2024-09-18T11:58:10.332973+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:58:10.332973+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1075, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1035}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:15.480 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:15.874 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:15.874 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3139/112 objects degraded (2802.679%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":893,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":60,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3778019328,"bytes_avail":382769037312,"bytes_total":386547056640,"degraded_objects":3139,"degraded_total":112,"degraded_ratio":28.026785714285715,"recovering_objects_per_sec":104,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":921,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:15.888 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:15.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:16.268 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:16.269 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:16.282 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-09-18T11:58:16.205851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26200,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689476,"kb_used_data":2256,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797884,"statfs":{"total":386547056640,"available":382769033216,"internally_reserved":0,"allocated":2310144,"data_stored":1183093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.078075"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13485,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:15.332651+0000","last_change":"2024-09-18T11:58:14.912652+0000","last_active":"2024-09-18T11:58:15.332651+0000","last_peered":"2024-09-18T11:58:15.332651+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:15.332651+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17368,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346587+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:05.346587+0000","last_peered":"2024-09-18T11:58:05.346587+0000","last_clean":"2024-09-18T11:58:05.346587+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:05.346587+0000","last_undegraded":"2024-09-18T11:58:05.346587+0000","last_fullsized":"2024-09-18T11:58:05.346587+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17042,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345751+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:05.345751+0000","last_peered":"2024-09-18T11:58:05.345751+0000","last_clean":"2024-09-18T11:58:05.345751+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:05.345751+0000","last_undegraded":"2024-09-18T11:58:05.345751+0000","last_fullsized":"2024-09-18T11:58:05.345751+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13043,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346143+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:05.346143+0000","last_peered":"2024-09-18T11:58:05.346143+0000","last_clean":"2024-09-18T11:58:05.346143+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:05.346143+0000","last_undegraded":"2024-09-18T11:58:05.346143+0000","last_fullsized":"2024-09-18T11:58:05.346143+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17158,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.421831+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:06.421831+0000","last_peered":"2024-09-18T11:58:06.421831+0000","last_clean":"2024-09-18T11:58:06.421831+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:06.421831+0000","last_undegraded":"2024-09-18T11:58:06.421831+0000","last_fullsized":"2024-09-18T11:58:06.421831+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17179,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353256+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:06.353256+0000","last_peered":"2024-09-18T11:58:06.353256+0000","last_clean":"2024-09-18T11:58:06.353256+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:06.353256+0000","last_undegraded":"2024-09-18T11:58:06.353256+0000","last_fullsized":"2024-09-18T11:58:06.353256+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17393,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.351660+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:06.351660+0000","last_peered":"2024-09-18T11:58:06.351660+0000","last_clean":"2024-09-18T11:58:06.351660+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:06.351660+0000","last_undegraded":"2024-09-18T11:58:06.351660+0000","last_fullsized":"2024-09-18T11:58:06.351660+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17049,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346586+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:05.346586+0000","last_peered":"2024-09-18T11:58:05.346586+0000","last_clean":"2024-09-18T11:58:05.346586+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:05.346586+0000","last_undegraded":"2024-09-18T11:58:05.346586+0000","last_fullsized":"2024-09-18T11:58:05.346586+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17548,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353176+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:06.353176+0000","last_peered":"2024-09-18T11:58:06.353176+0000","last_clean":"2024-09-18T11:58:06.353176+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:06.353176+0000","last_undegraded":"2024-09-18T11:58:06.353176+0000","last_fullsized":"2024-09-18T11:58:06.353176+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16671,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345841+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:05.345841+0000","last_peered":"2024-09-18T11:58:05.345841+0000","last_clean":"2024-09-18T11:58:05.345841+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:05.345841+0000","last_undegraded":"2024-09-18T11:58:05.345841+0000","last_fullsized":"2024-09-18T11:58:05.345841+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353476+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:06.353476+0000","last_peered":"2024-09-18T11:58:06.353476+0000","last_clean":"2024-09-18T11:58:06.353476+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:06.353476+0000","last_undegraded":"2024-09-18T11:58:06.353476+0000","last_fullsized":"2024-09-18T11:58:06.353476+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":15244,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:10.332973+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:10.332973+0000","last_peered":"2024-09-18T11:58:10.332973+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:10.332973+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1075,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13091,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346732+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:05.346732+0000","last_peered":"2024-09-18T11:58:05.346732+0000","last_clean":"2024-09-18T11:58:05.346732+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:05.346732+0000","last_undegraded":"2024-09-18T11:58:05.346732+0000","last_fullsized":"2024-09-18T11:58:05.346732+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13232,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346816+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:05.346816+0000","last_peered":"2024-09-18T11:58:05.346816+0000","last_clean":"2024-09-18T11:58:05.346816+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:05.346816+0000","last_undegraded":"2024-09-18T11:58:05.346816+0000","last_fullsized":"2024-09-18T11:58:05.346816+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17212,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346813+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:05.346813+0000","last_peered":"2024-09-18T11:58:05.346813+0000","last_clean":"2024-09-18T11:58:05.346813+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:05.346813+0000","last_undegraded":"2024-09-18T11:58:05.346813+0000","last_fullsized":"2024-09-18T11:58:05.346813+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12828,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346695+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:05.346695+0000","last_peered":"2024-09-18T11:58:05.346695+0000","last_clean":"2024-09-18T11:58:05.346695+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:05.346695+0000","last_undegraded":"2024-09-18T11:58:05.346695+0000","last_fullsized":"2024-09-18T11:58:05.346695+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26200,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924020,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447820,"statfs":{"total":96636764160,"available":95690567680,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.27600000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.25},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.28899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919348,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452492,"statfs":{"total":96636764160,"available":95695351808,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.16400000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.152},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.23100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.187},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29599999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930468,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441372,"statfs":{"total":96636764160,"available":95683964928,"internally_reserved":0,"allocated":815104,"data_stored":534494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.216},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.26300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.32800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915640,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456200,"statfs":{"total":96636764160,"available":95699148800,"internally_reserved":0,"allocated":503808,"data_stored":218587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.214}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.255}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:19.284 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:19.567 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:19.568 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:19.580 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-18T11:58:18.129628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26200,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689476,"kb_used_data":2256,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797884,"statfs":{"total":386547056640,"available":382769033216,"internally_reserved":0,"allocated":2310144,"data_stored":1183093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001508"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13485,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:15.332651+0000","last_change":"2024-09-18T11:58:14.912652+0000","last_active":"2024-09-18T11:58:15.332651+0000","last_peered":"2024-09-18T11:58:15.332651+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:15.332651+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17368,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346587+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:05.346587+0000","last_peered":"2024-09-18T11:58:05.346587+0000","last_clean":"2024-09-18T11:58:05.346587+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:05.346587+0000","last_undegraded":"2024-09-18T11:58:05.346587+0000","last_fullsized":"2024-09-18T11:58:05.346587+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17042,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345751+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:05.345751+0000","last_peered":"2024-09-18T11:58:05.345751+0000","last_clean":"2024-09-18T11:58:05.345751+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:05.345751+0000","last_undegraded":"2024-09-18T11:58:05.345751+0000","last_fullsized":"2024-09-18T11:58:05.345751+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13043,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346143+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:05.346143+0000","last_peered":"2024-09-18T11:58:05.346143+0000","last_clean":"2024-09-18T11:58:05.346143+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:05.346143+0000","last_undegraded":"2024-09-18T11:58:05.346143+0000","last_fullsized":"2024-09-18T11:58:05.346143+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17158,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.421831+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:06.421831+0000","last_peered":"2024-09-18T11:58:06.421831+0000","last_clean":"2024-09-18T11:58:06.421831+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:06.421831+0000","last_undegraded":"2024-09-18T11:58:06.421831+0000","last_fullsized":"2024-09-18T11:58:06.421831+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17179,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353256+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:06.353256+0000","last_peered":"2024-09-18T11:58:06.353256+0000","last_clean":"2024-09-18T11:58:06.353256+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:06.353256+0000","last_undegraded":"2024-09-18T11:58:06.353256+0000","last_fullsized":"2024-09-18T11:58:06.353256+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17393,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.351660+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:06.351660+0000","last_peered":"2024-09-18T11:58:06.351660+0000","last_clean":"2024-09-18T11:58:06.351660+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:06.351660+0000","last_undegraded":"2024-09-18T11:58:06.351660+0000","last_fullsized":"2024-09-18T11:58:06.351660+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17049,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346586+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:05.346586+0000","last_peered":"2024-09-18T11:58:05.346586+0000","last_clean":"2024-09-18T11:58:05.346586+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:05.346586+0000","last_undegraded":"2024-09-18T11:58:05.346586+0000","last_fullsized":"2024-09-18T11:58:05.346586+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17548,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353176+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:06.353176+0000","last_peered":"2024-09-18T11:58:06.353176+0000","last_clean":"2024-09-18T11:58:06.353176+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:06.353176+0000","last_undegraded":"2024-09-18T11:58:06.353176+0000","last_fullsized":"2024-09-18T11:58:06.353176+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16671,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345841+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:05.345841+0000","last_peered":"2024-09-18T11:58:05.345841+0000","last_clean":"2024-09-18T11:58:05.345841+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:05.345841+0000","last_undegraded":"2024-09-18T11:58:05.345841+0000","last_fullsized":"2024-09-18T11:58:05.345841+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353476+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:06.353476+0000","last_peered":"2024-09-18T11:58:06.353476+0000","last_clean":"2024-09-18T11:58:06.353476+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:06.353476+0000","last_undegraded":"2024-09-18T11:58:06.353476+0000","last_fullsized":"2024-09-18T11:58:06.353476+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":15244,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:10.332973+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:10.332973+0000","last_peered":"2024-09-18T11:58:10.332973+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:10.332973+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1075,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13091,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346732+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:05.346732+0000","last_peered":"2024-09-18T11:58:05.346732+0000","last_clean":"2024-09-18T11:58:05.346732+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:05.346732+0000","last_undegraded":"2024-09-18T11:58:05.346732+0000","last_fullsized":"2024-09-18T11:58:05.346732+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13232,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346816+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:05.346816+0000","last_peered":"2024-09-18T11:58:05.346816+0000","last_clean":"2024-09-18T11:58:05.346816+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:05.346816+0000","last_undegraded":"2024-09-18T11:58:05.346816+0000","last_fullsized":"2024-09-18T11:58:05.346816+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17212,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346813+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:05.346813+0000","last_peered":"2024-09-18T11:58:05.346813+0000","last_clean":"2024-09-18T11:58:05.346813+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:05.346813+0000","last_undegraded":"2024-09-18T11:58:05.346813+0000","last_fullsized":"2024-09-18T11:58:05.346813+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12828,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346695+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:05.346695+0000","last_peered":"2024-09-18T11:58:05.346695+0000","last_clean":"2024-09-18T11:58:05.346695+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:05.346695+0000","last_undegraded":"2024-09-18T11:58:05.346695+0000","last_fullsized":"2024-09-18T11:58:05.346695+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26200,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924020,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447820,"statfs":{"total":96636764160,"available":95690567680,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.27600000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.25},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.28899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919348,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452492,"statfs":{"total":96636764160,"available":95695351808,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.16400000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.152},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.23100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.187},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29599999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930468,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441372,"statfs":{"total":96636764160,"available":95683964928,"internally_reserved":0,"allocated":815104,"data_stored":534494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.216},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.26300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.32800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915640,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456200,"statfs":{"total":96636764160,"available":95699148800,"internally_reserved":0,"allocated":503808,"data_stored":218587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.214}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.255}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:19.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:19.856 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:19.857 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:19.869 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-18T11:58:18.129628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26200,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689476,"kb_used_data":2256,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797884,"statfs":{"total":386547056640,"available":382769033216,"internally_reserved":0,"allocated":2310144,"data_stored":1183093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001508"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13485,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:15.332651+0000","last_change":"2024-09-18T11:58:14.912652+0000","last_active":"2024-09-18T11:58:15.332651+0000","last_peered":"2024-09-18T11:58:15.332651+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:15.332651+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17368,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346587+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:05.346587+0000","last_peered":"2024-09-18T11:58:05.346587+0000","last_clean":"2024-09-18T11:58:05.346587+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:05.346587+0000","last_undegraded":"2024-09-18T11:58:05.346587+0000","last_fullsized":"2024-09-18T11:58:05.346587+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17042,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345751+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:05.345751+0000","last_peered":"2024-09-18T11:58:05.345751+0000","last_clean":"2024-09-18T11:58:05.345751+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:05.345751+0000","last_undegraded":"2024-09-18T11:58:05.345751+0000","last_fullsized":"2024-09-18T11:58:05.345751+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13043,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346143+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:05.346143+0000","last_peered":"2024-09-18T11:58:05.346143+0000","last_clean":"2024-09-18T11:58:05.346143+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:05.346143+0000","last_undegraded":"2024-09-18T11:58:05.346143+0000","last_fullsized":"2024-09-18T11:58:05.346143+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17158,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.421831+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:06.421831+0000","last_peered":"2024-09-18T11:58:06.421831+0000","last_clean":"2024-09-18T11:58:06.421831+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:06.421831+0000","last_undegraded":"2024-09-18T11:58:06.421831+0000","last_fullsized":"2024-09-18T11:58:06.421831+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17179,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353256+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:06.353256+0000","last_peered":"2024-09-18T11:58:06.353256+0000","last_clean":"2024-09-18T11:58:06.353256+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:06.353256+0000","last_undegraded":"2024-09-18T11:58:06.353256+0000","last_fullsized":"2024-09-18T11:58:06.353256+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17393,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.351660+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:06.351660+0000","last_peered":"2024-09-18T11:58:06.351660+0000","last_clean":"2024-09-18T11:58:06.351660+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:06.351660+0000","last_undegraded":"2024-09-18T11:58:06.351660+0000","last_fullsized":"2024-09-18T11:58:06.351660+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17049,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346586+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:05.346586+0000","last_peered":"2024-09-18T11:58:05.346586+0000","last_clean":"2024-09-18T11:58:05.346586+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:05.346586+0000","last_undegraded":"2024-09-18T11:58:05.346586+0000","last_fullsized":"2024-09-18T11:58:05.346586+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17548,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353176+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:06.353176+0000","last_peered":"2024-09-18T11:58:06.353176+0000","last_clean":"2024-09-18T11:58:06.353176+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:06.353176+0000","last_undegraded":"2024-09-18T11:58:06.353176+0000","last_fullsized":"2024-09-18T11:58:06.353176+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16671,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.345841+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:05.345841+0000","last_peered":"2024-09-18T11:58:05.345841+0000","last_clean":"2024-09-18T11:58:05.345841+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:05.345841+0000","last_undegraded":"2024-09-18T11:58:05.345841+0000","last_fullsized":"2024-09-18T11:58:05.345841+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353476+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:06.353476+0000","last_peered":"2024-09-18T11:58:06.353476+0000","last_clean":"2024-09-18T11:58:06.353476+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:06.353476+0000","last_undegraded":"2024-09-18T11:58:06.353476+0000","last_fullsized":"2024-09-18T11:58:06.353476+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":15244,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:10.332973+0000","last_change":"2024-09-18T11:58:04.532453+0000","last_active":"2024-09-18T11:58:10.332973+0000","last_peered":"2024-09-18T11:58:10.332973+0000","last_clean":"2024-09-18T11:43:39.896884+0000","last_became_active":"2024-09-18T11:44:47.480227+0000","last_became_peered":"2024-09-18T11:44:47.480227+0000","last_unstale":"2024-09-18T11:58:10.332973+0000","last_undegraded":"2024-09-18T11:44:47.211534+0000","last_fullsized":"2024-09-18T11:44:47.118954+0000","mapping_epoch":30,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1075,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13091,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346732+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:05.346732+0000","last_peered":"2024-09-18T11:58:05.346732+0000","last_clean":"2024-09-18T11:58:05.346732+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:05.346732+0000","last_undegraded":"2024-09-18T11:58:05.346732+0000","last_fullsized":"2024-09-18T11:58:05.346732+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13232,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346816+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:05.346816+0000","last_peered":"2024-09-18T11:58:05.346816+0000","last_clean":"2024-09-18T11:58:05.346816+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:05.346816+0000","last_undegraded":"2024-09-18T11:58:05.346816+0000","last_fullsized":"2024-09-18T11:58:05.346816+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17212,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346813+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:05.346813+0000","last_peered":"2024-09-18T11:58:05.346813+0000","last_clean":"2024-09-18T11:58:05.346813+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:05.346813+0000","last_undegraded":"2024-09-18T11:58:05.346813+0000","last_fullsized":"2024-09-18T11:58:05.346813+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12828,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T11:58:05.346695+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:05.346695+0000","last_peered":"2024-09-18T11:58:05.346695+0000","last_clean":"2024-09-18T11:58:05.346695+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:05.346695+0000","last_undegraded":"2024-09-18T11:58:05.346695+0000","last_fullsized":"2024-09-18T11:58:05.346695+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26200,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924020,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447820,"statfs":{"total":96636764160,"available":95690567680,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.27600000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.25},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.28899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919348,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452492,"statfs":{"total":96636764160,"available":95695351808,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.16400000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.152},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.23100000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.187},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29599999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930468,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441372,"statfs":{"total":96636764160,"available":95683964928,"internally_reserved":0,"allocated":815104,"data_stored":534494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.216},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.26300000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.25},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.32800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915640,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456200,"statfs":{"total":96636764160,"available":95699148800,"internally_reserved":0,"allocated":503808,"data_stored":218587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.214}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.255}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:19.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:58:19.870 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 13485, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:15.332651+0000', 'last_change': '2024-09-18T11:58:14.912652+0000', 'last_active': '2024-09-18T11:58:15.332651+0000', 'last_peered': '2024-09-18T11:58:15.332651+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:58:15.332651+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2048, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 82, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2048}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:19.871 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-18T11:58:19.871 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'13030", 'reported_seq': 15244, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:10.332973+0000', 'last_change': '2024-09-18T11:58:04.532453+0000', 'last_active': '2024-09-18T11:58:10.332973+0000', 'last_peered': '2024-09-18T11:58:10.332973+0000', 'last_clean': '2024-09-18T11:43:39.896884+0000', 'last_became_active': '2024-09-18T11:44:47.480227+0000', 'last_became_peered': '2024-09-18T11:44:47.480227+0000', 'last_unstale': '2024-09-18T11:58:10.332973+0000', 'last_undegraded': '2024-09-18T11:44:47.211534+0000', 'last_fullsized': '2024-09-18T11:44:47.118954+0000', 'mapping_epoch': 30, 'log_start': "31'3000", 'ondisk_log_start': "31'3000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6779, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 52119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1075, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1035}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:19.871 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:20.224 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:20.224 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2048/112 objects degraded (1828.571%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":898,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3769536512,"bytes_avail":382777520128,"bytes_total":386547056640,"degraded_objects":2048,"degraded_total":112,"degraded_ratio":18.285714285714285},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:20.237 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T11:58:20.237 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:20.517 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:20.517 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:20.529 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-09-18T11:58:20.129896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25151,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681188,"kb_used_data":2288,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806172,"statfs":{"total":386547056640,"available":382777520128,"internally_reserved":0,"allocated":2342912,"data_stored":1197375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-994,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001481"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":13485,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:15.332651+0000","last_change":"2024-09-18T11:58:14.912652+0000","last_active":"2024-09-18T11:58:15.332651+0000","last_peered":"2024-09-18T11:58:15.332651+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:15.332651+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17370,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199184+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:15.199184+0000","last_peered":"2024-09-18T11:58:15.199184+0000","last_clean":"2024-09-18T11:58:15.199184+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:15.199184+0000","last_undegraded":"2024-09-18T11:58:15.199184+0000","last_fullsized":"2024-09-18T11:58:15.199184+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17044,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197639+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:15.197639+0000","last_peered":"2024-09-18T11:58:15.197639+0000","last_clean":"2024-09-18T11:58:15.197639+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:15.197639+0000","last_undegraded":"2024-09-18T11:58:15.197639+0000","last_fullsized":"2024-09-18T11:58:15.197639+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13045,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198751+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:15.198751+0000","last_peered":"2024-09-18T11:58:15.198751+0000","last_clean":"2024-09-18T11:58:15.198751+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:15.198751+0000","last_undegraded":"2024-09-18T11:58:15.198751+0000","last_fullsized":"2024-09-18T11:58:15.198751+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17158,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.421831+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:06.421831+0000","last_peered":"2024-09-18T11:58:06.421831+0000","last_clean":"2024-09-18T11:58:06.421831+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:06.421831+0000","last_undegraded":"2024-09-18T11:58:06.421831+0000","last_fullsized":"2024-09-18T11:58:06.421831+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17179,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353256+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:06.353256+0000","last_peered":"2024-09-18T11:58:06.353256+0000","last_clean":"2024-09-18T11:58:06.353256+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:06.353256+0000","last_undegraded":"2024-09-18T11:58:06.353256+0000","last_fullsized":"2024-09-18T11:58:06.353256+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17393,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.351660+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:06.351660+0000","last_peered":"2024-09-18T11:58:06.351660+0000","last_clean":"2024-09-18T11:58:06.351660+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:06.351660+0000","last_undegraded":"2024-09-18T11:58:06.351660+0000","last_fullsized":"2024-09-18T11:58:06.351660+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17051,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198924+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:15.198924+0000","last_peered":"2024-09-18T11:58:15.198924+0000","last_clean":"2024-09-18T11:58:15.198924+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:15.198924+0000","last_undegraded":"2024-09-18T11:58:15.198924+0000","last_fullsized":"2024-09-18T11:58:15.198924+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17548,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353176+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:06.353176+0000","last_peered":"2024-09-18T11:58:06.353176+0000","last_clean":"2024-09-18T11:58:06.353176+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:06.353176+0000","last_undegraded":"2024-09-18T11:58:06.353176+0000","last_fullsized":"2024-09-18T11:58:06.353176+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16673,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197695+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:15.197695+0000","last_peered":"2024-09-18T11:58:15.197695+0000","last_clean":"2024-09-18T11:58:15.197695+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:15.197695+0000","last_undegraded":"2024-09-18T11:58:15.197695+0000","last_fullsized":"2024-09-18T11:58:15.197695+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T11:58:06.353476+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:06.353476+0000","last_peered":"2024-09-18T11:58:06.353476+0000","last_clean":"2024-09-18T11:58:06.353476+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:06.353476+0000","last_undegraded":"2024-09-18T11:58:06.353476+0000","last_fullsized":"2024-09-18T11:58:06.353476+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13140,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T11:58:16.269218+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:16.269218+0000","last_peered":"2024-09-18T11:58:16.269218+0000","last_clean":"2024-09-18T11:58:16.269218+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:16.269218+0000","last_undegraded":"2024-09-18T11:58:16.269218+0000","last_fullsized":"2024-09-18T11:58:16.269218+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13093,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198830+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:15.198830+0000","last_peered":"2024-09-18T11:58:15.198830+0000","last_clean":"2024-09-18T11:58:15.198830+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:15.198830+0000","last_undegraded":"2024-09-18T11:58:15.198830+0000","last_fullsized":"2024-09-18T11:58:15.198830+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13234,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198783+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:15.198783+0000","last_peered":"2024-09-18T11:58:15.198783+0000","last_clean":"2024-09-18T11:58:15.198783+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:15.198783+0000","last_undegraded":"2024-09-18T11:58:15.198783+0000","last_fullsized":"2024-09-18T11:58:15.198783+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17214,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198828+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:15.198828+0000","last_peered":"2024-09-18T11:58:15.198828+0000","last_clean":"2024-09-18T11:58:15.198828+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:15.198828+0000","last_undegraded":"2024-09-18T11:58:15.198828+0000","last_fullsized":"2024-09-18T11:58:15.198828+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12830,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199081+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:15.199081+0000","last_peered":"2024-09-18T11:58:15.199081+0000","last_clean":"2024-09-18T11:58:15.199081+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:15.199081+0000","last_undegraded":"2024-09-18T11:58:15.199081+0000","last_fullsized":"2024-09-18T11:58:15.199081+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25151,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924020,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447820,"statfs":{"total":96636764160,"available":95690567680,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.27600000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.26200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.25},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.28899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919360,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452480,"statfs":{"total":96636764160,"available":95695339520,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.215},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.27800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.248}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922168,"kb_used_data":812,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449672,"statfs":{"total":96636764160,"available":95692464128,"internally_reserved":0,"allocated":831488,"data_stored":541635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.20599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915640,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456200,"statfs":{"total":96636764160,"available":95699148800,"internally_reserved":0,"allocated":503808,"data_stored":218587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.214}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.255}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:23.530 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:23.809 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:23.821 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-18T11:58:22.130200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681208,"kb_used_data":2312,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806152,"statfs":{"total":386547056640,"available":382777499648,"internally_reserved":0,"allocated":2367488,"data_stored":1208076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001453"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":15258,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:20.332676+0000","last_change":"2024-09-18T11:58:14.912652+0000","last_active":"2024-09-18T11:58:20.332676+0000","last_peered":"2024-09-18T11:58:20.332676+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:20.332676+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17370,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199184+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:15.199184+0000","last_peered":"2024-09-18T11:58:15.199184+0000","last_clean":"2024-09-18T11:58:15.199184+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:15.199184+0000","last_undegraded":"2024-09-18T11:58:15.199184+0000","last_fullsized":"2024-09-18T11:58:15.199184+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17044,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197639+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:15.197639+0000","last_peered":"2024-09-18T11:58:15.197639+0000","last_clean":"2024-09-18T11:58:15.197639+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:15.197639+0000","last_undegraded":"2024-09-18T11:58:15.197639+0000","last_fullsized":"2024-09-18T11:58:15.197639+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13045,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198751+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:15.198751+0000","last_peered":"2024-09-18T11:58:15.198751+0000","last_clean":"2024-09-18T11:58:15.198751+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:15.198751+0000","last_undegraded":"2024-09-18T11:58:15.198751+0000","last_fullsized":"2024-09-18T11:58:15.198751+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17159,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198816+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:15.198816+0000","last_peered":"2024-09-18T11:58:15.198816+0000","last_clean":"2024-09-18T11:58:15.198816+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:15.198816+0000","last_undegraded":"2024-09-18T11:58:15.198816+0000","last_fullsized":"2024-09-18T11:58:15.198816+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17180,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199181+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:15.199181+0000","last_peered":"2024-09-18T11:58:15.199181+0000","last_clean":"2024-09-18T11:58:15.199181+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:15.199181+0000","last_undegraded":"2024-09-18T11:58:15.199181+0000","last_fullsized":"2024-09-18T11:58:15.199181+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17394,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.195781+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:15.195781+0000","last_peered":"2024-09-18T11:58:15.195781+0000","last_clean":"2024-09-18T11:58:15.195781+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:15.195781+0000","last_undegraded":"2024-09-18T11:58:15.195781+0000","last_fullsized":"2024-09-18T11:58:15.195781+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17051,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198924+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:15.198924+0000","last_peered":"2024-09-18T11:58:15.198924+0000","last_clean":"2024-09-18T11:58:15.198924+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:15.198924+0000","last_undegraded":"2024-09-18T11:58:15.198924+0000","last_fullsized":"2024-09-18T11:58:15.198924+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17549,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199271+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:15.199271+0000","last_peered":"2024-09-18T11:58:15.199271+0000","last_clean":"2024-09-18T11:58:15.199271+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:15.199271+0000","last_undegraded":"2024-09-18T11:58:15.199271+0000","last_fullsized":"2024-09-18T11:58:15.199271+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16673,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197695+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:15.197695+0000","last_peered":"2024-09-18T11:58:15.197695+0000","last_clean":"2024-09-18T11:58:15.197695+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:15.197695+0000","last_undegraded":"2024-09-18T11:58:15.197695+0000","last_fullsized":"2024-09-18T11:58:15.197695+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199025+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:15.199025+0000","last_peered":"2024-09-18T11:58:15.199025+0000","last_clean":"2024-09-18T11:58:15.199025+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:15.199025+0000","last_undegraded":"2024-09-18T11:58:15.199025+0000","last_fullsized":"2024-09-18T11:58:15.199025+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13140,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T11:58:16.269218+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:16.269218+0000","last_peered":"2024-09-18T11:58:16.269218+0000","last_clean":"2024-09-18T11:58:16.269218+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:16.269218+0000","last_undegraded":"2024-09-18T11:58:16.269218+0000","last_fullsized":"2024-09-18T11:58:16.269218+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13093,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198830+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:15.198830+0000","last_peered":"2024-09-18T11:58:15.198830+0000","last_clean":"2024-09-18T11:58:15.198830+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:15.198830+0000","last_undegraded":"2024-09-18T11:58:15.198830+0000","last_fullsized":"2024-09-18T11:58:15.198830+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13234,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198783+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:15.198783+0000","last_peered":"2024-09-18T11:58:15.198783+0000","last_clean":"2024-09-18T11:58:15.198783+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:15.198783+0000","last_undegraded":"2024-09-18T11:58:15.198783+0000","last_fullsized":"2024-09-18T11:58:15.198783+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17214,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198828+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:15.198828+0000","last_peered":"2024-09-18T11:58:15.198828+0000","last_clean":"2024-09-18T11:58:15.198828+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:15.198828+0000","last_undegraded":"2024-09-18T11:58:15.198828+0000","last_fullsized":"2024-09-18T11:58:15.198828+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12830,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199081+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:15.199081+0000","last_peered":"2024-09-18T11:58:15.199081+0000","last_clean":"2024-09-18T11:58:15.199081+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:15.199081+0000","last_undegraded":"2024-09-18T11:58:15.199081+0000","last_fullsized":"2024-09-18T11:58:15.199081+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924032,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447808,"statfs":{"total":96636764160,"available":95690555392,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.22600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.309}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919360,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452480,"statfs":{"total":96636764160,"available":95695339520,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.215},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.27800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.248}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922168,"kb_used_data":812,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449672,"statfs":{"total":96636764160,"available":95692464128,"internally_reserved":0,"allocated":831488,"data_stored":541635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.20599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915648,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456192,"statfs":{"total":96636764160,"available":95699140608,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.252}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:23.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:24.095 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:24.095 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:24.107 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-18T11:58:22.130200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3681208,"kb_used_data":2312,"kb_used_omap":14,"kb_used_meta":3678833,"kb_avail":373806152,"statfs":{"total":386547056640,"available":382777499648,"internally_reserved":0,"allocated":2367488,"data_stored":1208076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3767125019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001453"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":15258,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:20.332676+0000","last_change":"2024-09-18T11:58:14.912652+0000","last_active":"2024-09-18T11:58:20.332676+0000","last_peered":"2024-09-18T11:58:20.332676+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:20.332676+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17370,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199184+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:15.199184+0000","last_peered":"2024-09-18T11:58:15.199184+0000","last_clean":"2024-09-18T11:58:15.199184+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:15.199184+0000","last_undegraded":"2024-09-18T11:58:15.199184+0000","last_fullsized":"2024-09-18T11:58:15.199184+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17044,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197639+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:15.197639+0000","last_peered":"2024-09-18T11:58:15.197639+0000","last_clean":"2024-09-18T11:58:15.197639+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:15.197639+0000","last_undegraded":"2024-09-18T11:58:15.197639+0000","last_fullsized":"2024-09-18T11:58:15.197639+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13045,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198751+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:15.198751+0000","last_peered":"2024-09-18T11:58:15.198751+0000","last_clean":"2024-09-18T11:58:15.198751+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:15.198751+0000","last_undegraded":"2024-09-18T11:58:15.198751+0000","last_fullsized":"2024-09-18T11:58:15.198751+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17159,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198816+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:15.198816+0000","last_peered":"2024-09-18T11:58:15.198816+0000","last_clean":"2024-09-18T11:58:15.198816+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:15.198816+0000","last_undegraded":"2024-09-18T11:58:15.198816+0000","last_fullsized":"2024-09-18T11:58:15.198816+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17180,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199181+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:15.199181+0000","last_peered":"2024-09-18T11:58:15.199181+0000","last_clean":"2024-09-18T11:58:15.199181+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:15.199181+0000","last_undegraded":"2024-09-18T11:58:15.199181+0000","last_fullsized":"2024-09-18T11:58:15.199181+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17394,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.195781+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:15.195781+0000","last_peered":"2024-09-18T11:58:15.195781+0000","last_clean":"2024-09-18T11:58:15.195781+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:15.195781+0000","last_undegraded":"2024-09-18T11:58:15.195781+0000","last_fullsized":"2024-09-18T11:58:15.195781+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17051,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198924+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:15.198924+0000","last_peered":"2024-09-18T11:58:15.198924+0000","last_clean":"2024-09-18T11:58:15.198924+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:15.198924+0000","last_undegraded":"2024-09-18T11:58:15.198924+0000","last_fullsized":"2024-09-18T11:58:15.198924+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17549,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199271+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:15.199271+0000","last_peered":"2024-09-18T11:58:15.199271+0000","last_clean":"2024-09-18T11:58:15.199271+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:15.199271+0000","last_undegraded":"2024-09-18T11:58:15.199271+0000","last_fullsized":"2024-09-18T11:58:15.199271+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16673,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197695+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:15.197695+0000","last_peered":"2024-09-18T11:58:15.197695+0000","last_clean":"2024-09-18T11:58:15.197695+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:15.197695+0000","last_undegraded":"2024-09-18T11:58:15.197695+0000","last_fullsized":"2024-09-18T11:58:15.197695+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199025+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:15.199025+0000","last_peered":"2024-09-18T11:58:15.199025+0000","last_clean":"2024-09-18T11:58:15.199025+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:15.199025+0000","last_undegraded":"2024-09-18T11:58:15.199025+0000","last_fullsized":"2024-09-18T11:58:15.199025+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13140,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T11:58:16.269218+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:16.269218+0000","last_peered":"2024-09-18T11:58:16.269218+0000","last_clean":"2024-09-18T11:58:16.269218+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:16.269218+0000","last_undegraded":"2024-09-18T11:58:16.269218+0000","last_fullsized":"2024-09-18T11:58:16.269218+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13093,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198830+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:15.198830+0000","last_peered":"2024-09-18T11:58:15.198830+0000","last_clean":"2024-09-18T11:58:15.198830+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:15.198830+0000","last_undegraded":"2024-09-18T11:58:15.198830+0000","last_fullsized":"2024-09-18T11:58:15.198830+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13234,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198783+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:15.198783+0000","last_peered":"2024-09-18T11:58:15.198783+0000","last_clean":"2024-09-18T11:58:15.198783+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:15.198783+0000","last_undegraded":"2024-09-18T11:58:15.198783+0000","last_fullsized":"2024-09-18T11:58:15.198783+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17214,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198828+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:15.198828+0000","last_peered":"2024-09-18T11:58:15.198828+0000","last_clean":"2024-09-18T11:58:15.198828+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:15.198828+0000","last_undegraded":"2024-09-18T11:58:15.198828+0000","last_fullsized":"2024-09-18T11:58:15.198828+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12830,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199081+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:15.199081+0000","last_peered":"2024-09-18T11:58:15.199081+0000","last_clean":"2024-09-18T11:58:15.199081+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:15.199081+0000","last_undegraded":"2024-09-18T11:58:15.199081+0000","last_fullsized":"2024-09-18T11:58:15.199081+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924032,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447808,"statfs":{"total":96636764160,"available":95690555392,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.22600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.309}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919360,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452480,"statfs":{"total":96636764160,"available":95695339520,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.215},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.27800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.248}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922168,"kb_used_data":812,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449672,"statfs":{"total":96636764160,"available":95692464128,"internally_reserved":0,"allocated":831488,"data_stored":541635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.20599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915648,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456192,"statfs":{"total":96636764160,"available":95699140608,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.252}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:24.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:58:24.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 15258, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:20.332676+0000', 'last_change': '2024-09-18T11:58:14.912652+0000', 'last_active': '2024-09-18T11:58:20.332676+0000', 'last_peered': '2024-09-18T11:58:20.332676+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:58:20.332676+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1162, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 968, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1162}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:24.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:24.463 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1162/112 objects degraded (1037.500%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":902,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3778072576,"bytes_avail":382768984064,"bytes_total":386547056640,"degraded_objects":1162,"degraded_total":112,"degraded_ratio":10.375},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:24.476 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T11:58:24.476 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:24.755 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:24.756 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:24.767 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-18T11:58:24.130551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689524,"kb_used_data":2312,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797836,"statfs":{"total":386547056640,"available":382768984064,"internally_reserved":0,"allocated":2367488,"data_stored":1208076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.936244"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":15258,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:20.332676+0000","last_change":"2024-09-18T11:58:14.912652+0000","last_active":"2024-09-18T11:58:20.332676+0000","last_peered":"2024-09-18T11:58:20.332676+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:20.332676+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17370,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199184+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:15.199184+0000","last_peered":"2024-09-18T11:58:15.199184+0000","last_clean":"2024-09-18T11:58:15.199184+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:15.199184+0000","last_undegraded":"2024-09-18T11:58:15.199184+0000","last_fullsized":"2024-09-18T11:58:15.199184+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17044,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197639+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:15.197639+0000","last_peered":"2024-09-18T11:58:15.197639+0000","last_clean":"2024-09-18T11:58:15.197639+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:15.197639+0000","last_undegraded":"2024-09-18T11:58:15.197639+0000","last_fullsized":"2024-09-18T11:58:15.197639+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13045,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198751+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:15.198751+0000","last_peered":"2024-09-18T11:58:15.198751+0000","last_clean":"2024-09-18T11:58:15.198751+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:15.198751+0000","last_undegraded":"2024-09-18T11:58:15.198751+0000","last_fullsized":"2024-09-18T11:58:15.198751+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17159,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198816+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:15.198816+0000","last_peered":"2024-09-18T11:58:15.198816+0000","last_clean":"2024-09-18T11:58:15.198816+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:15.198816+0000","last_undegraded":"2024-09-18T11:58:15.198816+0000","last_fullsized":"2024-09-18T11:58:15.198816+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17180,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199181+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:15.199181+0000","last_peered":"2024-09-18T11:58:15.199181+0000","last_clean":"2024-09-18T11:58:15.199181+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:15.199181+0000","last_undegraded":"2024-09-18T11:58:15.199181+0000","last_fullsized":"2024-09-18T11:58:15.199181+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17394,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.195781+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:15.195781+0000","last_peered":"2024-09-18T11:58:15.195781+0000","last_clean":"2024-09-18T11:58:15.195781+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:15.195781+0000","last_undegraded":"2024-09-18T11:58:15.195781+0000","last_fullsized":"2024-09-18T11:58:15.195781+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17051,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198924+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:15.198924+0000","last_peered":"2024-09-18T11:58:15.198924+0000","last_clean":"2024-09-18T11:58:15.198924+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:15.198924+0000","last_undegraded":"2024-09-18T11:58:15.198924+0000","last_fullsized":"2024-09-18T11:58:15.198924+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17549,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199271+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:15.199271+0000","last_peered":"2024-09-18T11:58:15.199271+0000","last_clean":"2024-09-18T11:58:15.199271+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:15.199271+0000","last_undegraded":"2024-09-18T11:58:15.199271+0000","last_fullsized":"2024-09-18T11:58:15.199271+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16673,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197695+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:15.197695+0000","last_peered":"2024-09-18T11:58:15.197695+0000","last_clean":"2024-09-18T11:58:15.197695+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:15.197695+0000","last_undegraded":"2024-09-18T11:58:15.197695+0000","last_fullsized":"2024-09-18T11:58:15.197695+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199025+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:15.199025+0000","last_peered":"2024-09-18T11:58:15.199025+0000","last_clean":"2024-09-18T11:58:15.199025+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:15.199025+0000","last_undegraded":"2024-09-18T11:58:15.199025+0000","last_fullsized":"2024-09-18T11:58:15.199025+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13140,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T11:58:16.269218+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:16.269218+0000","last_peered":"2024-09-18T11:58:16.269218+0000","last_clean":"2024-09-18T11:58:16.269218+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:16.269218+0000","last_undegraded":"2024-09-18T11:58:16.269218+0000","last_fullsized":"2024-09-18T11:58:16.269218+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13093,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198830+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:15.198830+0000","last_peered":"2024-09-18T11:58:15.198830+0000","last_clean":"2024-09-18T11:58:15.198830+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:15.198830+0000","last_undegraded":"2024-09-18T11:58:15.198830+0000","last_fullsized":"2024-09-18T11:58:15.198830+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13234,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198783+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:15.198783+0000","last_peered":"2024-09-18T11:58:15.198783+0000","last_clean":"2024-09-18T11:58:15.198783+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:15.198783+0000","last_undegraded":"2024-09-18T11:58:15.198783+0000","last_fullsized":"2024-09-18T11:58:15.198783+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17214,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198828+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:15.198828+0000","last_peered":"2024-09-18T11:58:15.198828+0000","last_clean":"2024-09-18T11:58:15.198828+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:15.198828+0000","last_undegraded":"2024-09-18T11:58:15.198828+0000","last_fullsized":"2024-09-18T11:58:15.198828+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12830,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199081+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:15.199081+0000","last_peered":"2024-09-18T11:58:15.199081+0000","last_clean":"2024-09-18T11:58:15.199081+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:15.199081+0000","last_undegraded":"2024-09-18T11:58:15.199081+0000","last_fullsized":"2024-09-18T11:58:15.199081+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26037,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924032,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447808,"statfs":{"total":96636764160,"available":95690555392,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.19},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.22600000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.309}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.23599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919360,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452480,"statfs":{"total":96636764160,"available":95695339520,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.215},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.27800000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.248}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.29999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930484,"kb_used_data":812,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441356,"statfs":{"total":96636764160,"available":95683948544,"internally_reserved":0,"allocated":831488,"data_stored":541635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.192}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.30099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915648,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456192,"statfs":{"total":96636764160,"available":95699140608,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.252}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:27.769 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:28.036 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:28.037 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:28.048 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-09-18T11:58:27.270941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27147,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689524,"kb_used_data":2312,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797836,"statfs":{"total":386547056640,"available":382768984064,"internally_reserved":0,"allocated":2367488,"data_stored":1208076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.141313"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17478,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:25.334315+0000","last_change":"2024-09-18T11:58:14.912652+0000","last_active":"2024-09-18T11:58:25.334315+0000","last_peered":"2024-09-18T11:58:25.334315+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:25.334315+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2078,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-52}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17370,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199184+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:15.199184+0000","last_peered":"2024-09-18T11:58:15.199184+0000","last_clean":"2024-09-18T11:58:15.199184+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:15.199184+0000","last_undegraded":"2024-09-18T11:58:15.199184+0000","last_fullsized":"2024-09-18T11:58:15.199184+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17044,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197639+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:15.197639+0000","last_peered":"2024-09-18T11:58:15.197639+0000","last_clean":"2024-09-18T11:58:15.197639+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:15.197639+0000","last_undegraded":"2024-09-18T11:58:15.197639+0000","last_fullsized":"2024-09-18T11:58:15.197639+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13045,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198751+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:15.198751+0000","last_peered":"2024-09-18T11:58:15.198751+0000","last_clean":"2024-09-18T11:58:15.198751+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:15.198751+0000","last_undegraded":"2024-09-18T11:58:15.198751+0000","last_fullsized":"2024-09-18T11:58:15.198751+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17159,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198816+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:15.198816+0000","last_peered":"2024-09-18T11:58:15.198816+0000","last_clean":"2024-09-18T11:58:15.198816+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:15.198816+0000","last_undegraded":"2024-09-18T11:58:15.198816+0000","last_fullsized":"2024-09-18T11:58:15.198816+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17180,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199181+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:15.199181+0000","last_peered":"2024-09-18T11:58:15.199181+0000","last_clean":"2024-09-18T11:58:15.199181+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:15.199181+0000","last_undegraded":"2024-09-18T11:58:15.199181+0000","last_fullsized":"2024-09-18T11:58:15.199181+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17394,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.195781+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:15.195781+0000","last_peered":"2024-09-18T11:58:15.195781+0000","last_clean":"2024-09-18T11:58:15.195781+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:15.195781+0000","last_undegraded":"2024-09-18T11:58:15.195781+0000","last_fullsized":"2024-09-18T11:58:15.195781+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17051,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198924+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:15.198924+0000","last_peered":"2024-09-18T11:58:15.198924+0000","last_clean":"2024-09-18T11:58:15.198924+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:15.198924+0000","last_undegraded":"2024-09-18T11:58:15.198924+0000","last_fullsized":"2024-09-18T11:58:15.198924+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17549,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199271+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:15.199271+0000","last_peered":"2024-09-18T11:58:15.199271+0000","last_clean":"2024-09-18T11:58:15.199271+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:15.199271+0000","last_undegraded":"2024-09-18T11:58:15.199271+0000","last_fullsized":"2024-09-18T11:58:15.199271+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16673,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197695+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:15.197695+0000","last_peered":"2024-09-18T11:58:15.197695+0000","last_clean":"2024-09-18T11:58:15.197695+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:15.197695+0000","last_undegraded":"2024-09-18T11:58:15.197695+0000","last_fullsized":"2024-09-18T11:58:15.197695+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199025+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:15.199025+0000","last_peered":"2024-09-18T11:58:15.199025+0000","last_clean":"2024-09-18T11:58:15.199025+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:15.199025+0000","last_undegraded":"2024-09-18T11:58:15.199025+0000","last_fullsized":"2024-09-18T11:58:15.199025+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13140,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T11:58:16.269218+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:16.269218+0000","last_peered":"2024-09-18T11:58:16.269218+0000","last_clean":"2024-09-18T11:58:16.269218+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:16.269218+0000","last_undegraded":"2024-09-18T11:58:16.269218+0000","last_fullsized":"2024-09-18T11:58:16.269218+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13093,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198830+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:15.198830+0000","last_peered":"2024-09-18T11:58:15.198830+0000","last_clean":"2024-09-18T11:58:15.198830+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:15.198830+0000","last_undegraded":"2024-09-18T11:58:15.198830+0000","last_fullsized":"2024-09-18T11:58:15.198830+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13234,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198783+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:15.198783+0000","last_peered":"2024-09-18T11:58:15.198783+0000","last_clean":"2024-09-18T11:58:15.198783+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:15.198783+0000","last_undegraded":"2024-09-18T11:58:15.198783+0000","last_fullsized":"2024-09-18T11:58:15.198783+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17214,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198828+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:15.198828+0000","last_peered":"2024-09-18T11:58:15.198828+0000","last_clean":"2024-09-18T11:58:15.198828+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:15.198828+0000","last_undegraded":"2024-09-18T11:58:15.198828+0000","last_fullsized":"2024-09-18T11:58:15.198828+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12830,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199081+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:15.199081+0000","last_peered":"2024-09-18T11:58:15.199081+0000","last_clean":"2024-09-18T11:58:15.199081+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:15.199081+0000","last_undegraded":"2024-09-18T11:58:15.199081+0000","last_fullsized":"2024-09-18T11:58:15.199081+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27147,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924032,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447808,"statfs":{"total":96636764160,"available":95690555392,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.245}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.25800000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.29199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919360,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452480,"statfs":{"total":96636764160,"available":95695339520,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.252}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.183},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.22900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.214},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27100000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930484,"kb_used_data":812,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441356,"statfs":{"total":96636764160,"available":95683948544,"internally_reserved":0,"allocated":831488,"data_stored":541635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.192}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.30099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915648,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456192,"statfs":{"total":96636764160,"available":95699140608,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.252}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:28.049 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:28.307 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:28.308 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:28.319 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-18T11:58:28.131239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27147,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689524,"kb_used_data":2312,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797836,"statfs":{"total":386547056640,"available":382768984064,"internally_reserved":0,"allocated":2367488,"data_stored":1208076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1996,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001342"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17478,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:25.334315+0000","last_change":"2024-09-18T11:58:14.912652+0000","last_active":"2024-09-18T11:58:25.334315+0000","last_peered":"2024-09-18T11:58:25.334315+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:25.334315+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2078,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-52}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17370,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199184+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:15.199184+0000","last_peered":"2024-09-18T11:58:15.199184+0000","last_clean":"2024-09-18T11:58:15.199184+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:15.199184+0000","last_undegraded":"2024-09-18T11:58:15.199184+0000","last_fullsized":"2024-09-18T11:58:15.199184+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17044,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197639+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:15.197639+0000","last_peered":"2024-09-18T11:58:15.197639+0000","last_clean":"2024-09-18T11:58:15.197639+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:15.197639+0000","last_undegraded":"2024-09-18T11:58:15.197639+0000","last_fullsized":"2024-09-18T11:58:15.197639+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13045,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198751+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:15.198751+0000","last_peered":"2024-09-18T11:58:15.198751+0000","last_clean":"2024-09-18T11:58:15.198751+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:15.198751+0000","last_undegraded":"2024-09-18T11:58:15.198751+0000","last_fullsized":"2024-09-18T11:58:15.198751+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17159,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198816+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:15.198816+0000","last_peered":"2024-09-18T11:58:15.198816+0000","last_clean":"2024-09-18T11:58:15.198816+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:15.198816+0000","last_undegraded":"2024-09-18T11:58:15.198816+0000","last_fullsized":"2024-09-18T11:58:15.198816+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17180,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199181+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:15.199181+0000","last_peered":"2024-09-18T11:58:15.199181+0000","last_clean":"2024-09-18T11:58:15.199181+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:15.199181+0000","last_undegraded":"2024-09-18T11:58:15.199181+0000","last_fullsized":"2024-09-18T11:58:15.199181+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17394,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.195781+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:15.195781+0000","last_peered":"2024-09-18T11:58:15.195781+0000","last_clean":"2024-09-18T11:58:15.195781+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:15.195781+0000","last_undegraded":"2024-09-18T11:58:15.195781+0000","last_fullsized":"2024-09-18T11:58:15.195781+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17051,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198924+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:15.198924+0000","last_peered":"2024-09-18T11:58:15.198924+0000","last_clean":"2024-09-18T11:58:15.198924+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:15.198924+0000","last_undegraded":"2024-09-18T11:58:15.198924+0000","last_fullsized":"2024-09-18T11:58:15.198924+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17549,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199271+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:15.199271+0000","last_peered":"2024-09-18T11:58:15.199271+0000","last_clean":"2024-09-18T11:58:15.199271+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:15.199271+0000","last_undegraded":"2024-09-18T11:58:15.199271+0000","last_fullsized":"2024-09-18T11:58:15.199271+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16673,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197695+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:15.197695+0000","last_peered":"2024-09-18T11:58:15.197695+0000","last_clean":"2024-09-18T11:58:15.197695+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:15.197695+0000","last_undegraded":"2024-09-18T11:58:15.197695+0000","last_fullsized":"2024-09-18T11:58:15.197695+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199025+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:15.199025+0000","last_peered":"2024-09-18T11:58:15.199025+0000","last_clean":"2024-09-18T11:58:15.199025+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:15.199025+0000","last_undegraded":"2024-09-18T11:58:15.199025+0000","last_fullsized":"2024-09-18T11:58:15.199025+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13140,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T11:58:16.269218+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:16.269218+0000","last_peered":"2024-09-18T11:58:16.269218+0000","last_clean":"2024-09-18T11:58:16.269218+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:16.269218+0000","last_undegraded":"2024-09-18T11:58:16.269218+0000","last_fullsized":"2024-09-18T11:58:16.269218+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13093,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198830+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:15.198830+0000","last_peered":"2024-09-18T11:58:15.198830+0000","last_clean":"2024-09-18T11:58:15.198830+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:15.198830+0000","last_undegraded":"2024-09-18T11:58:15.198830+0000","last_fullsized":"2024-09-18T11:58:15.198830+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13234,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198783+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:15.198783+0000","last_peered":"2024-09-18T11:58:15.198783+0000","last_clean":"2024-09-18T11:58:15.198783+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:15.198783+0000","last_undegraded":"2024-09-18T11:58:15.198783+0000","last_fullsized":"2024-09-18T11:58:15.198783+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17214,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198828+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:15.198828+0000","last_peered":"2024-09-18T11:58:15.198828+0000","last_clean":"2024-09-18T11:58:15.198828+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:15.198828+0000","last_undegraded":"2024-09-18T11:58:15.198828+0000","last_fullsized":"2024-09-18T11:58:15.198828+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12830,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199081+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:15.199081+0000","last_peered":"2024-09-18T11:58:15.199081+0000","last_clean":"2024-09-18T11:58:15.199081+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:15.199081+0000","last_undegraded":"2024-09-18T11:58:15.199081+0000","last_fullsized":"2024-09-18T11:58:15.199081+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27147,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924032,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447808,"statfs":{"total":96636764160,"available":95690555392,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.245}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.25800000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.29199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919360,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452480,"statfs":{"total":96636764160,"available":95695339520,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.252}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.183},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.22900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.214},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27100000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930484,"kb_used_data":812,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441356,"statfs":{"total":96636764160,"available":95683948544,"internally_reserved":0,"allocated":831488,"data_stored":541635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.192}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.30099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915648,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456192,"statfs":{"total":96636764160,"available":95699140608,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.252}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:28.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-18T11:58:28.321 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'13250", 'reported_seq': 17478, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:25.334315+0000', 'last_change': '2024-09-18T11:58:14.912652+0000', 'last_active': '2024-09-18T11:58:25.334315+0000', 'last_peered': '2024-09-18T11:58:25.334315+0000', 'last_clean': '2024-09-18T11:43:36.867433+0000', 'last_became_active': '2024-09-18T11:44:47.509674+0000', 'last_became_peered': '2024-09-18T11:44:47.509674+0000', 'last_unstale': '2024-09-18T11:58:25.334315+0000', 'last_undegraded': '2024-09-18T11:44:47.183010+0000', 'last_fullsized': '2024-09-18T11:44:47.118689+0000', 'mapping_epoch': 30, 'log_start': "31'3200", 'ondisk_log_start': "31'3200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'last_clean_scrub_stamp': '2024-09-18T11:43:36.867433+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 52, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13250, 'num_write_kb': 53000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2078, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -52}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T11:58:28.321 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:28.491 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-09-18T11:58:28.490+0000 7f98e13c5640 -1 osd.1 pg_epoch: 63 pg[1.4s3( v 39'13172 (31'3100,39'13172] local-lis/les=36/37 n=4 ec=13/13 lis/c=36/36 les/c/f=37/37/0 sis=36) [3,0,2,1]p3(0) r=3 lpr=36 luod=0'0 crt=39'13172 mlcod 39'13172 active mbc={}] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-18T11:58:28.657 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:28.657 INFO:teuthology.orchestra.run.smithi071.stdout:{"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 52/112 objects degraded (46.429%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":906,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":63,"num_osds":4,"num_up_osds":4,"osd_up_since":1726659885,"num_in_osds":4,"osd_in_since":1726659803,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":189812,"bytes_used":3778072576,"bytes_avail":382768984064,"bytes_total":386547056640,"degraded_objects":52,"degraded_total":112,"degraded_ratio":0.4642857142857143,"recovering_objects_per_sec":249,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1996,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T11:53:04.038263+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:28.669 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:28.669 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:28.924 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:28.925 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:28.936 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-18T11:58:28.131239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27147,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689524,"kb_used_data":2312,"kb_used_omap":14,"kb_used_meta":3687153,"kb_avail":373797836,"statfs":{"total":386547056640,"available":382768984064,"internally_reserved":0,"allocated":2367488,"data_stored":1208076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775644699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1996,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001342"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17478,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:25.334315+0000","last_change":"2024-09-18T11:58:14.912652+0000","last_active":"2024-09-18T11:58:25.334315+0000","last_peered":"2024-09-18T11:58:25.334315+0000","last_clean":"2024-09-18T11:43:36.867433+0000","last_became_active":"2024-09-18T11:44:47.509674+0000","last_became_peered":"2024-09-18T11:44:47.509674+0000","last_unstale":"2024-09-18T11:58:25.334315+0000","last_undegraded":"2024-09-18T11:44:47.183010+0000","last_fullsized":"2024-09-18T11:44:47.118689+0000","mapping_epoch":30,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:51:44.323094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2078,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-52}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17370,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199184+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:15.199184+0000","last_peered":"2024-09-18T11:58:15.199184+0000","last_clean":"2024-09-18T11:58:15.199184+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:15.199184+0000","last_undegraded":"2024-09-18T11:58:15.199184+0000","last_fullsized":"2024-09-18T11:58:15.199184+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17044,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197639+0000","last_change":"2024-09-18T11:56:30.259041+0000","last_active":"2024-09-18T11:58:15.197639+0000","last_peered":"2024-09-18T11:58:15.197639+0000","last_clean":"2024-09-18T11:58:15.197639+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:15.197639+0000","last_undegraded":"2024-09-18T11:58:15.197639+0000","last_fullsized":"2024-09-18T11:58:15.197639+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12807,"num_write_kb":51223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13045,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198751+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:15.198751+0000","last_peered":"2024-09-18T11:58:15.198751+0000","last_clean":"2024-09-18T11:58:15.198751+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:15.198751+0000","last_undegraded":"2024-09-18T11:58:15.198751+0000","last_fullsized":"2024-09-18T11:58:15.198751+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17159,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198816+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:15.198816+0000","last_peered":"2024-09-18T11:58:15.198816+0000","last_clean":"2024-09-18T11:58:15.198816+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:15.198816+0000","last_undegraded":"2024-09-18T11:58:15.198816+0000","last_fullsized":"2024-09-18T11:58:15.198816+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17180,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199181+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:15.199181+0000","last_peered":"2024-09-18T11:58:15.199181+0000","last_clean":"2024-09-18T11:58:15.199181+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:15.199181+0000","last_undegraded":"2024-09-18T11:58:15.199181+0000","last_fullsized":"2024-09-18T11:58:15.199181+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17394,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.195781+0000","last_change":"2024-09-18T11:57:01.376777+0000","last_active":"2024-09-18T11:58:15.195781+0000","last_peered":"2024-09-18T11:58:15.195781+0000","last_clean":"2024-09-18T11:58:15.195781+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:15.195781+0000","last_undegraded":"2024-09-18T11:58:15.195781+0000","last_fullsized":"2024-09-18T11:58:15.195781+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":52052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17051,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198924+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:15.198924+0000","last_peered":"2024-09-18T11:58:15.198924+0000","last_clean":"2024-09-18T11:58:15.198924+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:15.198924+0000","last_undegraded":"2024-09-18T11:58:15.198924+0000","last_fullsized":"2024-09-18T11:58:15.198924+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17549,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199271+0000","last_change":"2024-09-18T11:52:37.480475+0000","last_active":"2024-09-18T11:58:15.199271+0000","last_peered":"2024-09-18T11:58:15.199271+0000","last_clean":"2024-09-18T11:58:15.199271+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:15.199271+0000","last_undegraded":"2024-09-18T11:58:15.199271+0000","last_fullsized":"2024-09-18T11:58:15.199271+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13169,"num_write_kb":52689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":5251072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16673,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.197695+0000","last_change":"2024-09-18T11:57:44.298755+0000","last_active":"2024-09-18T11:58:15.197695+0000","last_peered":"2024-09-18T11:58:15.197695+0000","last_clean":"2024-09-18T11:58:15.197695+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:15.197695+0000","last_undegraded":"2024-09-18T11:58:15.197695+0000","last_fullsized":"2024-09-18T11:58:15.197695+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":50366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17262,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199025+0000","last_change":"2024-09-18T11:56:50.330277+0000","last_active":"2024-09-18T11:58:15.199025+0000","last_peered":"2024-09-18T11:58:15.199025+0000","last_clean":"2024-09-18T11:58:15.199025+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:15.199025+0000","last_undegraded":"2024-09-18T11:58:15.199025+0000","last_fullsized":"2024-09-18T11:58:15.199025+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":52116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13140,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T11:58:16.269218+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:16.269218+0000","last_peered":"2024-09-18T11:58:16.269218+0000","last_clean":"2024-09-18T11:58:16.269218+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:16.269218+0000","last_undegraded":"2024-09-18T11:58:16.269218+0000","last_fullsized":"2024-09-18T11:58:16.269218+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13093,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198830+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:15.198830+0000","last_peered":"2024-09-18T11:58:15.198830+0000","last_clean":"2024-09-18T11:58:15.198830+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:15.198830+0000","last_undegraded":"2024-09-18T11:58:15.198830+0000","last_fullsized":"2024-09-18T11:58:15.198830+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13234,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198783+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:15.198783+0000","last_peered":"2024-09-18T11:58:15.198783+0000","last_clean":"2024-09-18T11:58:15.198783+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:15.198783+0000","last_undegraded":"2024-09-18T11:58:15.198783+0000","last_fullsized":"2024-09-18T11:58:15.198783+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17214,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.198828+0000","last_change":"2024-09-18T11:47:02.098591+0000","last_active":"2024-09-18T11:58:15.198828+0000","last_peered":"2024-09-18T11:58:15.198828+0000","last_clean":"2024-09-18T11:58:15.198828+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:15.198828+0000","last_undegraded":"2024-09-18T11:58:15.198828+0000","last_fullsized":"2024-09-18T11:58:15.198828+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":51605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17129472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12830,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T11:58:15.199081+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:15.199081+0000","last_peered":"2024-09-18T11:58:15.199081+0000","last_clean":"2024-09-18T11:58:15.199081+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:15.199081+0000","last_undegraded":"2024-09-18T11:58:15.199081+0000","last_fullsized":"2024-09-18T11:58:15.199081+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189812,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207059,"num_write_kb":828331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27147,"num_bytes_recovered":39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924032,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447808,"statfs":{"total":96636764160,"available":95690555392,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.245}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.25800000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.29199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919360,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452480,"statfs":{"total":96636764160,"available":95695339520,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.7629999999999999,"15min":4.2469999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.153},"max":{"1min":0.27800000000000002,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":4.9039999999999999,"15min":3.9279999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.40200000000000002,"5min":33.259,"15min":33.259},"last":0.252}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.183},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.22900000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.214},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.27100000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930484,"kb_used_data":812,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441356,"statfs":{"total":96636764160,"available":95683948544,"internally_reserved":0,"allocated":831488,"data_stored":541635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.192}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.30099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915648,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456192,"statfs":{"total":96636764160,"available":95699140608,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 11:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.23899999999999999,"5min":4.9889999999999999,"15min":4.5270000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.28599999999999998,"5min":28.969000000000001,"15min":32.795000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":4.6589999999999998,"15min":4.0720000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.34599999999999997,"5min":36.697000000000003,"15min":36.697000000000003},"last":0.252}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:31.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:32.209 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:32.209 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:32.221 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-18T11:58:32.132129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693744,"kb_used_data":2376,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793616,"statfs":{"total":386547056640,"available":382764662784,"internally_reserved":0,"allocated":2433024,"data_stored":1236344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-88127,"num_objects":-13,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-77504,"num_write_kb":-310051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11316,"num_bytes_recovered":-22380544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001578"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17596,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.343257+0000","last_change":"2024-09-18T11:58:27.343257+0000","last_active":"2024-09-18T11:58:27.343257+0000","last_peered":"2024-09-18T11:58:27.343257+0000","last_clean":"2024-09-18T11:58:27.343257+0000","last_became_active":"2024-09-18T11:58:27.278648+0000","last_became_peered":"2024-09-18T11:58:27.278648+0000","last_unstale":"2024-09-18T11:58:27.343257+0000","last_undegraded":"2024-09-18T11:58:27.343257+0000","last_fullsized":"2024-09-18T11:58:27.343257+0000","mapping_epoch":62,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:49:37.907698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17372,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281590+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:26.281590+0000","last_peered":"2024-09-18T11:58:26.281590+0000","last_clean":"2024-09-18T11:58:26.281590+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:26.281590+0000","last_undegraded":"2024-09-18T11:58:26.281590+0000","last_fullsized":"2024-09-18T11:58:26.281590+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17056,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.272856+0000","last_change":"2024-09-18T11:58:25.804269+0000","last_active":"2024-09-18T11:58:26.272856+0000","last_peered":"2024-09-18T11:58:26.272856+0000","last_clean":"2024-09-18T11:58:26.272856+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:26.272856+0000","last_undegraded":"2024-09-18T11:58:26.272856+0000","last_fullsized":"2024-09-18T11:58:26.272856+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-18T11:58:25.803717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:25.803717+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:24:10.013450+0000","scrub_duration":0.013251295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13047,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17161,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274385+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:26.274385+0000","last_peered":"2024-09-18T11:58:26.274385+0000","last_clean":"2024-09-18T11:58:26.274385+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:26.274385+0000","last_undegraded":"2024-09-18T11:58:26.274385+0000","last_fullsized":"2024-09-18T11:58:26.274385+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17182,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274432+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:26.274432+0000","last_peered":"2024-09-18T11:58:26.274432+0000","last_clean":"2024-09-18T11:58:26.274432+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:26.274432+0000","last_undegraded":"2024-09-18T11:58:26.274432+0000","last_fullsized":"2024-09-18T11:58:26.274432+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17406,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.515141+0000","last_change":"2024-09-18T11:58:26.515057+0000","last_active":"2024-09-18T11:58:26.515141+0000","last_peered":"2024-09-18T11:58:26.515141+0000","last_clean":"2024-09-18T11:58:26.515141+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:26.515141+0000","last_undegraded":"2024-09-18T11:58:26.515141+0000","last_fullsized":"2024-09-18T11:58:26.515141+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13014","last_scrub_stamp":"2024-09-18T11:58:26.514944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.514944+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:30:45.277062+0000","scrub_duration":0.016684516999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281528+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:26.281528+0000","last_peered":"2024-09-18T11:58:26.281528+0000","last_clean":"2024-09-18T11:58:26.281528+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:26.281528+0000","last_undegraded":"2024-09-18T11:58:26.281528+0000","last_fullsized":"2024-09-18T11:58:26.281528+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17561,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:28.493330+0000","last_change":"2024-09-18T11:58:28.493290+0000","last_active":"2024-09-18T11:58:28.493330+0000","last_peered":"2024-09-18T11:58:28.493330+0000","last_clean":"2024-09-18T11:58:28.493330+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:28.493330+0000","last_undegraded":"2024-09-18T11:58:28.493330+0000","last_fullsized":"2024-09-18T11:58:28.493330+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13172","last_scrub_stamp":"2024-09-18T11:58:28.493215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:28.493215+0000","objects_scrubbed":1,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:29:03.448332+0000","scrub_duration":0.014771559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16685,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.848737+0000","last_change":"2024-09-18T11:58:26.848668+0000","last_active":"2024-09-18T11:58:26.848737+0000","last_peered":"2024-09-18T11:58:26.848737+0000","last_clean":"2024-09-18T11:58:26.848737+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:26.848737+0000","last_undegraded":"2024-09-18T11:58:26.848737+0000","last_fullsized":"2024-09-18T11:58:26.848737+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12592","last_scrub_stamp":"2024-09-18T11:58:26.848552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.848552+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:15.284675+0000","scrub_duration":0.014433247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17274,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:29.487934+0000","last_change":"2024-09-18T11:58:29.487888+0000","last_active":"2024-09-18T11:58:29.487934+0000","last_peered":"2024-09-18T11:58:29.487934+0000","last_clean":"2024-09-18T11:58:29.487934+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:29.487934+0000","last_undegraded":"2024-09-18T11:58:29.487934+0000","last_fullsized":"2024-09-18T11:58:29.487934+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13030","last_scrub_stamp":"2024-09-18T11:58:29.487803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:29.487803+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:55:31.606347+0000","scrub_duration":0.01322956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281639+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:26.281639+0000","last_peered":"2024-09-18T11:58:26.281639+0000","last_clean":"2024-09-18T11:58:26.281639+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:26.281639+0000","last_undegraded":"2024-09-18T11:58:26.281639+0000","last_fullsized":"2024-09-18T11:58:26.281639+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13095,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281411+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:26.281411+0000","last_peered":"2024-09-18T11:58:26.281411+0000","last_clean":"2024-09-18T11:58:26.281411+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:26.281411+0000","last_undegraded":"2024-09-18T11:58:26.281411+0000","last_fullsized":"2024-09-18T11:58:26.281411+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13236,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17226,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.555177+0000","last_change":"2024-09-18T11:58:27.555131+0000","last_active":"2024-09-18T11:58:27.555177+0000","last_peered":"2024-09-18T11:58:27.555177+0000","last_clean":"2024-09-18T11:58:27.555177+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:27.555177+0000","last_undegraded":"2024-09-18T11:58:27.555177+0000","last_fullsized":"2024-09-18T11:58:27.555177+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12901","last_scrub_stamp":"2024-09-18T11:58:27.555031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:27.555031+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:00:35.229032+0000","scrub_duration":0.014757505000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12832,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281499+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:26.281499+0000","last_peered":"2024-09-18T11:58:26.281499+0000","last_clean":"2024-09-18T11:58:26.281499+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:26.281499+0000","last_undegraded":"2024-09-18T11:58:26.281499+0000","last_fullsized":"2024-09-18T11:58:26.281499+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924044,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447796,"statfs":{"total":96636764160,"available":95690543104,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.39100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.37}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452468,"statfs":{"total":96636764160,"available":95695327232,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":3.9630000000000001,"15min":4.1600000000000001},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.153},"max":{"1min":0.41499999999999998,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.28999999999999998,"5min":4.101,"15min":3.8530000000000002},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.54500000000000004,"5min":33.259,"15min":33.259},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.44700000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934668,"kb_used_data":828,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437172,"statfs":{"total":96636764160,"available":95679664128,"internally_reserved":0,"allocated":847872,"data_stored":548702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.35899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.377},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.39000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915660,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456180,"statfs":{"total":96636764160,"available":95699128320,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":3.3650000000000002,"15min":4.4880000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.34999999999999998,"5min":18.751000000000001,"15min":32.795000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.27200000000000002,"5min":3.4620000000000002,"15min":4.0110000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.36699999999999999,"5min":20.099,"15min":36.697000000000003},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.318},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:32.222 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-18T11:58:32.222 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-18T11:58:32.222 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:32.481 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:32.482 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:32.496 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-18T11:58:32.132129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693744,"kb_used_data":2376,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793616,"statfs":{"total":386547056640,"available":382764662784,"internally_reserved":0,"allocated":2433024,"data_stored":1236344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-88127,"num_objects":-13,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-77504,"num_write_kb":-310051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11316,"num_bytes_recovered":-22380544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001578"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17596,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.343257+0000","last_change":"2024-09-18T11:58:27.343257+0000","last_active":"2024-09-18T11:58:27.343257+0000","last_peered":"2024-09-18T11:58:27.343257+0000","last_clean":"2024-09-18T11:58:27.343257+0000","last_became_active":"2024-09-18T11:58:27.278648+0000","last_became_peered":"2024-09-18T11:58:27.278648+0000","last_unstale":"2024-09-18T11:58:27.343257+0000","last_undegraded":"2024-09-18T11:58:27.343257+0000","last_fullsized":"2024-09-18T11:58:27.343257+0000","mapping_epoch":62,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:49:37.907698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17372,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281590+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:26.281590+0000","last_peered":"2024-09-18T11:58:26.281590+0000","last_clean":"2024-09-18T11:58:26.281590+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:26.281590+0000","last_undegraded":"2024-09-18T11:58:26.281590+0000","last_fullsized":"2024-09-18T11:58:26.281590+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17056,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.272856+0000","last_change":"2024-09-18T11:58:25.804269+0000","last_active":"2024-09-18T11:58:26.272856+0000","last_peered":"2024-09-18T11:58:26.272856+0000","last_clean":"2024-09-18T11:58:26.272856+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:26.272856+0000","last_undegraded":"2024-09-18T11:58:26.272856+0000","last_fullsized":"2024-09-18T11:58:26.272856+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-18T11:58:25.803717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:25.803717+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:24:10.013450+0000","scrub_duration":0.013251295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13047,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17161,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274385+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:26.274385+0000","last_peered":"2024-09-18T11:58:26.274385+0000","last_clean":"2024-09-18T11:58:26.274385+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:26.274385+0000","last_undegraded":"2024-09-18T11:58:26.274385+0000","last_fullsized":"2024-09-18T11:58:26.274385+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17182,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274432+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:26.274432+0000","last_peered":"2024-09-18T11:58:26.274432+0000","last_clean":"2024-09-18T11:58:26.274432+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:26.274432+0000","last_undegraded":"2024-09-18T11:58:26.274432+0000","last_fullsized":"2024-09-18T11:58:26.274432+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17406,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.515141+0000","last_change":"2024-09-18T11:58:26.515057+0000","last_active":"2024-09-18T11:58:26.515141+0000","last_peered":"2024-09-18T11:58:26.515141+0000","last_clean":"2024-09-18T11:58:26.515141+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:26.515141+0000","last_undegraded":"2024-09-18T11:58:26.515141+0000","last_fullsized":"2024-09-18T11:58:26.515141+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13014","last_scrub_stamp":"2024-09-18T11:58:26.514944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.514944+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:30:45.277062+0000","scrub_duration":0.016684516999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281528+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:26.281528+0000","last_peered":"2024-09-18T11:58:26.281528+0000","last_clean":"2024-09-18T11:58:26.281528+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:26.281528+0000","last_undegraded":"2024-09-18T11:58:26.281528+0000","last_fullsized":"2024-09-18T11:58:26.281528+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17561,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:28.493330+0000","last_change":"2024-09-18T11:58:28.493290+0000","last_active":"2024-09-18T11:58:28.493330+0000","last_peered":"2024-09-18T11:58:28.493330+0000","last_clean":"2024-09-18T11:58:28.493330+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:28.493330+0000","last_undegraded":"2024-09-18T11:58:28.493330+0000","last_fullsized":"2024-09-18T11:58:28.493330+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13172","last_scrub_stamp":"2024-09-18T11:58:28.493215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:28.493215+0000","objects_scrubbed":1,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:29:03.448332+0000","scrub_duration":0.014771559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16685,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.848737+0000","last_change":"2024-09-18T11:58:26.848668+0000","last_active":"2024-09-18T11:58:26.848737+0000","last_peered":"2024-09-18T11:58:26.848737+0000","last_clean":"2024-09-18T11:58:26.848737+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:26.848737+0000","last_undegraded":"2024-09-18T11:58:26.848737+0000","last_fullsized":"2024-09-18T11:58:26.848737+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12592","last_scrub_stamp":"2024-09-18T11:58:26.848552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.848552+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:15.284675+0000","scrub_duration":0.014433247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17274,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:29.487934+0000","last_change":"2024-09-18T11:58:29.487888+0000","last_active":"2024-09-18T11:58:29.487934+0000","last_peered":"2024-09-18T11:58:29.487934+0000","last_clean":"2024-09-18T11:58:29.487934+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:29.487934+0000","last_undegraded":"2024-09-18T11:58:29.487934+0000","last_fullsized":"2024-09-18T11:58:29.487934+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13030","last_scrub_stamp":"2024-09-18T11:58:29.487803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:29.487803+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:55:31.606347+0000","scrub_duration":0.01322956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281639+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:26.281639+0000","last_peered":"2024-09-18T11:58:26.281639+0000","last_clean":"2024-09-18T11:58:26.281639+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:26.281639+0000","last_undegraded":"2024-09-18T11:58:26.281639+0000","last_fullsized":"2024-09-18T11:58:26.281639+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13095,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281411+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:26.281411+0000","last_peered":"2024-09-18T11:58:26.281411+0000","last_clean":"2024-09-18T11:58:26.281411+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:26.281411+0000","last_undegraded":"2024-09-18T11:58:26.281411+0000","last_fullsized":"2024-09-18T11:58:26.281411+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13236,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17226,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.555177+0000","last_change":"2024-09-18T11:58:27.555131+0000","last_active":"2024-09-18T11:58:27.555177+0000","last_peered":"2024-09-18T11:58:27.555177+0000","last_clean":"2024-09-18T11:58:27.555177+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:27.555177+0000","last_undegraded":"2024-09-18T11:58:27.555177+0000","last_fullsized":"2024-09-18T11:58:27.555177+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12901","last_scrub_stamp":"2024-09-18T11:58:27.555031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:27.555031+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:00:35.229032+0000","scrub_duration":0.014757505000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12832,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281499+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:26.281499+0000","last_peered":"2024-09-18T11:58:26.281499+0000","last_clean":"2024-09-18T11:58:26.281499+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:26.281499+0000","last_undegraded":"2024-09-18T11:58:26.281499+0000","last_fullsized":"2024-09-18T11:58:26.281499+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924044,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447796,"statfs":{"total":96636764160,"available":95690543104,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.39100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.37}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452468,"statfs":{"total":96636764160,"available":95695327232,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":3.9630000000000001,"15min":4.1600000000000001},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.153},"max":{"1min":0.41499999999999998,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.28999999999999998,"5min":4.101,"15min":3.8530000000000002},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.54500000000000004,"5min":33.259,"15min":33.259},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.44700000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934668,"kb_used_data":828,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437172,"statfs":{"total":96636764160,"available":95679664128,"internally_reserved":0,"allocated":847872,"data_stored":548702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.35899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.377},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.39000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915660,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456180,"statfs":{"total":96636764160,"available":95699128320,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":3.3650000000000002,"15min":4.4880000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.34999999999999998,"5min":18.751000000000001,"15min":32.795000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.27200000000000002,"5min":3.4620000000000002,"15min":4.0110000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.36699999999999999,"5min":20.099,"15min":36.697000000000003},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.318},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:32.498 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:32.752 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:32.753 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:32.764 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-18T11:58:32.132129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693744,"kb_used_data":2376,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793616,"statfs":{"total":386547056640,"available":382764662784,"internally_reserved":0,"allocated":2433024,"data_stored":1236344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-88127,"num_objects":-13,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-77504,"num_write_kb":-310051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11316,"num_bytes_recovered":-22380544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001578"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17596,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.343257+0000","last_change":"2024-09-18T11:58:27.343257+0000","last_active":"2024-09-18T11:58:27.343257+0000","last_peered":"2024-09-18T11:58:27.343257+0000","last_clean":"2024-09-18T11:58:27.343257+0000","last_became_active":"2024-09-18T11:58:27.278648+0000","last_became_peered":"2024-09-18T11:58:27.278648+0000","last_unstale":"2024-09-18T11:58:27.343257+0000","last_undegraded":"2024-09-18T11:58:27.343257+0000","last_fullsized":"2024-09-18T11:58:27.343257+0000","mapping_epoch":62,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:49:37.907698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17372,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281590+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:26.281590+0000","last_peered":"2024-09-18T11:58:26.281590+0000","last_clean":"2024-09-18T11:58:26.281590+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:26.281590+0000","last_undegraded":"2024-09-18T11:58:26.281590+0000","last_fullsized":"2024-09-18T11:58:26.281590+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17056,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.272856+0000","last_change":"2024-09-18T11:58:25.804269+0000","last_active":"2024-09-18T11:58:26.272856+0000","last_peered":"2024-09-18T11:58:26.272856+0000","last_clean":"2024-09-18T11:58:26.272856+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:26.272856+0000","last_undegraded":"2024-09-18T11:58:26.272856+0000","last_fullsized":"2024-09-18T11:58:26.272856+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-18T11:58:25.803717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:25.803717+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:24:10.013450+0000","scrub_duration":0.013251295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13047,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17161,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274385+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:26.274385+0000","last_peered":"2024-09-18T11:58:26.274385+0000","last_clean":"2024-09-18T11:58:26.274385+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:26.274385+0000","last_undegraded":"2024-09-18T11:58:26.274385+0000","last_fullsized":"2024-09-18T11:58:26.274385+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17182,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274432+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:26.274432+0000","last_peered":"2024-09-18T11:58:26.274432+0000","last_clean":"2024-09-18T11:58:26.274432+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:26.274432+0000","last_undegraded":"2024-09-18T11:58:26.274432+0000","last_fullsized":"2024-09-18T11:58:26.274432+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17406,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.515141+0000","last_change":"2024-09-18T11:58:26.515057+0000","last_active":"2024-09-18T11:58:26.515141+0000","last_peered":"2024-09-18T11:58:26.515141+0000","last_clean":"2024-09-18T11:58:26.515141+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:26.515141+0000","last_undegraded":"2024-09-18T11:58:26.515141+0000","last_fullsized":"2024-09-18T11:58:26.515141+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13014","last_scrub_stamp":"2024-09-18T11:58:26.514944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.514944+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:30:45.277062+0000","scrub_duration":0.016684516999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281528+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:26.281528+0000","last_peered":"2024-09-18T11:58:26.281528+0000","last_clean":"2024-09-18T11:58:26.281528+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:26.281528+0000","last_undegraded":"2024-09-18T11:58:26.281528+0000","last_fullsized":"2024-09-18T11:58:26.281528+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17561,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:28.493330+0000","last_change":"2024-09-18T11:58:28.493290+0000","last_active":"2024-09-18T11:58:28.493330+0000","last_peered":"2024-09-18T11:58:28.493330+0000","last_clean":"2024-09-18T11:58:28.493330+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:28.493330+0000","last_undegraded":"2024-09-18T11:58:28.493330+0000","last_fullsized":"2024-09-18T11:58:28.493330+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13172","last_scrub_stamp":"2024-09-18T11:58:28.493215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:28.493215+0000","objects_scrubbed":1,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:29:03.448332+0000","scrub_duration":0.014771559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16685,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.848737+0000","last_change":"2024-09-18T11:58:26.848668+0000","last_active":"2024-09-18T11:58:26.848737+0000","last_peered":"2024-09-18T11:58:26.848737+0000","last_clean":"2024-09-18T11:58:26.848737+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:26.848737+0000","last_undegraded":"2024-09-18T11:58:26.848737+0000","last_fullsized":"2024-09-18T11:58:26.848737+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12592","last_scrub_stamp":"2024-09-18T11:58:26.848552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.848552+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:15.284675+0000","scrub_duration":0.014433247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17274,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:29.487934+0000","last_change":"2024-09-18T11:58:29.487888+0000","last_active":"2024-09-18T11:58:29.487934+0000","last_peered":"2024-09-18T11:58:29.487934+0000","last_clean":"2024-09-18T11:58:29.487934+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:29.487934+0000","last_undegraded":"2024-09-18T11:58:29.487934+0000","last_fullsized":"2024-09-18T11:58:29.487934+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13030","last_scrub_stamp":"2024-09-18T11:58:29.487803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:29.487803+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:55:31.606347+0000","scrub_duration":0.01322956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281639+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:26.281639+0000","last_peered":"2024-09-18T11:58:26.281639+0000","last_clean":"2024-09-18T11:58:26.281639+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:26.281639+0000","last_undegraded":"2024-09-18T11:58:26.281639+0000","last_fullsized":"2024-09-18T11:58:26.281639+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13095,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281411+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:26.281411+0000","last_peered":"2024-09-18T11:58:26.281411+0000","last_clean":"2024-09-18T11:58:26.281411+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:26.281411+0000","last_undegraded":"2024-09-18T11:58:26.281411+0000","last_fullsized":"2024-09-18T11:58:26.281411+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13236,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17226,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.555177+0000","last_change":"2024-09-18T11:58:27.555131+0000","last_active":"2024-09-18T11:58:27.555177+0000","last_peered":"2024-09-18T11:58:27.555177+0000","last_clean":"2024-09-18T11:58:27.555177+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:27.555177+0000","last_undegraded":"2024-09-18T11:58:27.555177+0000","last_fullsized":"2024-09-18T11:58:27.555177+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12901","last_scrub_stamp":"2024-09-18T11:58:27.555031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:27.555031+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:00:35.229032+0000","scrub_duration":0.014757505000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12832,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281499+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:26.281499+0000","last_peered":"2024-09-18T11:58:26.281499+0000","last_clean":"2024-09-18T11:58:26.281499+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:26.281499+0000","last_undegraded":"2024-09-18T11:58:26.281499+0000","last_fullsized":"2024-09-18T11:58:26.281499+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924044,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447796,"statfs":{"total":96636764160,"available":95690543104,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.39100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.37}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452468,"statfs":{"total":96636764160,"available":95695327232,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":3.9630000000000001,"15min":4.1600000000000001},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.153},"max":{"1min":0.41499999999999998,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.28999999999999998,"5min":4.101,"15min":3.8530000000000002},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.54500000000000004,"5min":33.259,"15min":33.259},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.44700000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934668,"kb_used_data":828,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437172,"statfs":{"total":96636764160,"available":95679664128,"internally_reserved":0,"allocated":847872,"data_stored":548702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.35899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.377},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.39000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915660,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456180,"statfs":{"total":96636764160,"available":95699128320,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":3.3650000000000002,"15min":4.4880000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.34999999999999998,"5min":18.751000000000001,"15min":32.795000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.27200000000000002,"5min":3.4620000000000002,"15min":4.0110000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.36699999999999999,"5min":20.099,"15min":36.697000000000003},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.318},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:32.765 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-18T11:58:32.765 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T11:58:32.775 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-18T11:58:32.775 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:58:33.033 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:33.033 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":63,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:58:27.269504+0000","last_up_change":"2024-09-18T11:44:45.102723+0000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-18T11:43:35.889679+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":49,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":40343},{"type":"v1","addr":"172.21.15.71:6801","nonce":40343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":40343},{"type":"v1","addr":"172.21.15.71:6803","nonce":40343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":40343},{"type":"v1","addr":"172.21.15.71:6807","nonce":40343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":40343},{"type":"v1","addr":"172.21.15.71:6805","nonce":40343}]},"public_addr":"172.21.15.71:6801/40343","cluster_addr":"172.21.15.71:6803/40343","heartbeat_back_addr":"172.21.15.71:6807/40343","heartbeat_front_addr":"172.21.15.71:6805/40343","state":["exists","up"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":60,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":46018},{"type":"v1","addr":"172.21.15.71:6822","nonce":46018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":46018},{"type":"v1","addr":"172.21.15.71:6825","nonce":46018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6829","nonce":46018},{"type":"v1","addr":"172.21.15.71:6831","nonce":46018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":46018},{"type":"v1","addr":"172.21.15.71:6828","nonce":46018}]},"public_addr":"172.21.15.71:6822/46018","cluster_addr":"172.21.15.71:6825/46018","heartbeat_back_addr":"172.21.15.71:6831/46018","heartbeat_front_addr":"172.21.15.71:6828/46018","state":["exists","up"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":33275},{"type":"v1","addr":"172.21.15.71:6811","nonce":33275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":33275},{"type":"v1","addr":"172.21.15.71:6818","nonce":33275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6827","nonce":33275},{"type":"v1","addr":"172.21.15.71:6830","nonce":33275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":33275},{"type":"v1","addr":"172.21.15.71:6823","nonce":33275}]},"public_addr":"172.21.15.71:6811/33275","cluster_addr":"172.21.15.71:6818/33275","heartbeat_back_addr":"172.21.15.71:6830/33275","heartbeat_front_addr":"172.21.15.71:6823/33275","state":["exists","up"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":62,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":41313},{"type":"v1","addr":"172.21.15.71:6810","nonce":41313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":41313},{"type":"v1","addr":"172.21.15.71:6813","nonce":41313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6817","nonce":41313},{"type":"v1","addr":"172.21.15.71:6819","nonce":41313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":41313},{"type":"v1","addr":"172.21.15.71:6816","nonce":41313}]},"public_addr":"172.21.15.71:6810/41313","cluster_addr":"172.21.15.71:6813/41313","heartbeat_back_addr":"172.21.15.71:6819/41313","heartbeat_front_addr":"172.21.15.71:6816/41313","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.447050+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.374310+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.355034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.394307+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:58:33.046 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T11:58:33.046 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:33.306 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:33.307 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:33.318 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-18T11:58:32.132129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693744,"kb_used_data":2376,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793616,"statfs":{"total":386547056640,"available":382764662784,"internally_reserved":0,"allocated":2433024,"data_stored":1236344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-88127,"num_objects":-13,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-77504,"num_write_kb":-310051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11316,"num_bytes_recovered":-22380544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001578"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17596,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.343257+0000","last_change":"2024-09-18T11:58:27.343257+0000","last_active":"2024-09-18T11:58:27.343257+0000","last_peered":"2024-09-18T11:58:27.343257+0000","last_clean":"2024-09-18T11:58:27.343257+0000","last_became_active":"2024-09-18T11:58:27.278648+0000","last_became_peered":"2024-09-18T11:58:27.278648+0000","last_unstale":"2024-09-18T11:58:27.343257+0000","last_undegraded":"2024-09-18T11:58:27.343257+0000","last_fullsized":"2024-09-18T11:58:27.343257+0000","mapping_epoch":62,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:49:37.907698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17372,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281590+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:26.281590+0000","last_peered":"2024-09-18T11:58:26.281590+0000","last_clean":"2024-09-18T11:58:26.281590+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:26.281590+0000","last_undegraded":"2024-09-18T11:58:26.281590+0000","last_fullsized":"2024-09-18T11:58:26.281590+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17056,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.272856+0000","last_change":"2024-09-18T11:58:25.804269+0000","last_active":"2024-09-18T11:58:26.272856+0000","last_peered":"2024-09-18T11:58:26.272856+0000","last_clean":"2024-09-18T11:58:26.272856+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:26.272856+0000","last_undegraded":"2024-09-18T11:58:26.272856+0000","last_fullsized":"2024-09-18T11:58:26.272856+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-18T11:58:25.803717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:25.803717+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:24:10.013450+0000","scrub_duration":0.013251295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13047,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17161,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274385+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:26.274385+0000","last_peered":"2024-09-18T11:58:26.274385+0000","last_clean":"2024-09-18T11:58:26.274385+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:26.274385+0000","last_undegraded":"2024-09-18T11:58:26.274385+0000","last_fullsized":"2024-09-18T11:58:26.274385+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17182,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274432+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:26.274432+0000","last_peered":"2024-09-18T11:58:26.274432+0000","last_clean":"2024-09-18T11:58:26.274432+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:26.274432+0000","last_undegraded":"2024-09-18T11:58:26.274432+0000","last_fullsized":"2024-09-18T11:58:26.274432+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17406,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.515141+0000","last_change":"2024-09-18T11:58:26.515057+0000","last_active":"2024-09-18T11:58:26.515141+0000","last_peered":"2024-09-18T11:58:26.515141+0000","last_clean":"2024-09-18T11:58:26.515141+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:26.515141+0000","last_undegraded":"2024-09-18T11:58:26.515141+0000","last_fullsized":"2024-09-18T11:58:26.515141+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13014","last_scrub_stamp":"2024-09-18T11:58:26.514944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.514944+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:30:45.277062+0000","scrub_duration":0.016684516999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281528+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:26.281528+0000","last_peered":"2024-09-18T11:58:26.281528+0000","last_clean":"2024-09-18T11:58:26.281528+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:26.281528+0000","last_undegraded":"2024-09-18T11:58:26.281528+0000","last_fullsized":"2024-09-18T11:58:26.281528+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17561,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:28.493330+0000","last_change":"2024-09-18T11:58:28.493290+0000","last_active":"2024-09-18T11:58:28.493330+0000","last_peered":"2024-09-18T11:58:28.493330+0000","last_clean":"2024-09-18T11:58:28.493330+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:28.493330+0000","last_undegraded":"2024-09-18T11:58:28.493330+0000","last_fullsized":"2024-09-18T11:58:28.493330+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13172","last_scrub_stamp":"2024-09-18T11:58:28.493215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:28.493215+0000","objects_scrubbed":1,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:29:03.448332+0000","scrub_duration":0.014771559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16685,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.848737+0000","last_change":"2024-09-18T11:58:26.848668+0000","last_active":"2024-09-18T11:58:26.848737+0000","last_peered":"2024-09-18T11:58:26.848737+0000","last_clean":"2024-09-18T11:58:26.848737+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:26.848737+0000","last_undegraded":"2024-09-18T11:58:26.848737+0000","last_fullsized":"2024-09-18T11:58:26.848737+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12592","last_scrub_stamp":"2024-09-18T11:58:26.848552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.848552+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:15.284675+0000","scrub_duration":0.014433247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17274,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:29.487934+0000","last_change":"2024-09-18T11:58:29.487888+0000","last_active":"2024-09-18T11:58:29.487934+0000","last_peered":"2024-09-18T11:58:29.487934+0000","last_clean":"2024-09-18T11:58:29.487934+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:29.487934+0000","last_undegraded":"2024-09-18T11:58:29.487934+0000","last_fullsized":"2024-09-18T11:58:29.487934+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13030","last_scrub_stamp":"2024-09-18T11:58:29.487803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:29.487803+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:55:31.606347+0000","scrub_duration":0.01322956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281639+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:26.281639+0000","last_peered":"2024-09-18T11:58:26.281639+0000","last_clean":"2024-09-18T11:58:26.281639+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:26.281639+0000","last_undegraded":"2024-09-18T11:58:26.281639+0000","last_fullsized":"2024-09-18T11:58:26.281639+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13095,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281411+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:26.281411+0000","last_peered":"2024-09-18T11:58:26.281411+0000","last_clean":"2024-09-18T11:58:26.281411+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:26.281411+0000","last_undegraded":"2024-09-18T11:58:26.281411+0000","last_fullsized":"2024-09-18T11:58:26.281411+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13236,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17226,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.555177+0000","last_change":"2024-09-18T11:58:27.555131+0000","last_active":"2024-09-18T11:58:27.555177+0000","last_peered":"2024-09-18T11:58:27.555177+0000","last_clean":"2024-09-18T11:58:27.555177+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:27.555177+0000","last_undegraded":"2024-09-18T11:58:27.555177+0000","last_fullsized":"2024-09-18T11:58:27.555177+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12901","last_scrub_stamp":"2024-09-18T11:58:27.555031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:27.555031+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:00:35.229032+0000","scrub_duration":0.014757505000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12832,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281499+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:26.281499+0000","last_peered":"2024-09-18T11:58:26.281499+0000","last_clean":"2024-09-18T11:58:26.281499+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:26.281499+0000","last_undegraded":"2024-09-18T11:58:26.281499+0000","last_fullsized":"2024-09-18T11:58:26.281499+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924044,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447796,"statfs":{"total":96636764160,"available":95690543104,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.39100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.37}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452468,"statfs":{"total":96636764160,"available":95695327232,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":3.9630000000000001,"15min":4.1600000000000001},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.153},"max":{"1min":0.41499999999999998,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.28999999999999998,"5min":4.101,"15min":3.8530000000000002},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.54500000000000004,"5min":33.259,"15min":33.259},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.44700000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934668,"kb_used_data":828,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437172,"statfs":{"total":96636764160,"available":95679664128,"internally_reserved":0,"allocated":847872,"data_stored":548702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.35899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.377},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.39000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915660,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456180,"statfs":{"total":96636764160,"available":95699128320,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":3.3650000000000002,"15min":4.4880000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.34999999999999998,"5min":18.751000000000001,"15min":32.795000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.27200000000000002,"5min":3.4620000000000002,"15min":4.0110000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.36699999999999999,"5min":20.099,"15min":36.697000000000003},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.318},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:33.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:33.576 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:33.577 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:33.588 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-18T11:58:32.132129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693744,"kb_used_data":2376,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793616,"statfs":{"total":386547056640,"available":382764662784,"internally_reserved":0,"allocated":2433024,"data_stored":1236344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-88127,"num_objects":-13,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-77504,"num_write_kb":-310051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11316,"num_bytes_recovered":-22380544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001578"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17596,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.343257+0000","last_change":"2024-09-18T11:58:27.343257+0000","last_active":"2024-09-18T11:58:27.343257+0000","last_peered":"2024-09-18T11:58:27.343257+0000","last_clean":"2024-09-18T11:58:27.343257+0000","last_became_active":"2024-09-18T11:58:27.278648+0000","last_became_peered":"2024-09-18T11:58:27.278648+0000","last_unstale":"2024-09-18T11:58:27.343257+0000","last_undegraded":"2024-09-18T11:58:27.343257+0000","last_fullsized":"2024-09-18T11:58:27.343257+0000","mapping_epoch":62,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:49:37.907698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17372,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281590+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:26.281590+0000","last_peered":"2024-09-18T11:58:26.281590+0000","last_clean":"2024-09-18T11:58:26.281590+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:26.281590+0000","last_undegraded":"2024-09-18T11:58:26.281590+0000","last_fullsized":"2024-09-18T11:58:26.281590+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17056,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.272856+0000","last_change":"2024-09-18T11:58:25.804269+0000","last_active":"2024-09-18T11:58:26.272856+0000","last_peered":"2024-09-18T11:58:26.272856+0000","last_clean":"2024-09-18T11:58:26.272856+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:26.272856+0000","last_undegraded":"2024-09-18T11:58:26.272856+0000","last_fullsized":"2024-09-18T11:58:26.272856+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-18T11:58:25.803717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:25.803717+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:24:10.013450+0000","scrub_duration":0.013251295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13047,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17161,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274385+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:26.274385+0000","last_peered":"2024-09-18T11:58:26.274385+0000","last_clean":"2024-09-18T11:58:26.274385+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:26.274385+0000","last_undegraded":"2024-09-18T11:58:26.274385+0000","last_fullsized":"2024-09-18T11:58:26.274385+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17182,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274432+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:26.274432+0000","last_peered":"2024-09-18T11:58:26.274432+0000","last_clean":"2024-09-18T11:58:26.274432+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:26.274432+0000","last_undegraded":"2024-09-18T11:58:26.274432+0000","last_fullsized":"2024-09-18T11:58:26.274432+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17406,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.515141+0000","last_change":"2024-09-18T11:58:26.515057+0000","last_active":"2024-09-18T11:58:26.515141+0000","last_peered":"2024-09-18T11:58:26.515141+0000","last_clean":"2024-09-18T11:58:26.515141+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:26.515141+0000","last_undegraded":"2024-09-18T11:58:26.515141+0000","last_fullsized":"2024-09-18T11:58:26.515141+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13014","last_scrub_stamp":"2024-09-18T11:58:26.514944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.514944+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:30:45.277062+0000","scrub_duration":0.016684516999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281528+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:26.281528+0000","last_peered":"2024-09-18T11:58:26.281528+0000","last_clean":"2024-09-18T11:58:26.281528+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:26.281528+0000","last_undegraded":"2024-09-18T11:58:26.281528+0000","last_fullsized":"2024-09-18T11:58:26.281528+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17561,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:28.493330+0000","last_change":"2024-09-18T11:58:28.493290+0000","last_active":"2024-09-18T11:58:28.493330+0000","last_peered":"2024-09-18T11:58:28.493330+0000","last_clean":"2024-09-18T11:58:28.493330+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:28.493330+0000","last_undegraded":"2024-09-18T11:58:28.493330+0000","last_fullsized":"2024-09-18T11:58:28.493330+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13172","last_scrub_stamp":"2024-09-18T11:58:28.493215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:28.493215+0000","objects_scrubbed":1,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:29:03.448332+0000","scrub_duration":0.014771559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16685,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.848737+0000","last_change":"2024-09-18T11:58:26.848668+0000","last_active":"2024-09-18T11:58:26.848737+0000","last_peered":"2024-09-18T11:58:26.848737+0000","last_clean":"2024-09-18T11:58:26.848737+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:26.848737+0000","last_undegraded":"2024-09-18T11:58:26.848737+0000","last_fullsized":"2024-09-18T11:58:26.848737+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12592","last_scrub_stamp":"2024-09-18T11:58:26.848552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.848552+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:15.284675+0000","scrub_duration":0.014433247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17274,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:29.487934+0000","last_change":"2024-09-18T11:58:29.487888+0000","last_active":"2024-09-18T11:58:29.487934+0000","last_peered":"2024-09-18T11:58:29.487934+0000","last_clean":"2024-09-18T11:58:29.487934+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:29.487934+0000","last_undegraded":"2024-09-18T11:58:29.487934+0000","last_fullsized":"2024-09-18T11:58:29.487934+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13030","last_scrub_stamp":"2024-09-18T11:58:29.487803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:29.487803+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:55:31.606347+0000","scrub_duration":0.01322956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281639+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:26.281639+0000","last_peered":"2024-09-18T11:58:26.281639+0000","last_clean":"2024-09-18T11:58:26.281639+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:26.281639+0000","last_undegraded":"2024-09-18T11:58:26.281639+0000","last_fullsized":"2024-09-18T11:58:26.281639+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13095,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281411+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:26.281411+0000","last_peered":"2024-09-18T11:58:26.281411+0000","last_clean":"2024-09-18T11:58:26.281411+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:26.281411+0000","last_undegraded":"2024-09-18T11:58:26.281411+0000","last_fullsized":"2024-09-18T11:58:26.281411+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13236,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17226,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.555177+0000","last_change":"2024-09-18T11:58:27.555131+0000","last_active":"2024-09-18T11:58:27.555177+0000","last_peered":"2024-09-18T11:58:27.555177+0000","last_clean":"2024-09-18T11:58:27.555177+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:27.555177+0000","last_undegraded":"2024-09-18T11:58:27.555177+0000","last_fullsized":"2024-09-18T11:58:27.555177+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12901","last_scrub_stamp":"2024-09-18T11:58:27.555031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:27.555031+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:00:35.229032+0000","scrub_duration":0.014757505000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12832,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281499+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:26.281499+0000","last_peered":"2024-09-18T11:58:26.281499+0000","last_clean":"2024-09-18T11:58:26.281499+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:26.281499+0000","last_undegraded":"2024-09-18T11:58:26.281499+0000","last_fullsized":"2024-09-18T11:58:26.281499+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924044,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447796,"statfs":{"total":96636764160,"available":95690543104,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.39100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.37}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452468,"statfs":{"total":96636764160,"available":95695327232,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":3.9630000000000001,"15min":4.1600000000000001},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.153},"max":{"1min":0.41499999999999998,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.28999999999999998,"5min":4.101,"15min":3.8530000000000002},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.54500000000000004,"5min":33.259,"15min":33.259},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.44700000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934668,"kb_used_data":828,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437172,"statfs":{"total":96636764160,"available":95679664128,"internally_reserved":0,"allocated":847872,"data_stored":548702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.35899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.377},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.39000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915660,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456180,"statfs":{"total":96636764160,"available":95699128320,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":3.3650000000000002,"15min":4.4880000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.34999999999999998,"5min":18.751000000000001,"15min":32.795000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.27200000000000002,"5min":3.4620000000000002,"15min":4.0110000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.36699999999999999,"5min":20.099,"15min":36.697000000000003},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.318},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:33.589 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T11:58:33.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:33.846 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:33.846 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:33.857 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-18T11:58:32.132129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693744,"kb_used_data":2376,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793616,"statfs":{"total":386547056640,"available":382764662784,"internally_reserved":0,"allocated":2433024,"data_stored":1236344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-88127,"num_objects":-13,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-77504,"num_write_kb":-310051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11316,"num_bytes_recovered":-22380544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001578"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17596,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.343257+0000","last_change":"2024-09-18T11:58:27.343257+0000","last_active":"2024-09-18T11:58:27.343257+0000","last_peered":"2024-09-18T11:58:27.343257+0000","last_clean":"2024-09-18T11:58:27.343257+0000","last_became_active":"2024-09-18T11:58:27.278648+0000","last_became_peered":"2024-09-18T11:58:27.278648+0000","last_unstale":"2024-09-18T11:58:27.343257+0000","last_undegraded":"2024-09-18T11:58:27.343257+0000","last_fullsized":"2024-09-18T11:58:27.343257+0000","mapping_epoch":62,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:49:37.907698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17372,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281590+0000","last_change":"2024-09-18T11:49:42.131954+0000","last_active":"2024-09-18T11:58:26.281590+0000","last_peered":"2024-09-18T11:58:26.281590+0000","last_clean":"2024-09-18T11:58:26.281590+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:26.281590+0000","last_undegraded":"2024-09-18T11:58:26.281590+0000","last_fullsized":"2024-09-18T11:58:26.281590+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2106,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17056,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.272856+0000","last_change":"2024-09-18T11:58:25.804269+0000","last_active":"2024-09-18T11:58:26.272856+0000","last_peered":"2024-09-18T11:58:26.272856+0000","last_clean":"2024-09-18T11:58:26.272856+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:26.272856+0000","last_undegraded":"2024-09-18T11:58:26.272856+0000","last_fullsized":"2024-09-18T11:58:26.272856+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-18T11:58:25.803717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:25.803717+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:24:10.013450+0000","scrub_duration":0.013251295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13047,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17161,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274385+0000","last_change":"2024-09-18T11:58:06.421109+0000","last_active":"2024-09-18T11:58:26.274385+0000","last_peered":"2024-09-18T11:58:26.274385+0000","last_clean":"2024-09-18T11:58:26.274385+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:26.274385+0000","last_undegraded":"2024-09-18T11:58:26.274385+0000","last_fullsized":"2024-09-18T11:58:26.274385+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":51573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17182,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274432+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:26.274432+0000","last_peered":"2024-09-18T11:58:26.274432+0000","last_clean":"2024-09-18T11:58:26.274432+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:26.274432+0000","last_undegraded":"2024-09-18T11:58:26.274432+0000","last_fullsized":"2024-09-18T11:58:26.274432+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17406,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.515141+0000","last_change":"2024-09-18T11:58:26.515057+0000","last_active":"2024-09-18T11:58:26.515141+0000","last_peered":"2024-09-18T11:58:26.515141+0000","last_clean":"2024-09-18T11:58:26.515141+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:26.515141+0000","last_undegraded":"2024-09-18T11:58:26.515141+0000","last_fullsized":"2024-09-18T11:58:26.515141+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13014","last_scrub_stamp":"2024-09-18T11:58:26.514944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.514944+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:30:45.277062+0000","scrub_duration":0.016684516999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281528+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:26.281528+0000","last_peered":"2024-09-18T11:58:26.281528+0000","last_clean":"2024-09-18T11:58:26.281528+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:26.281528+0000","last_undegraded":"2024-09-18T11:58:26.281528+0000","last_fullsized":"2024-09-18T11:58:26.281528+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17561,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:28.493330+0000","last_change":"2024-09-18T11:58:28.493290+0000","last_active":"2024-09-18T11:58:28.493330+0000","last_peered":"2024-09-18T11:58:28.493330+0000","last_clean":"2024-09-18T11:58:28.493330+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:28.493330+0000","last_undegraded":"2024-09-18T11:58:28.493330+0000","last_fullsized":"2024-09-18T11:58:28.493330+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13172","last_scrub_stamp":"2024-09-18T11:58:28.493215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:28.493215+0000","objects_scrubbed":1,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:29:03.448332+0000","scrub_duration":0.014771559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16685,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.848737+0000","last_change":"2024-09-18T11:58:26.848668+0000","last_active":"2024-09-18T11:58:26.848737+0000","last_peered":"2024-09-18T11:58:26.848737+0000","last_clean":"2024-09-18T11:58:26.848737+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:26.848737+0000","last_undegraded":"2024-09-18T11:58:26.848737+0000","last_fullsized":"2024-09-18T11:58:26.848737+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12592","last_scrub_stamp":"2024-09-18T11:58:26.848552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.848552+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:15.284675+0000","scrub_duration":0.014433247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17274,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:29.487934+0000","last_change":"2024-09-18T11:58:29.487888+0000","last_active":"2024-09-18T11:58:29.487934+0000","last_peered":"2024-09-18T11:58:29.487934+0000","last_clean":"2024-09-18T11:58:29.487934+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:29.487934+0000","last_undegraded":"2024-09-18T11:58:29.487934+0000","last_fullsized":"2024-09-18T11:58:29.487934+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13030","last_scrub_stamp":"2024-09-18T11:58:29.487803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:29.487803+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:55:31.606347+0000","scrub_duration":0.01322956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281639+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:26.281639+0000","last_peered":"2024-09-18T11:58:26.281639+0000","last_clean":"2024-09-18T11:58:26.281639+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:26.281639+0000","last_undegraded":"2024-09-18T11:58:26.281639+0000","last_fullsized":"2024-09-18T11:58:26.281639+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13095,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281411+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:26.281411+0000","last_peered":"2024-09-18T11:58:26.281411+0000","last_clean":"2024-09-18T11:58:26.281411+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:26.281411+0000","last_undegraded":"2024-09-18T11:58:26.281411+0000","last_fullsized":"2024-09-18T11:58:26.281411+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13236,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17226,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.555177+0000","last_change":"2024-09-18T11:58:27.555131+0000","last_active":"2024-09-18T11:58:27.555177+0000","last_peered":"2024-09-18T11:58:27.555177+0000","last_clean":"2024-09-18T11:58:27.555177+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:27.555177+0000","last_undegraded":"2024-09-18T11:58:27.555177+0000","last_fullsized":"2024-09-18T11:58:27.555177+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12901","last_scrub_stamp":"2024-09-18T11:58:27.555031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:27.555031+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:00:35.229032+0000","scrub_duration":0.014757505000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12832,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281499+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:26.281499+0000","last_peered":"2024-09-18T11:58:26.281499+0000","last_clean":"2024-09-18T11:58:26.281499+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:26.281499+0000","last_undegraded":"2024-09-18T11:58:26.281499+0000","last_fullsized":"2024-09-18T11:58:26.281499+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101685,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129555,"num_write_kb":518280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14721,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924044,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447796,"statfs":{"total":96636764160,"available":95690543104,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.39100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.318}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.37}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452468,"statfs":{"total":96636764160,"available":95695327232,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":3.9630000000000001,"15min":4.1600000000000001},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.153},"max":{"1min":0.41499999999999998,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.28999999999999998,"5min":4.101,"15min":3.8530000000000002},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.54500000000000004,"5min":33.259,"15min":33.259},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":5.1820000000000004,"15min":4.173},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":0.30499999999999999,"5min":22.73,"15min":22.73},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.26000000000000001,"5min":4.9340000000000002,"15min":3.6930000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.32600000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.44700000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934668,"kb_used_data":828,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437172,"statfs":{"total":96636764160,"available":95679664128,"internally_reserved":0,"allocated":847872,"data_stored":548702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.35899999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.377},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.39000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915660,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456180,"statfs":{"total":96636764160,"available":95699128320,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":3.3650000000000002,"15min":4.4880000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.34999999999999998,"5min":18.751000000000001,"15min":32.795000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.27200000000000002,"5min":3.4620000000000002,"15min":4.0110000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.36699999999999999,"5min":20.099,"15min":36.697000000000003},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed Sep 18 11:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":5.5529999999999999,"15min":4.4619999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.32400000000000001,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.318},{"interface":"front","average":{"1min":0.28299999999999997,"5min":6.806,"15min":5.415},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.40300000000000002,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:33.859 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:58:34.116 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:34.116 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":63,"fsid":"c7ad3eaa-5b13-4721-8f42-ca8e4f8544ab","created":"2024-09-18T11:43:21.913078+0000","modified":"2024-09-18T11:58:27.269504+0000","last_up_change":"2024-09-18T11:44:45.102723+0000","last_in_change":"2024-09-18T11:43:23.196826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-18T11:43:35.889679+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"182a21f6-a1df-455b-ac1c-b1a90ea63d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":49,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":40343},{"type":"v1","addr":"172.21.15.71:6801","nonce":40343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":40343},{"type":"v1","addr":"172.21.15.71:6803","nonce":40343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":40343},{"type":"v1","addr":"172.21.15.71:6807","nonce":40343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":40343},{"type":"v1","addr":"172.21.15.71:6805","nonce":40343}]},"public_addr":"172.21.15.71:6801/40343","cluster_addr":"172.21.15.71:6803/40343","heartbeat_back_addr":"172.21.15.71:6807/40343","heartbeat_front_addr":"172.21.15.71:6805/40343","state":["exists","up"]},{"osd":1,"uuid":"9a5cd690-48d3-4692-890f-ac01f7d42f93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":60,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":46018},{"type":"v1","addr":"172.21.15.71:6822","nonce":46018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":46018},{"type":"v1","addr":"172.21.15.71:6825","nonce":46018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6829","nonce":46018},{"type":"v1","addr":"172.21.15.71:6831","nonce":46018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":46018},{"type":"v1","addr":"172.21.15.71:6828","nonce":46018}]},"public_addr":"172.21.15.71:6822/46018","cluster_addr":"172.21.15.71:6825/46018","heartbeat_back_addr":"172.21.15.71:6831/46018","heartbeat_front_addr":"172.21.15.71:6828/46018","state":["exists","up"]},{"osd":2,"uuid":"b66ec834-18a9-428d-9af7-657a268cb3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":33275},{"type":"v1","addr":"172.21.15.71:6811","nonce":33275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":33275},{"type":"v1","addr":"172.21.15.71:6818","nonce":33275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6827","nonce":33275},{"type":"v1","addr":"172.21.15.71:6830","nonce":33275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":33275},{"type":"v1","addr":"172.21.15.71:6823","nonce":33275}]},"public_addr":"172.21.15.71:6811/33275","cluster_addr":"172.21.15.71:6818/33275","heartbeat_back_addr":"172.21.15.71:6830/33275","heartbeat_front_addr":"172.21.15.71:6823/33275","state":["exists","up"]},{"osd":3,"uuid":"abb2daba-a64d-4db4-82fe-9dfcb8cbd7c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":62,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":41313},{"type":"v1","addr":"172.21.15.71:6810","nonce":41313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":41313},{"type":"v1","addr":"172.21.15.71:6813","nonce":41313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6817","nonce":41313},{"type":"v1","addr":"172.21.15.71:6819","nonce":41313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":41313},{"type":"v1","addr":"172.21.15.71:6816","nonce":41313}]},"public_addr":"172.21.15.71:6810/41313","cluster_addr":"172.21.15.71:6813/41313","heartbeat_back_addr":"172.21.15.71:6819/41313","heartbeat_front_addr":"172.21.15.71:6816/41313","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.447050+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.374310+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.355034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:43:28.394307+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:58:35.129 INFO:tasks.ceph:Scrubbing osd.0 2024-09-18T11:58:35.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T11:58:35.255 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:58:35.255 INFO:teuthology.orchestra.run.smithi071.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T11:58:35.255 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:58:35.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-18T11:58:35.528 INFO:teuthology.orchestra.run.smithi071.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T11:58:35.541 INFO:tasks.ceph:Scrubbing osd.1 2024-09-18T11:58:35.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T11:58:35.656 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:58:35.656 INFO:teuthology.orchestra.run.smithi071.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T11:58:35.656 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:58:35.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-18T11:58:35.924 INFO:teuthology.orchestra.run.smithi071.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T11:58:35.937 INFO:tasks.ceph:Scrubbing osd.2 2024-09-18T11:58:35.937 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T11:58:36.051 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:58:36.051 INFO:teuthology.orchestra.run.smithi071.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T11:58:36.051 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:58:36.061 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-18T11:58:36.322 INFO:teuthology.orchestra.run.smithi071.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T11:58:36.334 INFO:tasks.ceph:Scrubbing osd.3 2024-09-18T11:58:36.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-18T11:58:36.449 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-18T11:58:36.449 INFO:teuthology.orchestra.run.smithi071.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T11:58:36.449 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-18T11:58:36.459 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-18T11:58:36.717 INFO:teuthology.orchestra.run.smithi071.stderr:instructed osd(s) 3 to deep-scrub 2024-09-18T11:58:36.729 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:36.985 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:36.986 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:36.997 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-18T11:58:36.132742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74569,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103612,"num_write_kb":414487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10529,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693424,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793936,"statfs":{"total":386547056640,"available":382764990464,"internally_reserved":0,"allocated":2113536,"data_stored":916856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-115243,"num_objects":-17,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-17,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-103447,"num_write_kb":-413844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-16618,"num_bytes_recovered":-39600128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.862894"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17596,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.343257+0000","last_change":"2024-09-18T11:58:27.343257+0000","last_active":"2024-09-18T11:58:27.343257+0000","last_peered":"2024-09-18T11:58:27.343257+0000","last_clean":"2024-09-18T11:58:27.343257+0000","last_became_active":"2024-09-18T11:58:27.278648+0000","last_became_peered":"2024-09-18T11:58:27.278648+0000","last_unstale":"2024-09-18T11:58:27.343257+0000","last_undegraded":"2024-09-18T11:58:27.343257+0000","last_fullsized":"2024-09-18T11:58:27.343257+0000","mapping_epoch":62,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:49:37.907698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17382,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.493538+0000","last_change":"2024-09-18T11:58:32.493475+0000","last_active":"2024-09-18T11:58:32.493538+0000","last_peered":"2024-09-18T11:58:32.493538+0000","last_clean":"2024-09-18T11:58:32.493538+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:32.493538+0000","last_undegraded":"2024-09-18T11:58:32.493538+0000","last_fullsized":"2024-09-18T11:58:32.493538+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13054","last_scrub_stamp":"2024-09-18T11:58:32.493312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:32.493312+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:31:29.092825+0000","scrub_duration":0.015390114999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17056,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.272856+0000","last_change":"2024-09-18T11:58:25.804269+0000","last_active":"2024-09-18T11:58:26.272856+0000","last_peered":"2024-09-18T11:58:26.272856+0000","last_clean":"2024-09-18T11:58:26.272856+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:26.272856+0000","last_undegraded":"2024-09-18T11:58:26.272856+0000","last_fullsized":"2024-09-18T11:58:26.272856+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-18T11:58:25.803717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:25.803717+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:24:10.013450+0000","scrub_duration":0.013251295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13047,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17171,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:31.547734+0000","last_change":"2024-09-18T11:58:31.547692+0000","last_active":"2024-09-18T11:58:31.547734+0000","last_peered":"2024-09-18T11:58:31.547734+0000","last_clean":"2024-09-18T11:58:31.547734+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:31.547734+0000","last_undegraded":"2024-09-18T11:58:31.547734+0000","last_fullsized":"2024-09-18T11:58:31.547734+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12893","last_scrub_stamp":"2024-09-18T11:58:31.547587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:31.547587+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:34:48.227673+0000","scrub_duration":0.016232783000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17182,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.274432+0000","last_change":"2024-09-18T11:57:34.281895+0000","last_active":"2024-09-18T11:58:26.274432+0000","last_peered":"2024-09-18T11:58:26.274432+0000","last_clean":"2024-09-18T11:58:26.274432+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:26.274432+0000","last_undegraded":"2024-09-18T11:58:26.274432+0000","last_fullsized":"2024-09-18T11:58:26.274432+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:20.829232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17407,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.273214+0000","last_change":"2024-09-18T11:58:26.515057+0000","last_active":"2024-09-18T11:58:27.273214+0000","last_peered":"2024-09-18T11:58:27.273214+0000","last_clean":"2024-09-18T11:58:27.273214+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:27.273214+0000","last_undegraded":"2024-09-18T11:58:27.273214+0000","last_fullsized":"2024-09-18T11:58:27.273214+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13014","last_scrub_stamp":"2024-09-18T11:58:26.514944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.514944+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.016684516999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281528+0000","last_change":"2024-09-18T11:55:53.200074+0000","last_active":"2024-09-18T11:58:26.281528+0000","last_peered":"2024-09-18T11:58:26.281528+0000","last_clean":"2024-09-18T11:58:26.281528+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:26.281528+0000","last_undegraded":"2024-09-18T11:58:26.281528+0000","last_fullsized":"2024-09-18T11:58:26.281528+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":51492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17561,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:28.493330+0000","last_change":"2024-09-18T11:58:28.493290+0000","last_active":"2024-09-18T11:58:28.493330+0000","last_peered":"2024-09-18T11:58:28.493330+0000","last_clean":"2024-09-18T11:58:28.493330+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:28.493330+0000","last_undegraded":"2024-09-18T11:58:28.493330+0000","last_fullsized":"2024-09-18T11:58:28.493330+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13172","last_scrub_stamp":"2024-09-18T11:58:28.493215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:28.493215+0000","objects_scrubbed":1,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:29:03.448332+0000","scrub_duration":0.014771559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16685,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.848737+0000","last_change":"2024-09-18T11:58:26.848668+0000","last_active":"2024-09-18T11:58:26.848737+0000","last_peered":"2024-09-18T11:58:26.848737+0000","last_clean":"2024-09-18T11:58:26.848737+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:26.848737+0000","last_undegraded":"2024-09-18T11:58:26.848737+0000","last_fullsized":"2024-09-18T11:58:26.848737+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12592","last_scrub_stamp":"2024-09-18T11:58:26.848552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:26.848552+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:15.284675+0000","scrub_duration":0.014433247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17274,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:29.487934+0000","last_change":"2024-09-18T11:58:29.487888+0000","last_active":"2024-09-18T11:58:29.487934+0000","last_peered":"2024-09-18T11:58:29.487934+0000","last_clean":"2024-09-18T11:58:29.487934+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:29.487934+0000","last_undegraded":"2024-09-18T11:58:29.487934+0000","last_fullsized":"2024-09-18T11:58:29.487934+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13030","last_scrub_stamp":"2024-09-18T11:58:29.487803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:29.487803+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:55:31.606347+0000","scrub_duration":0.01322956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281639+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:26.281639+0000","last_peered":"2024-09-18T11:58:26.281639+0000","last_clean":"2024-09-18T11:58:26.281639+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:26.281639+0000","last_undegraded":"2024-09-18T11:58:26.281639+0000","last_fullsized":"2024-09-18T11:58:26.281639+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13095,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281411+0000","last_change":"2024-09-18T11:57:11.388493+0000","last_active":"2024-09-18T11:58:26.281411+0000","last_peered":"2024-09-18T11:58:26.281411+0000","last_clean":"2024-09-18T11:58:26.281411+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:26.281411+0000","last_undegraded":"2024-09-18T11:58:26.281411+0000","last_fullsized":"2024-09-18T11:58:26.281411+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12965,"num_write_kb":51863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13236,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281386+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:26.281386+0000","last_peered":"2024-09-18T11:58:26.281386+0000","last_clean":"2024-09-18T11:58:26.281386+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:26.281386+0000","last_undegraded":"2024-09-18T11:58:26.281386+0000","last_fullsized":"2024-09-18T11:58:26.281386+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17226,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.555177+0000","last_change":"2024-09-18T11:58:27.555131+0000","last_active":"2024-09-18T11:58:27.555177+0000","last_peered":"2024-09-18T11:58:27.555177+0000","last_clean":"2024-09-18T11:58:27.555177+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:27.555177+0000","last_undegraded":"2024-09-18T11:58:27.555177+0000","last_fullsized":"2024-09-18T11:58:27.555177+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12901","last_scrub_stamp":"2024-09-18T11:58:27.555031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:58:27.555031+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:00:35.229032+0000","scrub_duration":0.014757505000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12832,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T11:58:26.281499+0000","last_change":"2024-09-18T11:56:40.305343+0000","last_active":"2024-09-18T11:58:26.281499+0000","last_peered":"2024-09-18T11:58:26.281499+0000","last_clean":"2024-09-18T11:58:26.281499+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:26.281499+0000","last_undegraded":"2024-09-18T11:58:26.281499+0000","last_fullsized":"2024-09-18T11:58:26.281499+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74569,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103612,"num_write_kb":414487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10529,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924044,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447796,"statfs":{"total":96636764160,"available":95690543104,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":5.9870000000000001,"15min":5.1260000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.45300000000000001,"5min":23.504000000000001,"15min":23.504000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":5.2229999999999999,"15min":4.8979999999999997},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":21.968,"15min":31.675000000000001},"last":0.26700000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.32500000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.25600000000000001,"5min":5.4649999999999999,"15min":4.9059999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.32400000000000001,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.26400000000000001,"5min":3.0640000000000001,"15min":3.6899999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.33400000000000002,"5min":24.974,"15min":24.974},"last":0.28799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452468,"statfs":{"total":96636764160,"available":95695327232,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":3.9630000000000001,"15min":4.1600000000000001},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.153},"max":{"1min":0.41499999999999998,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.28999999999999998,"5min":4.101,"15min":3.8530000000000002},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.54500000000000004,"5min":33.259,"15min":33.259},"last":0.33100000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.307}]},{"osd":3,"last update":"Wed Sep 18 11:58:32 2024","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":4.0129999999999999,"15min":4.0309999999999997},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.53000000000000003,"5min":22.73,"15min":22.73},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.29999999999999999,"5min":3.379,"15min":3.5990000000000002},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.44700000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.34699999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934348,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437492,"statfs":{"total":96636764160,"available":95679991808,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.27200000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.29199999999999998}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.309}]}]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915660,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456180,"statfs":{"total":96636764160,"available":95699128320,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.439},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.497}]},{"osd":2,"last update":"Wed Sep 18 11:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":3.3650000000000002,"15min":4.4880000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.34999999999999998,"5min":18.751000000000001,"15min":32.795000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.27200000000000002,"5min":3.4620000000000002,"15min":4.0110000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.36699999999999999,"5min":20.099,"15min":36.697000000000003},"last":0.55100000000000005}]},{"osd":3,"last update":"Wed Sep 18 11:58:31 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":4.2039999999999997,"15min":4.327},"min":{"1min":0.20599999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.57399999999999995,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.33000000000000002,"5min":5.0430000000000001,"15min":5.335},"min":{"1min":0.215,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.59699999999999998,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:37.001 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.001 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2024-09-18T11:58:32.493312+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=32, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.001 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2024-09-18T11:58:25.803717+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=25, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.002 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.002 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-18T11:58:31.547587+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.002 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.002 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-18T11:58:26.514944+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=26, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.002 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.002 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-18T11:58:28.493215+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=28, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.002 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-18T11:58:26.848552+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=26, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.002 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-18T11:58:29.487803+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=29, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.002 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.003 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.003 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.003 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2024-09-18T11:58:27.555031+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=27, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.003 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:37.003 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T11:58:57.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:57.276 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:58:57.276 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:58:57.288 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-09-18T11:58:56.136998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40674,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65073,"num_write_kb":260312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6465,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693424,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793936,"statfs":{"total":386547056640,"available":382764990464,"internally_reserved":0,"allocated":2113536,"data_stored":916856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20337,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-25668,"num_write_kb":-102683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002572"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17607,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:47.726800+0000","last_change":"2024-09-18T11:58:47.726743+0000","last_active":"2024-09-18T11:58:47.726800+0000","last_peered":"2024-09-18T11:58:47.726800+0000","last_clean":"2024-09-18T11:58:47.726800+0000","last_became_active":"2024-09-18T11:58:27.278648+0000","last_became_peered":"2024-09-18T11:58:27.278648+0000","last_unstale":"2024-09-18T11:58:47.726800+0000","last_undegraded":"2024-09-18T11:58:47.726800+0000","last_fullsized":"2024-09-18T11:58:47.726800+0000","mapping_epoch":62,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13250","last_scrub_stamp":"2024-09-18T11:58:47.726637+0000","last_deep_scrub":"39'13250","last_deep_scrub_stamp":"2024-09-18T11:58:47.726637+0000","last_clean_scrub_stamp":"2024-09-18T11:58:47.726637+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:23:00.890304+0000","scrub_duration":0.01526856,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17393,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:43.441264+0000","last_change":"2024-09-18T11:58:43.441208+0000","last_active":"2024-09-18T11:58:43.441264+0000","last_peered":"2024-09-18T11:58:43.441264+0000","last_clean":"2024-09-18T11:58:43.441264+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:43.441264+0000","last_undegraded":"2024-09-18T11:58:43.441264+0000","last_fullsized":"2024-09-18T11:58:43.441264+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13054","last_scrub_stamp":"2024-09-18T11:58:43.441107+0000","last_deep_scrub":"39'13054","last_deep_scrub_stamp":"2024-09-18T11:58:43.441107+0000","last_clean_scrub_stamp":"2024-09-18T11:58:43.441107+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:30:35.398598+0000","scrub_duration":0.01525637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17067,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:43.829302+0000","last_change":"2024-09-18T11:58:43.829245+0000","last_active":"2024-09-18T11:58:43.829302+0000","last_peered":"2024-09-18T11:58:43.829302+0000","last_clean":"2024-09-18T11:58:43.829302+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:43.829302+0000","last_undegraded":"2024-09-18T11:58:43.829302+0000","last_fullsized":"2024-09-18T11:58:43.829302+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-18T11:58:43.829142+0000","last_deep_scrub":"39'12808","last_deep_scrub_stamp":"2024-09-18T11:58:43.829142+0000","last_clean_scrub_stamp":"2024-09-18T11:58:43.829142+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:07:21.672169+0000","scrub_duration":0.015043776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13048,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.277354+0000","last_change":"2024-09-18T11:57:22.573687+0000","last_active":"2024-09-18T11:58:27.277354+0000","last_peered":"2024-09-18T11:58:27.277354+0000","last_clean":"2024-09-18T11:58:27.277354+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:27.277354+0000","last_undegraded":"2024-09-18T11:58:27.277354+0000","last_fullsized":"2024-09-18T11:58:27.277354+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":51693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17182,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:43.716552+0000","last_change":"2024-09-18T11:58:43.716505+0000","last_active":"2024-09-18T11:58:43.716552+0000","last_peered":"2024-09-18T11:58:43.716552+0000","last_clean":"2024-09-18T11:58:43.716552+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:43.716552+0000","last_undegraded":"2024-09-18T11:58:43.716552+0000","last_fullsized":"2024-09-18T11:58:43.716552+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12893","last_scrub_stamp":"2024-09-18T11:58:43.716401+0000","last_deep_scrub":"39'12893","last_deep_scrub_stamp":"2024-09-18T11:58:43.716401+0000","last_clean_scrub_stamp":"2024-09-18T11:58:43.716401+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:47:23.453481+0000","scrub_duration":0.014943036999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17193,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:50.613227+0000","last_change":"2024-09-18T11:58:50.613177+0000","last_active":"2024-09-18T11:58:50.613227+0000","last_peered":"2024-09-18T11:58:50.613227+0000","last_clean":"2024-09-18T11:58:50.613227+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:50.613227+0000","last_undegraded":"2024-09-18T11:58:50.613227+0000","last_fullsized":"2024-09-18T11:58:50.613227+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12786","last_scrub_stamp":"2024-09-18T11:58:50.613044+0000","last_deep_scrub":"39'12786","last_deep_scrub_stamp":"2024-09-18T11:58:50.613044+0000","last_clean_scrub_stamp":"2024-09-18T11:58:50.613044+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:40:20.718661+0000","scrub_duration":0.016952386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17417,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:38.435767+0000","last_change":"2024-09-18T11:58:38.435702+0000","last_active":"2024-09-18T11:58:38.435767+0000","last_peered":"2024-09-18T11:58:38.435767+0000","last_clean":"2024-09-18T11:58:38.435767+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:38.435767+0000","last_undegraded":"2024-09-18T11:58:38.435767+0000","last_fullsized":"2024-09-18T11:58:38.435767+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13014","last_scrub_stamp":"2024-09-18T11:58:38.435593+0000","last_deep_scrub":"39'13014","last_deep_scrub_stamp":"2024-09-18T11:58:38.435593+0000","last_clean_scrub_stamp":"2024-09-18T11:58:38.435593+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:22:58.329618+0000","scrub_duration":0.017001823999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17074,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:45.438529+0000","last_change":"2024-09-18T11:58:45.438485+0000","last_active":"2024-09-18T11:58:45.438529+0000","last_peered":"2024-09-18T11:58:45.438529+0000","last_clean":"2024-09-18T11:58:45.438529+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:45.438529+0000","last_undegraded":"2024-09-18T11:58:45.438529+0000","last_fullsized":"2024-09-18T11:58:45.438529+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12873","last_scrub_stamp":"2024-09-18T11:58:45.438379+0000","last_deep_scrub":"39'12873","last_deep_scrub_stamp":"2024-09-18T11:58:45.438379+0000","last_clean_scrub_stamp":"2024-09-18T11:58:45.438379+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:43.116840+0000","scrub_duration":0.013516843000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17572,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:49.634588+0000","last_change":"2024-09-18T11:58:49.634517+0000","last_active":"2024-09-18T11:58:49.634588+0000","last_peered":"2024-09-18T11:58:49.634588+0000","last_clean":"2024-09-18T11:58:49.634588+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:49.634588+0000","last_undegraded":"2024-09-18T11:58:49.634588+0000","last_fullsized":"2024-09-18T11:58:49.634588+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13172","last_scrub_stamp":"2024-09-18T11:58:49.634363+0000","last_deep_scrub":"39'13172","last_deep_scrub_stamp":"2024-09-18T11:58:49.634363+0000","last_clean_scrub_stamp":"2024-09-18T11:58:49.634363+0000","objects_scrubbed":1,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:15:05.080917+0000","scrub_duration":0.015709093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16696,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:41.769937+0000","last_change":"2024-09-18T11:58:41.769886+0000","last_active":"2024-09-18T11:58:41.769937+0000","last_peered":"2024-09-18T11:58:41.769937+0000","last_clean":"2024-09-18T11:58:41.769937+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:41.769937+0000","last_undegraded":"2024-09-18T11:58:41.769937+0000","last_fullsized":"2024-09-18T11:58:41.769937+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12592","last_scrub_stamp":"2024-09-18T11:58:41.769775+0000","last_deep_scrub":"39'12592","last_deep_scrub_stamp":"2024-09-18T11:58:41.769775+0000","last_clean_scrub_stamp":"2024-09-18T11:58:41.769775+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:48:08.734438+0000","scrub_duration":0.015605657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17285,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:39.731412+0000","last_change":"2024-09-18T11:58:39.731365+0000","last_active":"2024-09-18T11:58:39.731412+0000","last_peered":"2024-09-18T11:58:39.731412+0000","last_clean":"2024-09-18T11:58:39.731412+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:39.731412+0000","last_undegraded":"2024-09-18T11:58:39.731412+0000","last_fullsized":"2024-09-18T11:58:39.731412+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13030","last_scrub_stamp":"2024-09-18T11:58:39.730586+0000","last_deep_scrub":"39'13030","last_deep_scrub_stamp":"2024-09-18T11:58:39.730586+0000","last_clean_scrub_stamp":"2024-09-18T11:58:39.730586+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:59:49.538014+0000","scrub_duration":0.016966094000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13143,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.278803+0000","last_change":"2024-09-18T11:58:16.269218+0000","last_active":"2024-09-18T11:58:27.278803+0000","last_peered":"2024-09-18T11:58:27.278803+0000","last_clean":"2024-09-18T11:58:27.278803+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:27.278803+0000","last_undegraded":"2024-09-18T11:58:27.278803+0000","last_fullsized":"2024-09-18T11:58:27.278803+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":52119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.478226+0000","last_change":"2024-09-18T11:58:52.478167+0000","last_active":"2024-09-18T11:58:52.478226+0000","last_peered":"2024-09-18T11:58:52.478226+0000","last_clean":"2024-09-18T11:58:52.478226+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:52.478226+0000","last_undegraded":"2024-09-18T11:58:52.478226+0000","last_fullsized":"2024-09-18T11:58:52.478226+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12966","last_scrub_stamp":"2024-09-18T11:58:52.478035+0000","last_deep_scrub":"39'12966","last_deep_scrub_stamp":"2024-09-18T11:58:52.478035+0000","last_clean_scrub_stamp":"2024-09-18T11:58:52.478035+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:24:48.782488+0000","scrub_duration":0.014405585,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13237,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.277301+0000","last_change":"2024-09-18T11:57:55.368006+0000","last_active":"2024-09-18T11:58:27.277301+0000","last_peered":"2024-09-18T11:58:27.277301+0000","last_clean":"2024-09-18T11:58:27.277301+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:27.277301+0000","last_undegraded":"2024-09-18T11:58:27.277301+0000","last_fullsized":"2024-09-18T11:58:27.277301+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:43:36.867433+0000","last_clean_scrub_stamp":"2024-09-18T11:43:36.867433+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13558,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":52356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17237,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:38.441844+0000","last_change":"2024-09-18T11:58:38.441804+0000","last_active":"2024-09-18T11:58:38.441844+0000","last_peered":"2024-09-18T11:58:38.441844+0000","last_clean":"2024-09-18T11:58:38.441844+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:38.441844+0000","last_undegraded":"2024-09-18T11:58:38.441844+0000","last_fullsized":"2024-09-18T11:58:38.441844+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12901","last_scrub_stamp":"2024-09-18T11:58:38.441718+0000","last_deep_scrub":"39'12901","last_deep_scrub_stamp":"2024-09-18T11:58:38.441718+0000","last_clean_scrub_stamp":"2024-09-18T11:58:38.441718+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:41:35.817491+0000","scrub_duration":0.017122160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12843,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:47.488963+0000","last_change":"2024-09-18T11:58:47.488917+0000","last_active":"2024-09-18T11:58:47.488963+0000","last_peered":"2024-09-18T11:58:47.488963+0000","last_clean":"2024-09-18T11:58:47.488963+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:47.488963+0000","last_undegraded":"2024-09-18T11:58:47.488963+0000","last_fullsized":"2024-09-18T11:58:47.488963+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12705","last_scrub_stamp":"2024-09-18T11:58:47.488816+0000","last_deep_scrub":"39'12705","last_deep_scrub_stamp":"2024-09-18T11:58:47.488816+0000","last_clean_scrub_stamp":"2024-09-18T11:58:47.488816+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:04:43.604943+0000","scrub_duration":0.012449253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40674,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65073,"num_write_kb":260312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6465,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924044,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447796,"statfs":{"total":96636764160,"available":95690543104,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:37 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.2750000000000004,"15min":4.9530000000000003},"min":{"1min":0.17899999999999999,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.35299999999999998,"5min":21.800999999999998,"15min":23.504000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.25700000000000001,"5min":3.7669999999999999,"15min":4.7370000000000001},"min":{"1min":0.214,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.39100000000000001,"5min":21.968,"15min":31.675000000000001},"last":0.439}]},{"osd":1,"last update":"Wed Sep 18 11:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":4.6989999999999998,"15min":5.8330000000000002},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.40600000000000003,"5min":33.281999999999996,"15min":55.762999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.246,"5min":3.5600000000000001,"15min":4.7729999999999997},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.307,"5min":25.013999999999999,"15min":57.372},"last":0.36799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:37 2024","interfaces":[{"interface":"back","average":{"1min":0.26800000000000002,"5min":4.3799999999999999,"15min":4.8579999999999997},"min":{"1min":0.224,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.33600000000000002,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.26700000000000002,"5min":2.327,"15min":3.5350000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.37,"5min":24.974,"15min":24.974},"last":0.42299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452468,"statfs":{"total":96636764160,"available":95695327232,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":3.9630000000000001,"15min":4.1600000000000001},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.153},"max":{"1min":0.41499999999999998,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.28999999999999998,"5min":4.101,"15min":3.8530000000000002},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.54500000000000004,"5min":33.259,"15min":33.259},"last":0.41299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.22900000000000001,"5min":5.5789999999999997,"15min":6.2859999999999996},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.34100000000000003,"5min":46.793999999999997,"15min":46.793999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.255,"5min":3.9470000000000001,"15min":4.2220000000000004},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.307,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.32700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:32 2024","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":4.0129999999999999,"15min":4.0309999999999997},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.53000000000000003,"5min":22.73,"15min":22.73},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":3.379,"15min":3.5990000000000002},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.44700000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.44400000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934348,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437492,"statfs":{"total":96636764160,"available":95679991808,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.36899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.38700000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.311},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.35399999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915660,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456180,"statfs":{"total":96636764160,"available":95699128320,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.36299999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":3.3650000000000002,"15min":4.4880000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.34999999999999998,"5min":18.751000000000001,"15min":32.795000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.27200000000000002,"5min":3.4620000000000002,"15min":4.0110000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.36699999999999999,"5min":20.099,"15min":36.697000000000003},"last":0.40400000000000003}]},{"osd":3,"last update":"Wed Sep 18 11:58:31 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":4.2039999999999997,"15min":4.327},"min":{"1min":0.20599999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.57399999999999995,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.312},{"interface":"front","average":{"1min":0.33000000000000002,"5min":5.0430000000000001,"15min":5.335},"min":{"1min":0.215,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.59699999999999998,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:57.289 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:57.290 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:57.290 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2024-09-18T11:43:36.867433+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=43, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=58, tm_sec=34, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T11:58:57.290 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T11:59:17.292 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:17.562 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-18T11:59:17.562 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-09-18T11:59:17.573 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2024-09-18T11:59:16.141052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26036,"num_write_kb":104144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4284,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693424,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":3691313,"kb_avail":373793936,"statfs":{"total":386547056640,"available":382764990464,"internally_reserved":0,"allocated":2113536,"data_stored":916856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779904539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002325"},"pg_stats":[{"pgid":"1.f","version":"39'13250","reported_seq":17607,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:47.726800+0000","last_change":"2024-09-18T11:58:47.726743+0000","last_active":"2024-09-18T11:58:47.726800+0000","last_peered":"2024-09-18T11:58:47.726800+0000","last_clean":"2024-09-18T11:58:47.726800+0000","last_became_active":"2024-09-18T11:58:27.278648+0000","last_became_peered":"2024-09-18T11:58:27.278648+0000","last_unstale":"2024-09-18T11:58:47.726800+0000","last_undegraded":"2024-09-18T11:58:47.726800+0000","last_fullsized":"2024-09-18T11:58:47.726800+0000","mapping_epoch":62,"log_start":"31'3200","ondisk_log_start":"31'3200","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13250","last_scrub_stamp":"2024-09-18T11:58:47.726637+0000","last_deep_scrub":"39'13250","last_deep_scrub_stamp":"2024-09-18T11:58:47.726637+0000","last_clean_scrub_stamp":"2024-09-18T11:58:47.726637+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:23:00.890304+0000","scrub_duration":0.01526856,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":53000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'13054","reported_seq":17393,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:43.441264+0000","last_change":"2024-09-18T11:58:43.441208+0000","last_active":"2024-09-18T11:58:43.441264+0000","last_peered":"2024-09-18T11:58:43.441264+0000","last_clean":"2024-09-18T11:58:43.441264+0000","last_became_active":"2024-09-18T11:49:42.042793+0000","last_became_peered":"2024-09-18T11:49:42.042793+0000","last_unstale":"2024-09-18T11:58:43.441264+0000","last_undegraded":"2024-09-18T11:58:43.441264+0000","last_fullsized":"2024-09-18T11:58:43.441264+0000","mapping_epoch":34,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13054","last_scrub_stamp":"2024-09-18T11:58:43.441107+0000","last_deep_scrub":"39'13054","last_deep_scrub_stamp":"2024-09-18T11:58:43.441107+0000","last_clean_scrub_stamp":"2024-09-18T11:58:43.441107+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:30:35.398598+0000","scrub_duration":0.01525637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12808","reported_seq":17067,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:43.829302+0000","last_change":"2024-09-18T11:58:43.829245+0000","last_active":"2024-09-18T11:58:43.829302+0000","last_peered":"2024-09-18T11:58:43.829302+0000","last_clean":"2024-09-18T11:58:43.829302+0000","last_became_active":"2024-09-18T11:56:30.192053+0000","last_became_peered":"2024-09-18T11:56:30.192053+0000","last_unstale":"2024-09-18T11:58:43.829302+0000","last_undegraded":"2024-09-18T11:58:43.829302+0000","last_fullsized":"2024-09-18T11:58:43.829302+0000","mapping_epoch":40,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-18T11:58:43.829142+0000","last_deep_scrub":"39'12808","last_deep_scrub_stamp":"2024-09-18T11:58:43.829142+0000","last_clean_scrub_stamp":"2024-09-18T11:58:43.829142+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:07:21.672169+0000","scrub_duration":0.015043776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12924","reported_seq":13058,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:55.424704+0000","last_change":"2024-09-18T11:58:55.424641+0000","last_active":"2024-09-18T11:58:55.424704+0000","last_peered":"2024-09-18T11:58:55.424704+0000","last_clean":"2024-09-18T11:58:55.424704+0000","last_became_active":"2024-09-18T11:57:22.504655+0000","last_became_peered":"2024-09-18T11:57:22.504655+0000","last_unstale":"2024-09-18T11:58:55.424704+0000","last_undegraded":"2024-09-18T11:58:55.424704+0000","last_fullsized":"2024-09-18T11:58:55.424704+0000","mapping_epoch":50,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12924","last_scrub_stamp":"2024-09-18T11:58:55.424474+0000","last_deep_scrub":"39'12924","last_deep_scrub_stamp":"2024-09-18T11:58:55.424474+0000","last_clean_scrub_stamp":"2024-09-18T11:58:55.424474+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:25:44.185756+0000","scrub_duration":0.013532216999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12893","reported_seq":17182,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:43.716552+0000","last_change":"2024-09-18T11:58:43.716505+0000","last_active":"2024-09-18T11:58:43.716552+0000","last_peered":"2024-09-18T11:58:43.716552+0000","last_clean":"2024-09-18T11:58:43.716552+0000","last_became_active":"2024-09-18T11:58:06.356942+0000","last_became_peered":"2024-09-18T11:58:06.356942+0000","last_unstale":"2024-09-18T11:58:43.716552+0000","last_undegraded":"2024-09-18T11:58:43.716552+0000","last_fullsized":"2024-09-18T11:58:43.716552+0000","mapping_epoch":58,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12893","last_scrub_stamp":"2024-09-18T11:58:43.716401+0000","last_deep_scrub":"39'12893","last_deep_scrub_stamp":"2024-09-18T11:58:43.716401+0000","last_clean_scrub_stamp":"2024-09-18T11:58:43.716401+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:47:23.453481+0000","scrub_duration":0.014943036999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12786","reported_seq":17193,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:50.613227+0000","last_change":"2024-09-18T11:58:50.613177+0000","last_active":"2024-09-18T11:58:50.613227+0000","last_peered":"2024-09-18T11:58:50.613227+0000","last_clean":"2024-09-18T11:58:50.613227+0000","last_became_active":"2024-09-18T11:57:34.207866+0000","last_became_peered":"2024-09-18T11:57:34.207866+0000","last_unstale":"2024-09-18T11:58:50.613227+0000","last_undegraded":"2024-09-18T11:58:50.613227+0000","last_fullsized":"2024-09-18T11:58:50.613227+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12786","last_scrub_stamp":"2024-09-18T11:58:50.613044+0000","last_deep_scrub":"39'12786","last_deep_scrub_stamp":"2024-09-18T11:58:50.613044+0000","last_clean_scrub_stamp":"2024-09-18T11:58:50.613044+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:40:20.718661+0000","scrub_duration":0.016952386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'13014","reported_seq":17417,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:38.435767+0000","last_change":"2024-09-18T11:58:38.435702+0000","last_active":"2024-09-18T11:58:38.435767+0000","last_peered":"2024-09-18T11:58:38.435767+0000","last_clean":"2024-09-18T11:58:38.435767+0000","last_became_active":"2024-09-18T11:57:01.294840+0000","last_became_peered":"2024-09-18T11:57:01.294840+0000","last_unstale":"2024-09-18T11:58:38.435767+0000","last_undegraded":"2024-09-18T11:58:38.435767+0000","last_fullsized":"2024-09-18T11:58:38.435767+0000","mapping_epoch":46,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13014","last_scrub_stamp":"2024-09-18T11:58:38.435593+0000","last_deep_scrub":"39'13014","last_deep_scrub_stamp":"2024-09-18T11:58:38.435593+0000","last_clean_scrub_stamp":"2024-09-18T11:58:38.435593+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:22:58.329618+0000","scrub_duration":0.017001823999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12873","reported_seq":17074,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:45.438529+0000","last_change":"2024-09-18T11:58:45.438485+0000","last_active":"2024-09-18T11:58:45.438529+0000","last_peered":"2024-09-18T11:58:45.438529+0000","last_clean":"2024-09-18T11:58:45.438529+0000","last_became_active":"2024-09-18T11:55:53.059193+0000","last_became_peered":"2024-09-18T11:55:53.059193+0000","last_unstale":"2024-09-18T11:58:45.438529+0000","last_undegraded":"2024-09-18T11:58:45.438529+0000","last_fullsized":"2024-09-18T11:58:45.438529+0000","mapping_epoch":38,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12873","last_scrub_stamp":"2024-09-18T11:58:45.438379+0000","last_deep_scrub":"39'12873","last_deep_scrub_stamp":"2024-09-18T11:58:45.438379+0000","last_clean_scrub_stamp":"2024-09-18T11:58:45.438379+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:43.116840+0000","scrub_duration":0.013516843000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'13172","reported_seq":17572,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:49.634588+0000","last_change":"2024-09-18T11:58:49.634517+0000","last_active":"2024-09-18T11:58:49.634588+0000","last_peered":"2024-09-18T11:58:49.634588+0000","last_clean":"2024-09-18T11:58:49.634588+0000","last_became_active":"2024-09-18T11:52:37.328998+0000","last_became_peered":"2024-09-18T11:52:37.328998+0000","last_unstale":"2024-09-18T11:58:49.634588+0000","last_undegraded":"2024-09-18T11:58:49.634588+0000","last_fullsized":"2024-09-18T11:58:49.634588+0000","mapping_epoch":36,"log_start":"31'3100","ondisk_log_start":"31'3100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13172","last_scrub_stamp":"2024-09-18T11:58:49.634363+0000","last_deep_scrub":"39'13172","last_deep_scrub_stamp":"2024-09-18T11:58:49.634363+0000","last_clean_scrub_stamp":"2024-09-18T11:58:49.634363+0000","objects_scrubbed":1,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:15:05.080917+0000","scrub_duration":0.015709093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12592","reported_seq":16696,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:41.769937+0000","last_change":"2024-09-18T11:58:41.769886+0000","last_active":"2024-09-18T11:58:41.769937+0000","last_peered":"2024-09-18T11:58:41.769937+0000","last_clean":"2024-09-18T11:58:41.769937+0000","last_became_active":"2024-09-18T11:57:44.230417+0000","last_became_peered":"2024-09-18T11:57:44.230417+0000","last_unstale":"2024-09-18T11:58:41.769937+0000","last_undegraded":"2024-09-18T11:58:41.769937+0000","last_fullsized":"2024-09-18T11:58:41.769937+0000","mapping_epoch":54,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12592","last_scrub_stamp":"2024-09-18T11:58:41.769775+0000","last_deep_scrub":"39'12592","last_deep_scrub_stamp":"2024-09-18T11:58:41.769775+0000","last_clean_scrub_stamp":"2024-09-18T11:58:41.769775+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:48:08.734438+0000","scrub_duration":0.015605657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13030","reported_seq":17285,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:39.731412+0000","last_change":"2024-09-18T11:58:39.731365+0000","last_active":"2024-09-18T11:58:39.731412+0000","last_peered":"2024-09-18T11:58:39.731412+0000","last_clean":"2024-09-18T11:58:39.731412+0000","last_became_active":"2024-09-18T11:56:50.261849+0000","last_became_peered":"2024-09-18T11:56:50.261849+0000","last_unstale":"2024-09-18T11:58:39.731412+0000","last_undegraded":"2024-09-18T11:58:39.731412+0000","last_fullsized":"2024-09-18T11:58:39.731412+0000","mapping_epoch":44,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13030","last_scrub_stamp":"2024-09-18T11:58:39.730586+0000","last_deep_scrub":"39'13030","last_deep_scrub_stamp":"2024-09-18T11:58:39.730586+0000","last_clean_scrub_stamp":"2024-09-18T11:58:39.730586+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:59:49.538014+0000","scrub_duration":0.016966094000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'13030","reported_seq":13153,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:57.356547+0000","last_change":"2024-09-18T11:58:57.356511+0000","last_active":"2024-09-18T11:58:57.356547+0000","last_peered":"2024-09-18T11:58:57.356547+0000","last_clean":"2024-09-18T11:58:57.356547+0000","last_became_active":"2024-09-18T11:58:16.214308+0000","last_became_peered":"2024-09-18T11:58:16.214308+0000","last_unstale":"2024-09-18T11:58:57.356547+0000","last_undegraded":"2024-09-18T11:58:57.356547+0000","last_fullsized":"2024-09-18T11:58:57.356547+0000","mapping_epoch":60,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13030","last_scrub_stamp":"2024-09-18T11:58:57.356399+0000","last_deep_scrub":"39'13030","last_deep_scrub_stamp":"2024-09-18T11:58:57.356399+0000","last_clean_scrub_stamp":"2024-09-18T11:58:57.356399+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:37:09.743926+0000","scrub_duration":0.011805349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12966","reported_seq":13106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.478226+0000","last_change":"2024-09-18T11:58:52.478167+0000","last_active":"2024-09-18T11:58:52.478226+0000","last_peered":"2024-09-18T11:58:52.478226+0000","last_clean":"2024-09-18T11:58:52.478226+0000","last_became_active":"2024-09-18T11:57:11.330999+0000","last_became_peered":"2024-09-18T11:57:11.330999+0000","last_unstale":"2024-09-18T11:58:52.478226+0000","last_undegraded":"2024-09-18T11:58:52.478226+0000","last_fullsized":"2024-09-18T11:58:52.478226+0000","mapping_epoch":48,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12966","last_scrub_stamp":"2024-09-18T11:58:52.478035+0000","last_deep_scrub":"39'12966","last_deep_scrub_stamp":"2024-09-18T11:58:52.478035+0000","last_clean_scrub_stamp":"2024-09-18T11:58:52.478035+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:24:48.782488+0000","scrub_duration":0.014405585,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13088","reported_seq":13247,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:56.378027+0000","last_change":"2024-09-18T11:58:56.377958+0000","last_active":"2024-09-18T11:58:56.378027+0000","last_peered":"2024-09-18T11:58:56.378027+0000","last_clean":"2024-09-18T11:58:56.378027+0000","last_became_active":"2024-09-18T11:57:55.310808+0000","last_became_peered":"2024-09-18T11:57:55.310808+0000","last_unstale":"2024-09-18T11:58:56.378027+0000","last_undegraded":"2024-09-18T11:58:56.378027+0000","last_fullsized":"2024-09-18T11:58:56.378027+0000","mapping_epoch":56,"log_start":"31'3000","ondisk_log_start":"31'3000","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13088","last_scrub_stamp":"2024-09-18T11:58:56.377802+0000","last_deep_scrub":"39'13088","last_deep_scrub_stamp":"2024-09-18T11:58:56.377802+0000","last_clean_scrub_stamp":"2024-09-18T11:58:56.377802+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:00.007137+0000","scrub_duration":0.01508643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12901","reported_seq":17237,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:38.441844+0000","last_change":"2024-09-18T11:58:38.441804+0000","last_active":"2024-09-18T11:58:38.441844+0000","last_peered":"2024-09-18T11:58:38.441844+0000","last_clean":"2024-09-18T11:58:38.441844+0000","last_became_active":"2024-09-18T11:47:02.086850+0000","last_became_peered":"2024-09-18T11:47:02.086850+0000","last_unstale":"2024-09-18T11:58:38.441844+0000","last_undegraded":"2024-09-18T11:58:38.441844+0000","last_fullsized":"2024-09-18T11:58:38.441844+0000","mapping_epoch":32,"log_start":"31'2900","ondisk_log_start":"31'2900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12901","last_scrub_stamp":"2024-09-18T11:58:38.441718+0000","last_deep_scrub":"39'12901","last_deep_scrub_stamp":"2024-09-18T11:58:38.441718+0000","last_clean_scrub_stamp":"2024-09-18T11:58:38.441718+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:41:35.817491+0000","scrub_duration":0.017122160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12705","reported_seq":12843,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T11:58:47.488963+0000","last_change":"2024-09-18T11:58:47.488917+0000","last_active":"2024-09-18T11:58:47.488963+0000","last_peered":"2024-09-18T11:58:47.488963+0000","last_clean":"2024-09-18T11:58:47.488963+0000","last_became_active":"2024-09-18T11:56:40.235445+0000","last_became_peered":"2024-09-18T11:56:40.235445+0000","last_unstale":"2024-09-18T11:58:47.488963+0000","last_undegraded":"2024-09-18T11:58:47.488963+0000","last_fullsized":"2024-09-18T11:58:47.488963+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12705","last_scrub_stamp":"2024-09-18T11:58:47.488816+0000","last_deep_scrub":"39'12705","last_deep_scrub_stamp":"2024-09-18T11:58:47.488816+0000","last_clean_scrub_stamp":"2024-09-18T11:58:47.488816+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:04:43.604943+0000","scrub_duration":0.012449253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6779,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26036,"num_write_kb":104144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4284,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":924044,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923515,"kb_avail":93447796,"statfs":{"total":96636764160,"available":95690543104,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945679937},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:37 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":4.2750000000000004,"15min":4.9530000000000003},"min":{"1min":0.17899999999999999,"5min":0.16800000000000001,"15min":0.155},"max":{"1min":0.35299999999999998,"5min":21.800999999999998,"15min":23.504000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.25700000000000001,"5min":3.7669999999999999,"15min":4.7370000000000001},"min":{"1min":0.214,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.39100000000000001,"5min":21.968,"15min":31.675000000000001},"last":0.30099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:59:05 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":3.0859999999999999,"15min":5.4900000000000002},"min":{"1min":0.20699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.49199999999999999,"5min":18.143000000000001,"15min":55.762999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.308,"5min":2.2389999999999999,"15min":4.5129999999999999},"min":{"1min":0.19800000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.40899999999999997,"5min":14.680999999999999,"15min":57.372},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:37 2024","interfaces":[{"interface":"back","average":{"1min":0.26800000000000002,"5min":4.3799999999999999,"15min":4.8579999999999997},"min":{"1min":0.224,"5min":0.20399999999999999,"15min":0.19500000000000001},"max":{"1min":0.33600000000000002,"5min":28.202999999999999,"15min":42.752000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.26700000000000002,"5min":2.327,"15min":3.5350000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.37,"5min":24.974,"15min":24.974},"last":0.32000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":918846,"kb_avail":93452468,"statfs":{"total":96636764160,"available":95695327232,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940898648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":3.9630000000000001,"15min":4.1600000000000001},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.153},"max":{"1min":0.41499999999999998,"5min":33.225000000000001,"15min":33.225000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.28999999999999998,"5min":4.101,"15min":3.8530000000000002},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.17699999999999999},"max":{"1min":0.54500000000000004,"5min":33.259,"15min":33.259},"last":0.39100000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":3.7599999999999998,"15min":5.9089999999999998},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":0.47199999999999998,"5min":31.425999999999998,"15min":46.793999999999997},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32800000000000001,"5min":2.52,"15min":4},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.182},"max":{"1min":0.49099999999999999,"5min":21.504999999999999,"15min":26.545000000000002},"last":0.30599999999999999}]},{"osd":3,"last update":"Wed Sep 18 11:58:32 2024","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":4.0129999999999999,"15min":4.0309999999999997},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.53000000000000003,"5min":22.73,"15min":22.73},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":3.379,"15min":3.5990000000000002},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.44700000000000001,"5min":24.135000000000002,"15min":24.135000000000002},"last":0.41499999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934348,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437492,"statfs":{"total":96636764160,"available":95679991808,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22700000000000001,"5min":4.9859999999999998,"15min":4.96},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.313,"5min":36.302999999999997,"15min":45.462000000000003},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.24299999999999999,"5min":4.6310000000000002,"15min":5.3140000000000001},"min":{"1min":0.191,"5min":0.183,"15min":0.18099999999999999},"max":{"1min":0.34499999999999997,"5min":27.09,"15min":54.710999999999999},"last":0.34799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":5.0060000000000002,"15min":5.484},"min":{"1min":0.214,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":0.29999999999999999,"5min":45.701999999999998,"15min":45.701999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":4.1920000000000002,"15min":4.556},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":0.32600000000000001,"5min":25.481000000000002,"15min":28.094999999999999},"last":0.38}]},{"osd":3,"last update":"Wed Sep 18 11:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":4.5149999999999997,"15min":4.7359999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":0.32700000000000001,"5min":43.334000000000003,"15min":43.334000000000003},"last":0.312},{"interface":"front","average":{"1min":0.26000000000000001,"5min":5.3390000000000004,"15min":5.7060000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.32800000000000001,"5min":45.786999999999999,"15min":45.786999999999999},"last":0.39500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915660,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":915131,"kb_avail":93456180,"statfs":{"total":96636764160,"available":95699128320,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937094721},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":5.0090000000000003,"15min":5.2519999999999998},"min":{"1min":0.161,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.60299999999999998,"5min":25.960999999999999,"15min":28.059000000000001},"last":0.434},{"interface":"front","average":{"1min":0.23400000000000001,"5min":4.5389999999999997,"15min":4.7539999999999996},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.35099999999999998,"5min":27.818000000000001,"15min":34.786999999999999},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":3.3650000000000002,"15min":4.4880000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.14299999999999999},"max":{"1min":0.34999999999999998,"5min":18.751000000000001,"15min":32.795000000000002},"last":0.504},{"interface":"front","average":{"1min":0.27200000000000002,"5min":3.4620000000000002,"15min":4.0110000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.36699999999999999,"5min":20.099,"15min":36.697000000000003},"last":0.45000000000000001}]},{"osd":3,"last update":"Wed Sep 18 11:58:31 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":4.2039999999999997,"15min":4.327},"min":{"1min":0.20599999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.57399999999999995,"5min":37.414000000000001,"15min":37.414000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.33000000000000002,"5min":5.0430000000000001,"15min":5.335},"min":{"1min":0.215,"5min":0.214,"15min":0.17299999999999999},"max":{"1min":0.59699999999999998,"5min":48.185000000000002,"15min":48.185000000000002},"last":0.46999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:17.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-18T11:59:17.848 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T11:59:17.848 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T11:59:17.848 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-18T11:59:17.848 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:59:22.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-18T11:59:23.950 INFO:tasks.ceph.osd.0:Stopped 2024-09-18T11:59:23.950 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-18T11:59:23.950 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:59:24.026 INFO:tasks.ceph.osd.1:Stopped 2024-09-18T11:59:24.026 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T11:59:24.026 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:59:27.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-18T11:59:27.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-18T11:59:30.128 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T11:59:30.128 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-18T11:59:30.128 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:59:33.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-18T11:59:33.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-18T11:59:36.229 INFO:tasks.ceph.osd.3:Stopped 2024-09-18T11:59:36.230 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T11:59:36.230 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-18T11:59:36.230 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:59:36.257 INFO:tasks.ceph.mgr.x:Stopped 2024-09-18T11:59:36.258 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T11:59:36.258 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T11:59:36.258 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:59:36.311 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T11:59:36.311 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T11:59:36.311 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:59:36.364 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T11:59:36.364 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T11:59:36.364 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:59:36.417 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T11:59:36.417 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T11:59:36.418 DEBUG:teuthology.orchestra.run.smithi071:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-09-18T11:59:36.599 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T11:59:36.599 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-18T11:59:37.273 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T11:59:37.273 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-18T11:59:37.389 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T11:59:37.389 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-18T11:59:37.477 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T11:59:37.477 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-18T11:59:37.567 INFO:tasks.ceph:Archiving mon data... 2024-09-18T11:59:37.568 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910981/data/mon.a.tgz 2024-09-18T11:59:37.568 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-09-18T11:59:37.588 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.SdTvnN7Sew 2024-09-18T11:59:37.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.SdTvnN7Sew 2024-09-18T11:59:37.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.SdTvnN7Sew 2024-09-18T11:59:37.859 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.SdTvnN7Sew is 287KB 2024-09-18T11:59:37.955 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.SdTvnN7Sew 2024-09-18T11:59:37.970 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910981/data/mon.b.tgz 2024-09-18T11:59:37.970 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-09-18T11:59:38.024 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.wRQZch7bhK 2024-09-18T11:59:38.025 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.wRQZch7bhK 2024-09-18T11:59:38.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-18T11:59:38.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.wRQZch7bhK 2024-09-18T11:59:38.219 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.wRQZch7bhK is 288KB 2024-09-18T11:59:38.314 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.wRQZch7bhK 2024-09-18T11:59:38.329 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910981/data/mon.c.tgz 2024-09-18T11:59:38.329 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-09-18T11:59:38.383 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.Y0RjASV1mP 2024-09-18T11:59:38.384 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Y0RjASV1mP 2024-09-18T11:59:38.498 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.Y0RjASV1mP 2024-09-18T11:59:38.582 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.Y0RjASV1mP is 288KB 2024-09-18T11:59:38.648 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.Y0RjASV1mP 2024-09-18T11:59:38.663 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T11:59:38.663 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T11:59:38.742 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-18T11:59:38.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T11:59:38.813 INFO:teuthology.orchestra.run.smithi071.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T11:59:38.818 INFO:tasks.ceph:Archiving crash dumps... 2024-09-18T11:59:38.819 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910981/remote/smithi071/crash 2024-09-18T11:59:38.820 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T11:59:38.883 INFO:tasks.ceph:Compressing logs... 2024-09-18T11:59:38.883 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T11:59:39.231 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-18T11:59:39.231 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55129.log 2024-09-18T11:59:39.231 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71938.log 2024-09-18T11:59:39.231 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60256.log 2024-09-18T11:59:39.231 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38614.log 2024-09-18T11:59:39.231 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71938.log.gz 2024-09-18T11:59:39.231 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51042.log 2024-09-18T11:59:39.231 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60256.log.gz 2024-09-18T11:59:39.231 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71266.log 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37487.log 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39850.log 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71266.log.gz 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66350.log 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64460.log 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68829.log 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66350.log.gz 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42385.log 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64460.log.gz 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38466.log 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68829.log.gz 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52429.log 2024-09-18T11:59:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57776.log 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56810.log 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70131.log 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68703.log 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38799.log 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70131.log.gz 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65594.log 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68703.log.gz 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61687.log 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60382.log 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65594.log.gz 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73576.log 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61687.log.gz 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68030.log 2024-09-18T11:59:39.233 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60382.log.gz 2024-09-18T11:59:39.234 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70887.log 2024-09-18T11:59:39.236 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73576.log.gz 2024-09-18T11:59:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2024-09-18T11:59:39.236 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68030.log.gz 2024-09-18T11:59:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60889.log 2024-09-18T11:59:39.236 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70887.log.gz 2024-09-18T11:59:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45387.log 2024-09-18T11:59:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55885.log 2024-09-18T11:59:39.236 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60889.log.gz 2024-09-18T11:59:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66896.log 2024-09-18T11:59:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61561.log 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67400.log 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66896.log.gz 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54877.log 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61561.log.gz 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69165.log 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67400.log.gz 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43781.log 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54877.log: /var/log/ceph/ceph-client.admin.69165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43739.log 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69165.log.gz 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51084.log 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72484.log 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43739.log: /var/log/ceph/ceph-client.admin.51084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61057.log 2024-09-18T11:59:39.237 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37487.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38799.log.gz -- replaced with /var/log/ceph/ceph-client.admin.38466.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42890.log 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72484.log: -- replaced with /var/log/ceph/ceph-client.admin.38614.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39850.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42385.log.gz -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72484.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43781.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43739.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51084.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52429.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45387.log.gz 2024-09-18T11:59:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61057.log.gz 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51042.log.gz 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55129.log.gz 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54877.log.gz 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72022.log 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55885.log.gz 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56810.log.gz 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57776.log.gz 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43361.log 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42890.log.gz 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68282.log 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72022.log.gz 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70509.log 2024-09-18T11:59:39.239 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43361.log.gz 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64838.log 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68282.log.gz 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44703.log 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70509.log.gz 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53197.log 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64838.log.gz 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61393.log 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44703.log.gz 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43403.log 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53197.log.gz 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58490.log 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61393.log.gz 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2024-09-18T11:59:39.240 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43403.log.gz 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59540.log 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58490.log.gz 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71812.log 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55465.log 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59540.log.gz 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.71812.log: /var/log/ceph/ceph-client.admin.67862.log 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71812.log.gz 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39058.log 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55465.log.gz 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57692.log 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67862.log.gz 2024-09-18T11:59:39.241 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48899.log 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39058.log.gz 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52345.log 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57692.log.gz 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63073.log 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48899.log.gz 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56305.log 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52345.log.gz 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38910.log 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63073.log.gz 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60973.log 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56305.log.gz 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57860.log 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38910.log.gz 2024-09-18T11:59:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63158.log 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60973.log.gz 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62443.log 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57860.log.gz 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33271.log 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63158.log.gz 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62443.log.gz 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47513.log 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.33271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51210.log 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33271.log.gz 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67148.log 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47513.log.gz 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.51210.log: /var/log/ceph/ceph-client.admin.73954.log 2024-09-18T11:59:39.243 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51210.log.gz 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57020.log 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67148.log.gz 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60214.log 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73954.log.gz 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45165.log 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57020.log.gz 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48563.log 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60214.log.gz 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50496.log 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38577.log 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48563.log.gz 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70173.log 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50496.log.gz 2024-09-18T11:59:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43613.log 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38577.log.gz 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70173.log.gz 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58952.log 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69627.log 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43613.log.gz 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48983.log 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58952.log.gz 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50706.log 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69627.log.gz 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64208.log 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48983.log.gz 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59920.log 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50706.log.gz 2024-09-18T11:59:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38429.log 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64208.log.gz 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62485.log 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59920.log.gz 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51378.log 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38429.log.gz 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65426.log 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62485.log.gz 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51000.log 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51378.log.gz 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66098.log 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65426.log.gz 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38318.log 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51000.log.gz 2024-09-18T11:59:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64880.log 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66098.log.gz 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38318.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38318.log.gz 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43823.log 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64880.log.gz 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66812.log 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43823.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.43823.log.gz 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-client.admin.39406.log 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44201.log 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66812.log.gz 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44285.log 0.0% 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43949.log 2024-09-18T11:59:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43445.log 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44285.log.gz 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43949.log.gz 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63914.log 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73828.log 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43445.log.gz 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.45165.log.gz 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51252.log 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63914.log.gz 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61981.log 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73828.log.gz 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47933.log 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51252.log.gz 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50244.log 2024-09-18T11:59:39.248 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61981.log.gz 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32896.log 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47933.log.gz 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50664.log 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50244.log.gz 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.32896.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.32896.log.gz 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.66728.log 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59836.log 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50664.log.gz 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49614.log 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66728.log.gz 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59836.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.59836.log.gz -- 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.48521.log 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53743.log 2024-09-18T11:59:39.249 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49614.log.gz 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58028.log 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48521.log.gz 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59456.log 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53743.log.gz 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58028.log.gzgzip 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68114.log 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51462.log 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59456.log.gz 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45313.log 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68114.log.gz 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48437.log 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51462.log.gz 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45313.log.gz 2024-09-18T11:59:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52849.log 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45572.log 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48437.log.gz 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47471.log 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52849.log.gz 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40146.log 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45572.log.gz 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72568.log 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47471.log.gz 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45683.log 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40146.log.gz 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53617.log 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72568.log.gz 2024-09-18T11:59:39.251 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43251.log 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45683.log.gz 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45091.log 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53617.log.gz 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61435.log 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68450.log 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45091.log.gz 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61435.log.gz 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73660.log 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68408.log 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68450.log.gz 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53113.log 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73660.log.gz 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71055.log 2024-09-18T11:59:39.252 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68408.log.gz 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51336.log 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53113.log.gz 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr: 28.3% -- replaced with /var/log/ceph/ceph-client.admin.44201.log.gz 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42260.log 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71055.log.gz 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57650.log 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51336.log.gz 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66560.log 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42260.log.gz 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33113.log 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57650.log.gz 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56894.log 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66560.log.gz 2024-09-18T11:59:39.253 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.33113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57314.log 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33113.log.gz 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55843.log 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56894.log.gz 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73072.log 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57314.log.gz 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47723.log 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55843.log.gz 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66308.log 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73072.log.gz 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64334.log 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47723.log.gz 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66308.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.66308.log.gz /var/log/ceph/ceph-client.admin.42386.log 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T11:59:39.254 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70677.log 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64334.log.gz 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45054.log 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42386.log.gz 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70677.log.gz 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44243.log 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39887.log 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45054.log.gz 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62191.log 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44243.log.gz 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70929.log 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39887.log.gz 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62149.log 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62191.log.gz 2024-09-18T11:59:39.255 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44159.log 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70929.log.gz 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58910.log 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62149.log.gz 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47303.log 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44159.log.gz 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49992.log 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58910.log.gz 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65930.log 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47303.log.gz 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68661.log 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49992.log.gz 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61099.log 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65930.log.gz 2024-09-18T11:59:39.256 INFO:teuthology.orchestra.run.smithi071.stderr:gzip/var/log/ceph/ceph-client.admin.68661.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67610.log 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68661.log.gz 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52093.log 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61099.log.gz 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57440.log 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67610.log.gz 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73450.log 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52093.log.gz 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58280.log 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57440.log.gz 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52009.log 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73450.log.gz 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38947.log 2024-09-18T11:59:39.257 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58280.log.gz 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39665.log 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52009.log.gz 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45498.log 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38947.log.gz 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71308.log 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39665.log.gz 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.45498.log: -- /var/log/ceph/ceph-client.admin.47807.log 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45498.log.gz 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56978.log 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71308.log.gz 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64754.log 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47807.log.gz 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52681.log 2024-09-18T11:59:39.258 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56978.log.gz 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65678.log 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64754.log.gz 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55927.log 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52681.log.gz 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45794.log 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65678.log.gz 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54751.log 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55927.log.gz 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38207.log 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45794.log.gz 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56011.log 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54751.log.gz 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38207.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38207.log.gz 2024-09-18T11:59:39.259 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60004.log 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43228.log 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56011.log.gz 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45942.log 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60004.log.gz 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36958.log 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43228.log.gz 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63704.log 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45942.log.gz 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.36958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66392.log 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36958.log.gz 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59036.log 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63704.log.gz 2024-09-18T11:59:39.260 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61603.log 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66392.log.gz 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70257.log 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59036.log.gz 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61897.log 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61603.log.gz 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69669.log 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70257.log.gz 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70803.log 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61897.log.gz 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51925.log 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69669.log.gz 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37445.log 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70803.log.gz 2024-09-18T11:59:39.261 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65300.log 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51925.log.gz 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62107.log 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37445.log.gz 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50328.log 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65300.log.gz 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44495.log 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62107.log.gz 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72988.log 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50328.log.gz 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54709.log 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44495.log.gz 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60511.log 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72988.log.gz 2024-09-18T11:59:39.262 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60088.log 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54709.log.gz 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60511.log.gz 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69879.log 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57986.log 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60088.log.gz 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65006.log 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69879.log.gz 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44579.log 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57986.log.gz 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49446.log 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65006.log.gz 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45239.log 2024-09-18T11:59:39.263 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44579.log.gz 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55717.log 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49446.log.gz 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45239.log.gz 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63662.log 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45424.log 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55717.log.gz 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40035.log 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63662.log.gz 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54373.log 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45424.log.gz 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39517.log 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40035.log.gz 2024-09-18T11:59:39.264 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51589.log 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54373.log.gz 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62779.log 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39517.log.gz 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40239.log 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51589.log.gz 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57398.log 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62779.log.gz 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40239.log.gz 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39156.log 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57272.log 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57398.log.gz 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49866.log 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39156.log.gz 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72904.log 2024-09-18T11:59:39.265 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57272.log.gz 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50370.log 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49866.log.gz 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64796.log 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72904.log.gz 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68366.log 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50370.log.gz 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55423.log 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64796.log.gz 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56179.log 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr: 27.9% -- replaced with /var/log/ceph/ceph-client.admin.68366.log.gz 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73198.log 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55423.log.gz 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43655.log 2024-09-18T11:59:39.266 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56179.log.gz 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47765.log 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73198.log.gz 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68324.log 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43655.log.gz 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49698.log 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49656.log 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68324.log.gz 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49698.log: 28.8% -- replaced with /var/log/ceph/ceph-client.admin.47765.log.gz 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59794.log 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49698.log.gz 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59162.log 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49656.log.gz 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr:gzip/var/log/ceph/ceph-client.admin.59794.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50958.log 2024-09-18T11:59:39.267 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59794.log.gz 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63620.log 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59162.log.gz 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73114.log 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50958.log.gz 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr: 95.5% -- replaced with /var/log/ceph/ceph-client.admin.43251.log.gz 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58994.log 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63620.log.gz 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63998.log 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73114.log.gz 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58994.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.58994.log.gz -5 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr: --verbose -- /var/log/ceph/ceph-client.admin.65636.log 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39702.log 2024-09-18T11:59:39.268 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63998.log.gz 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41209.log 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65636.log.gz 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46925.log 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39702.log.gz 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59752.log 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41209.log.gz 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.46925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67526.log 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46925.log.gz 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43907.log 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59752.log.gz 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61183.log 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67526.log.gz 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54247.log 2024-09-18T11:59:39.269 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43907.log.gz 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61183.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61183.log.gz 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39628.log 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54247.log.gz 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69963.log 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.39628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57524.log 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39628.log.gz 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45276.log 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69963.log.gz 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53029.log 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57524.log.gz 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70761.log 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45276.log.gz 2024-09-18T11:59:39.270 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52177.log 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53029.log.gz 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42808.log 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70761.log.gz 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73618.log 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52177.log.gz 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69711.log 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42808.log.gz 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45350.log 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73618.log.gz 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73870.log 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69711.log.gz 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64670.log 2024-09-18T11:59:39.271 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45350.log.gz 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67190.log 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73870.log.gz 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70047.log 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64670.log.gz 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51715.log 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67190.log.gz 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37277.log 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70047.log.gz 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64712.log 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51715.log.gz 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67904.log 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37277.log.gz 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69921.log 2024-09-18T11:59:39.272 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64712.log.gz 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49067.log 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67904.log.gz 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37883.log 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69921.log.gz 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58742.log 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49067.log.gz 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68745.log 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37883.log.gz 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42976.log 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58742.log.gz 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68745.log.gzgzip 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50118.log 2024-09-18T11:59:39.273 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44915.log 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42976.log.gz 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70341.log 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50118.log.gz 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59120.log 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44915.log.gz 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67988.log 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70341.log.gz 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53953.log 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59120.log.gz 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48395.log 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67988.log.gz 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37361.log 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53953.log.gz 2024-09-18T11:59:39.274 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45646.log 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48395.log.gz 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45128.log 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37361.log.gz 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65804.log 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45646.log.gz 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61015.log 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45128.log.gz 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72358.log 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65804.log.gz 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61015.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.61015.log.gz --verbose 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-client.admin.48101.log 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60172.log 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72358.log.gz 2024-09-18T11:59:39.275 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44980.log 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48101.log.gz 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42598.log 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60172.log.gz 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48479.log 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44980.log.gz 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71896.log 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42598.log.gz 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52135.log 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48479.log.gz 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37799.log 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71896.log.gz 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67274.log 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52135.log.gz 2024-09-18T11:59:39.276 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73282.log 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37799.log.gz 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43571.log 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67274.log.gz 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53449.log 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73282.log.gz 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50622.log 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43571.log.gz 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49277.log 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53449.log.gz 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49151.log 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50622.log.gz 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61939.log 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49277.log.gz 2024-09-18T11:59:39.277 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62653.log 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49151.log.gz 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51967.log 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61939.log.gz 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.62653.log: /var/log/ceph/ceph-client.admin.38073.log 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62653.log.gz 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49950.log 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51967.log.gz 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49362.log 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38073.log.gz 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64040.log 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49950.log.gz 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39998.log 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49362.log.gz 2024-09-18T11:59:39.278 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69375.log 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64040.log.gz 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39998.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39998.log.gz 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.53323.log 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38096.log 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69375.log.gz 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68577.log 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53323.log.gz 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47681.log 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38096.log.gz 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39591.log 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68577.log.gz 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56768.log 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47681.log.gz 2024-09-18T11:59:39.279 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39591.log.gz 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44828.log 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56768.log.gz 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53533.log 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65342.log 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44828.log.gz 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58322.log 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53533.log.gz 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51757.log 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65342.log.gz 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55045.log 2024-09-18T11:59:39.280 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58322.log.gz 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51757.log.gz 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71392.log 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68072.log 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55045.log.gz 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71392.log.gzgzip 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53785.log 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72148.log 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68072.log.gz 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53785.log.gz 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65384.log 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44621.log 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72148.log.gz 2024-09-18T11:59:39.281 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65384.log.gz 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46883.log 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62695.log 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44621.log.gz 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.46883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68787.log 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr: 95.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46883.log.gz 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43141.log 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62695.log.gz 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52891.log 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68787.log.gz 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43141.log.gz 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57146.log 2024-09-18T11:59:39.282 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52891.log.gz 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54541.log 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67064.log 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57146.log.gz 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53071.log 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54541.log.gz 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69459.log 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67064.log.gz 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53071.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.53071.log.gz 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr: --verbose -- /var/log/ceph/ceph-client.admin.61813.log 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48605.log 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69459.log.gz 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61813.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.61813.log.gz /var/log/ceph/ceph-client.admin.55213.log 2024-09-18T11:59:39.283 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51504.log 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48605.log.gz 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66644.log 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55213.log.gz 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49530.log 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51504.log.gz 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67778.log 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66644.log.gz 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73156.log 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49530.log.gz 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44075.log 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67778.log.gz 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61645.log 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73156.log.gz 2024-09-18T11:59:39.284 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44075.log.gz 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69081.log 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71854.log 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61645.log.gz 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38688.log 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69081.log.gz 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49320.log 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71854.log.gz 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39554.log 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38688.log.gz 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63410.log 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49320.log.gz 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48857.log 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39554.log.gz 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73492.log 2024-09-18T11:59:39.285 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63410.log.gz 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44117.log 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48857.log.gz 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43102.log 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73492.log.gz 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39813.log 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44117.log.gz 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51883.log 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43102.log.gz 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69501.log 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39813.log.gz 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53281.log 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51883.log.gz 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47849.log 2024-09-18T11:59:39.286 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69501.log.gz 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53281.log.gz 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68913.log 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72820.log 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47849.log.gz 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55171.log 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68913.log.gz 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68240.log 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72820.log.gz 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74038.log 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55171.log.gz 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52219.log 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68240.log.gz 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.74038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50412.log 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74038.log.gz 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44700.log 2024-09-18T11:59:39.287 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52219.log.gz 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38170.log 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50412.log.gz 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61225.log 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44700.log.gz 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48689.log 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38170.log.gz 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70299.log 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61225.log.gz 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46967.log 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48689.log.gz 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48017.log 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70299.log.gz 2024-09-18T11:59:39.288 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.46967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66686.log 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46967.log.gz 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62401.log 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48017.log.gz 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53869.log 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66686.log.gz 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52555.log 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62401.log.gz 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53869.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53869.log.gz 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51631.log 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52555.log.gz 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73786.log 2024-09-18T11:59:39.289 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69795.log 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51631.log.gz 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55297.log 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73786.log.gz 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62233.log 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69795.log.gz 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62065.log 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55297.log.gz 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62233.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.62233.log.gz 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.53701.log 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42178.log 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62065.log.gz 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59626.log 2024-09-18T11:59:39.290 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53701.log.gz 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71476.log 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42178.log.gz 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59626.log.gz 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60847.log 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71476.log.gz 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67316.log 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39739.log 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60847.log.gz 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58658.log 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67316.log.gz 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58196.log 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39739.log.gz 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49572.log 2024-09-18T11:59:39.291 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58658.log.gz 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65888.log 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58196.log.gz 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49572.log.gz 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37798.log 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54163.log 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65888.log.gz 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48059.log 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37798.log.gz 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62359.log 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.54163.log.gz 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41251.log 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48059.log.gz 2024-09-18T11:59:39.292 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39369.log 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62359.log.gz 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71224.log 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41251.log.gz 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62317.log 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39369.log.gz 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60637.log 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71224.log.gz 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43529.log 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62317.log.gz 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47555.log 2024-09-18T11:59:39.293 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60637.log.gz 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50874.log 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43529.log.gz 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40072.log 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47555.log.gz 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65468.log 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50874.log.gz 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63284.log 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40072.log.gz 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71560.log 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65468.log.gz 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62023.log 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63284.log.gz 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39776.log 2024-09-18T11:59:39.294 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71560.log.gz 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50076.log 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62023.log.gz 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67946.log 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39776.log.gz 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54331.log 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50076.log.gz 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49109.log 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67946.log.gz 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64376.log 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54331.log.gz 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65972.log 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49109.log.gz 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54835.log 2024-09-18T11:59:39.295 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64376.log.gz 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48143.log 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65972.log.gz 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67358.log 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54835.log.gz 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65258.log 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48143.log.gz 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72694.log 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67358.log.gz 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65258.log.gz 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74163.log 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42682.log 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72694.log.gz 2024-09-18T11:59:39.296 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.74163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54625.log 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74163.log.gz 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr:gzip/var/log/ceph/ceph-client.admin.42682.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50538.log 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42682.log.gz 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.54625.log: -- /var/log/ceph/ceph-client.admin.71182.log 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54625.log.gz 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49488.log 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50538.log.gz 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57818.log 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71182.log.gz 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36773.log 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49488.log.gz 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69585.log 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57818.log.gz 2024-09-18T11:59:39.297 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.36773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39198.log 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36773.log.gz 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56852.log 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69585.log.gz 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66182.log 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39198.log.gz 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50790.log 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56852.log.gz 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36959.log 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66182.log.gz 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37967.log 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50790.log.gz 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.36959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33033.log 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36959.log.gz 2024-09-18T11:59:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68997.log 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37967.log.gz 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.33033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55339.log 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33033.log.gz 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43487.log 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68997.log.gz 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72106.log 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55339.log.gz 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48185.log 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43487.log.gz 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39924.log 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72106.log.gz 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39295.log 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48185.log.gz 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45868.log 2024-09-18T11:59:39.299 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39924.log.gz 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45720.log 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39295.log.gz 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38133.log 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45868.log.gz 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68871.log 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45720.log.gz 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38984.log 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38133.log.gz 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37573.log 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68871.log.gz 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47219.log 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38984.log.gz 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63368.log 2024-09-18T11:59:39.300 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37573.log.gz 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67820.log 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47219.log.gz 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45461.log 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63368.log.gz 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72610.log 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67820.log.gz 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72064.log 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45461.log.gz 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72610.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72610.log.gz 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72064.log.gz 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64124.log 2024-09-18T11:59:39.301 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51168.log 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45905.log 0.0% 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64124.log.gz 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71644.log 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51168.log.gz 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61771.log 0.0% 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45905.log.gz 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71770.log 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71644.log.gz 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57188.log 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61771.log.gz 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71770.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.71770.log.gz 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.43015.log 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58784.log 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57188.log.gz 2024-09-18T11:59:39.302 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60469.log 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43015.log.gz 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47387.log 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58784.log.gz 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38762.log 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60469.log.gz 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47387.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.47387.log.gz /var/log/ceph/ceph-client.admin.54961.log 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57734.log 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38762.log.gz 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54961.log: /var/log/ceph/ceph-client.admin.50748.log 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54961.log.gz 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71980.log 2024-09-18T11:59:39.303 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57734.log.gz 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71518.log 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50748.log.gz 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65720.log 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71980.log.gz 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37714.log 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71518.log.gz 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72778.log 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65720.log.gz 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59204.log 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37714.log.gz 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37639.log 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72778.log.gz 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52051.log 2024-09-18T11:59:39.304 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59204.log.gz 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66938.log 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37639.log.gz 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39021.log 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52051.log.gz 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42889.log 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66938.log.gz 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66014.log 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39021.log.gz 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42889.log.gz 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66518.log 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37127.log 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66014.log.gz 2024-09-18T11:59:39.305 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49824.log 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66518.log.gz 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67568.log/var/log/ceph/ceph-client.admin.37127.log: 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37127.log.gz 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67106.log 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49824.log.gz 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37715.log 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67568.log.gz 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48227.log 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67106.log.gz 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65174.log 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37715.log.gz 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56726.log 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48227.log.gz 2024-09-18T11:59:39.306 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50286.log 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65174.log.gz 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62611.log 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56726.log.gz 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44411.log 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50286.log.gz 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63830.log 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62611.log.gz 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67652.log 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44411.log.gz 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37087.log 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63830.log.gz 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73240.log 2024-09-18T11:59:39.307 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67652.log.gz 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64418.log 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37087.log.gz 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57230.log 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73240.log.gz 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55003.log 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64418.log.gz 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57230.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57230.log.gz 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65090.log 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55003.log.gz 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70467.log 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65510.log 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65090.log.gz 2024-09-18T11:59:39.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70719.log 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70467.log.gz 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68535.log 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65510.log.gz 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47975.log 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70719.log.gz 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5/var/log/ceph/ceph-client.admin.68535.log: --verbose -- /var/log/ceph/ceph-client.admin.65762.log 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68535.log.gz 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64292.log 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47975.log.gz 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52387.log 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65762.log.gz 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42511.log 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64292.log.gz 2024-09-18T11:59:39.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64964.log 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52387.log.gz 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42511.log.gz 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68493.log 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70089.log 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64964.log.gz 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55591.log 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68493.log.gz 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61519.log 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70089.log.gz 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55591.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26985.log 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55591.log.gz 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53659.log 2024-09-18T11:59:39.310 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61519.log.gz 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.tmp-client.admin.26985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72400.log 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26985.log.gz 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60130.log 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53659.log.gz 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63536.log 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.72400.log.gz 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72190.log 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60130.log.gz 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63536.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.63536.log.gz 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71686.log 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39332.log 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72190.log.gz 2024-09-18T11:59:39.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49740.log 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71686.log.gz 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45017.log 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39332.log.gz 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36815.log 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49740.log.gz 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36957.log 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45017.log.gz 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62863.log 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.36815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36815.log.gz 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.36957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56473.log 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36957.log.gz 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48353.log 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62863.log.gz 2024-09-18T11:59:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52639.log 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56473.log.gz 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55381.log 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48353.log.gz 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68198.log 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52639.log.gz 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55381.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.71013.log 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55381.log.gz 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64502.log 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68198.log.gz 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48647.log 0.0% 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71013.log.gz 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72736.log 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64502.log.gz 2024-09-18T11:59:39.313 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48647.log.gz 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65552.log 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60340.log 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72736.log.gz 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57482.log 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65552.log.gz 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49404.log 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60340.log.gz 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67694.log 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57482.log.gz 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71728.log 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49404.log.gz 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57944.log 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67694.log.gz 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70425.log 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71728.log.gz 2024-09-18T11:59:39.314 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38392.log 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57944.log.gz 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33153.log 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70425.log.gz 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42259.log 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38392.log.gz 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52723.log 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.33153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33153.log.gz 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64250.log 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42259.log.gz 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51841.log 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52723.log.gz 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73030.log 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64250.log.gz 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49235.log 2024-09-18T11:59:39.315 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51841.log.gz 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73030.log.gz 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54037.log 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65216.log 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49235.log.gz 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48311.log 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54037.log.gz 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39443.log 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65216.log.gz 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56263.log 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48311.log.gz 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66266.log 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39443.log.gz 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52807.log 2024-09-18T11:59:39.316 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56263.log.gz 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69039.log 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66266.log.gz 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52807.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.58154.log -- replaced with /var/log/ceph/ceph-client.admin.52807.log.gz 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59668.log 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69039.log.gz 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38836.log 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58154.log.gz 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63956.log 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59668.log.gz 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32720.log 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38836.log.gz 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63956.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.37882.log -- replaced with /var/log/ceph/ceph-client.admin.63956.log.gz 2024-09-18T11:59:39.317 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.32720.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32720.log.gz 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57104.log 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37882.log.gz 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53239.log 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57104.log.gz 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36857.log 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53239.log.gz 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40109.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.40109.log.gz 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.69207.log 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.36857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61351.log 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36857.log.gz 2024-09-18T11:59:39.318 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74080.log 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69207.log.gz 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68156.log 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61351.log.gz 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.74080.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.74080.log.gz 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60427.log 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69333.log 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68156.log.gz 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47429.log 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60427.log.gz 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73996.log 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69333.log.gz 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63326.log 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47429.log.gz 2024-09-18T11:59:39.319 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73996.log.gz 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53827.log 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56095.log 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63326.log.gz 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53827.log.gz 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55255.log 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69543.log 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56095.log.gz 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55255.log.gz 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68955.log 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60763.log 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69543.log.gz 2024-09-18T11:59:39.320 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68955.log.gz 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48731.log 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33365.log 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60763.log.gz 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48731.log.gz 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38031.log 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr: 95.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.33365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33073.log 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33365.log.gz 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38281.log 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38031.log.gz 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.33073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63788.log 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33073.log.gz 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50034.log 2024-09-18T11:59:39.321 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38281.log.gz 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62821.log 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63788.log.gz 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50034.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.50034.log.gz /var/log/ceph/ceph-client.admin.55507.log 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61141.log 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62821.log.gz 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45831.log 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55507.log.gz 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61141.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.61141.log.gz -- 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.45609.log 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63242.log 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45831.log.gz 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44829.log 2024-09-18T11:59:39.322 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45609.log.gz 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63242.log.gz 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57608.log 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60721.log 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44829.log.gz 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37086.log 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57608.log.gz 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63578.log 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60721.log.gz 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37086.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37086.log.gz -5 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr: --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58868.log 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63578.log.gz 2024-09-18T11:59:39.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73912.log 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51673.log 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58868.log.gz 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66056.log 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73912.log.gz 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54499.log 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51673.log.gz 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66056.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66056.log.gz -5 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr: --verbose -- /var/log/ceph/ceph-client.admin.65132.log 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70593.log 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54499.log.gz 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45757.log 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65132.log.gz 2024-09-18T11:59:39.324 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63116.log 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70593.log.gz 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44033.log 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45757.log.gz 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43865.log 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63116.log.gz 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50916.log 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44033.log.gz 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72442.log 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43865.log.gz 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50916.log.gz 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70635.log 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71434.log 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72442.log.gz 2024-09-18T11:59:39.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66476.log 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70635.log.gz 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38725.log 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71434.log.gz 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69249.log 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66476.log.gz 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56053.log 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38725.log.gz 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73744.log 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69249.log.gz 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37529.log 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56053.log.gz 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73744.log.gz 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58112.log 2024-09-18T11:59:39.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62947.log 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37529.log.gz 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71139.log 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58112.log.gz 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56936.log 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62947.log.gz 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72274.log 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71139.log.gz 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55969.log 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56936.log.gz 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42766.log 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72274.log.gz 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56641.log 2024-09-18T11:59:39.327 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55969.log.gz 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42766.log.gz 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58574.log 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69837.log 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56641.log.gz 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73534.log 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58574.log.gz 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70845.log 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69837.log.gz 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43016.log 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73534.log.gz 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70845.log.gz 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39480.log 2024-09-18T11:59:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67232.log 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43016.log.gz 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39480.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.39480.log.gz -- 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.47345.log 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44957.log 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67232.log.gz 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47345.log.gz 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52513.log 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42888.log 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44957.log.gz 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52513.log.gz 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43319.log 2024-09-18T11:59:39.329 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42888.log.gz 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59498.log 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gz 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43319.log.gzgzip 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47093.log 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38244.log 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59498.log.gz 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47261.log 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47093.log.gz 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60595.log 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38244.log.gz 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73702.log 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47261.log.gz 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59878.log 2024-09-18T11:59:39.330 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60595.log.gz 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59246.log 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73702.log.gz 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52261.log 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59878.log.gz 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56557.log 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59246.log.gz 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53911.log 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52261.log.gz 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55801.log 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56557.log.gz 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44453.log 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53911.log.gz 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59372.log 2024-09-18T11:59:39.331 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55801.log.gz 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54289.log 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44453.log.gz 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52597.log 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59372.log.gz 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64628.log 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54289.log.gz 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43697.log 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52597.log.gz 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54415.log 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64628.log.gz 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70551.log 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43697.log.gz 2024-09-18T11:59:39.332 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53155.log 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54415.log.gz 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50160.log 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70551.log.gz 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53155.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.64544.log 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53155.log.gz 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52765.log 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50160.log.gz 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45535.log 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64544.log.gz 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58406.log 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52765.log.gz 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64166.log 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45535.log.gz 2024-09-18T11:59:39.333 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58070.log 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58406.log.gz 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60679.log 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64166.log.gz 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68619.log 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58070.log.gz 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67484.log 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60679.log.gz 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.68619.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68619.log.gz 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63200.log 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67484.log.gz 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59962.log 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56389.log 2024-09-18T11:59:39.334 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63200.log.gz 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62989.log 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59962.log.gz 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51126.log 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56389.log.gz 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48773.log 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62989.log.gz 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51126.log.gz 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55633.log 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49025.log 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48773.log.gz 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64082.log 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55633.log.gz 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69123.log 2024-09-18T11:59:39.335 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49025.log.gz 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60046.log 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64082.log.gz 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60931.log 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69123.log.gz 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56347.log 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60046.log.gz 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48269.log 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60931.log.gz 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58364.log 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56347.log.gz 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57902.log 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48269.log.gz 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58364.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.58532.log 2024-09-18T11:59:39.336 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58364.log.gz 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:gzip/var/log/ceph/ceph-client.admin.57902.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37403.log 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57902.log.gz 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55549.log 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58532.log.gz 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54919.log 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37403.log.gz 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59583.log 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55549.log.gz 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61855.log 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54919.log.gz 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73366.log 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59583.log.gz 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38355.log 2024-09-18T11:59:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61855.log.gz 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39258.log 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73366.log.gz 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49193.log 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38355.log.gz 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36715.log 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39258.log.gz 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66224.log 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49193.log.gz 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.36715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60298.log 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36715.log.gz 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52471.log 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66224.log.gz 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44369.log 2024-09-18T11:59:39.338 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60298.log.gz 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58238.log 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52471.log.gz 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54205.log 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44369.log.gz 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42258.log 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58238.log.gz 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38873.log 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54205.log.gz 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42512.log 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42258.log.gz 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40281.log 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38873.log.gz 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55759.log 2024-09-18T11:59:39.339 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42512.log.gz 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66854.log 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40281.log.gz 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42384.log 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55759.log.gz 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66854.log.gz 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55675.log 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53575.log 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42384.log.gz 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43142.log 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55675.log.gz 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:gzip/var/log/ceph/ceph-client.admin.53575.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53995.log 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53575.log.gz 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56683.log 2024-09-18T11:59:39.340 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43142.log.gz 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43991.log 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53995.log.gz 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71602.log 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56683.log.gz 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69417.log 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43991.log.gz 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37947.log 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71602.log.gz 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47177.log 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69417.log.gz 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66980.log 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37947.log.gz 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51294.log 2024-09-18T11:59:39.341 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47177.log.gz 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57566.log 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66980.log.gz 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44327.log 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51294.log.gz 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54667.log 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57566.log.gz 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65048.log 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44327.log.gz 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54793.log 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54667.log.gz 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56137.log 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65048.log.gz 2024-09-18T11:59:39.342 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61729.log 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54793.log.gz 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56221.log 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56137.log.gz 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66770.log 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61729.log.gz 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37616.log 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56221.log.gz 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67022.log 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66770.log.gz 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37126.log 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37616.log.gz 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70383.log 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67022.log.gz 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56599.log 2024-09-18T11:59:39.343 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37126.log.gz 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47639.log 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70383.log.gz 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63452.log 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56599.log.gz 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66140.log 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47639.log.gz 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47051.log 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63452.log.gz 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59078.log 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66140.log.gz 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44537.log 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47051.log.gz 2024-09-18T11:59:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59078.log.gz 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64586.log 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59414.log 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44537.log.gz 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74123.log 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64586.log.gz 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69753.log 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59414.log.gz 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.74123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59710.log 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74123.log.gz 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58826.log 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69753.log.gz 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63872.log 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59710.log.gz 2024-09-18T11:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36956.log 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58826.log.gz 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51799.log 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63872.log.gz 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.36956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71097.log 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36956.log.gz 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49782.log 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51799.log.gz 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45202.log 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71097.log.gz 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55087.log 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49782.log.gz 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.45202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60805.log 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45202.log.gz 2024-09-18T11:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52303.log 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55087.log.gz 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.60805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72862.log 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60805.log.gz 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39961.log 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52303.log.gz 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47009.log 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72862.log.gz 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66602.log 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39961.log.gz 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63746.log 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47009.log.gz 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38651.log 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66602.log.gz 2024-09-18T11:59:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70971.log 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63746.log.gz 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59330.log 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38651.log.gz 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61309.log 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70971.log.gz 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61477.log 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59330.log.gz 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42472.log 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61309.log.gz 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53407.log 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61477.log.gz 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56431.log 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42472.log.gz 2024-09-18T11:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72946.log 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53407.log.gz 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42724.log 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56431.log.gz 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54583.log 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72946.log.gz 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70215.log 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42724.log.gz 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44699.log 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54583.log.gz 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63494.log 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70215.log.gz 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47891.log 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44699.log.gz 2024-09-18T11:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62275.log 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63494.log.gz 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64922.log 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47891.log.gz 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73408.log 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62275.log.gz 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.64922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73324.log 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64922.log.gz 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38503.log 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73408.log.gz 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.73324.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73324.log.gz 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62569.log 2024-09-18T11:59:39.350 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38503.log.gz 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44789.log 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53491.log 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62569.log.gz 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59288.log 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44789.log.gz 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50580.log 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53491.log.gz 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.59288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50202.log 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59288.log.gz 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56515.log 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50580.log.gz 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52975.log 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50202.log.gz 2024-09-18T11:59:39.351 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52933.log 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.56515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56515.log.gz 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57356.log 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52975.log.gz 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.52933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37319.log 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52933.log.gz 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67736.log 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57356.log.gz 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70005.log 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37319.log.gz 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47135.log 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67736.log.gz 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.70005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72232.log 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70005.log.gz 2024-09-18T11:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69291.log 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47135.log.gz 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67442.log 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72232.log.gz 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.69291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72316.log 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69291.log.gz 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.67442.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.67442.log.gz 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58700.log 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72526.log 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72316.log.gz 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72652.log 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58700.log.gz 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60553.log 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72526.log.gz 2024-09-18T11:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.72652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72652.log.gz 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66434.log 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:gzip/var/log/ceph/ceph-client.admin.60553.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63031.log 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60553.log.gz 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.66434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62527.log 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66434.log.gz 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.63031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62905.log 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63031.log.gz 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50832.log 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62527.log.gz 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65846.log 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62905.log.gz 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48815.log 2024-09-18T11:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50832.log.gz 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.65846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51420.log 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65846.log.gz 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61267.log 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48815.log.gz 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50454.log 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51420.log.gz 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.61267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53365.log 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61267.log.gz 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48941.log 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50454.log.gz 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.53365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62737.log 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53365.log.gz 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58616.log 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.48941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48941.log.gz 2024-09-18T11:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.62737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49908.log 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62737.log.gz 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58616.log.gz 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57062.log 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47597.log 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49908.log.gz 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.57062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57062.log.gz 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42640.log 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.47597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54457.log 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47597.log.gz 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58448.log 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42640.log.gz 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54121.log 2024-09-18T11:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54457.log.gz 2024-09-18T11:59:39.357 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.58448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54079.log 2024-09-18T11:59:39.357 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58448.log.gz 2024-09-18T11:59:39.357 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37235.log 2024-09-18T11:59:39.357 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54121.log.gz 2024-09-18T11:59:39.357 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.54079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71350.log 2024-09-18T11:59:39.357 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54079.log.gz 2024-09-18T11:59:39.357 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37235.log.gz 2024-09-18T11:59:39.357 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.71350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71350.log.gz 2024-09-18T11:59:40.547 INFO:teuthology.orchestra.run.smithi071.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-18T11:59:40.916 INFO:teuthology.orchestra.run.smithi071.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-18T11:59:41.006 INFO:teuthology.orchestra.run.smithi071.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-18T11:59:44.091 INFO:teuthology.orchestra.run.smithi071.stderr: 89.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-18T12:09:30.480 INFO:teuthology.orchestra.run.smithi071.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-18T12:11:04.640 INFO:teuthology.orchestra.run.smithi071.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-18T12:12:24.753 INFO:teuthology.orchestra.run.smithi071.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-18T12:13:05.090 INFO:teuthology.orchestra.run.smithi071.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-18T12:13:05.092 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-18T12:13:05.092 INFO:teuthology.orchestra.run.smithi071.stderr:real 13m26.153s 2024-09-18T12:13:05.092 INFO:teuthology.orchestra.run.smithi071.stderr:user 5m53.030s 2024-09-18T12:13:05.092 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m16.135s 2024-09-18T12:13:05.092 INFO:tasks.ceph:Archiving logs... 2024-09-18T12:13:05.093 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910981/remote/smithi071/log 2024-09-18T12:13:05.094 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T12:14:08.017 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T12:14:08.027 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T12:14:08.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T12:14:08.179 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T12:14:08.187 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T12:14:08.188 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T12:14:08.205 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-09-18T12:14:08.248 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T12:14:08.248 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-09-18T12:14:08.248 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv01.front.sepia.ceph.com 3 6 377 58 +15us[ +21us] +/- 42ms 2024-09-18T12:14:08.248 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 57 -1320us[-1320us] +/- 64ms 2024-09-18T12:14:08.248 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +1369us[+1369us] +/- 64ms 2024-09-18T12:14:08.248 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:14:08.249 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T12:14:08.257 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T12:14:08.258 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T12:14:08.266 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T12:14:08.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-18T12:14:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:14:08.479 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T12:14:08.489 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T12:14:08.500 INFO:teuthology.task.internal:Duration was 2228.268282 seconds 2024-09-18T12:14:08.500 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T12:14:08.511 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T12:14:08.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T12:14:08.591 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T12:14:08.649 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T12:14:08.649 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-09-18T12:14:08.650 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T12:14:08.669 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T12:14:08.669 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T12:14:08.737 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T12:14:08.737 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T12:14:08.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T12:14:08.931 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T12:14:08.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T12:14:08.962 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T12:14:08.972 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:14:09.042 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-09-18T12:14:09.081 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:14:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:14:09.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T12:14:09.147 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T12:14:09.147 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910981/remote/smithi071 2024-09-18T12:14:09.147 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T12:14:09.211 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T12:14:09.211 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T12:14:09.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T12:14:09.270 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T12:14:09.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T12:14:09.279 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T12:14:09.279 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T12:14:09.317 INFO:teuthology.orchestra.run.smithi071.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 12:14 /home/ubuntu/cephtest 2024-09-18T12:14:09.317 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T12:14:09.327 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T12:14:09.349 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} duration: 2228.2682819366455 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T12:14:09.349 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T12:14:09.464 INFO:teuthology.run:pass